merge dang-mapping branch

README-osx.md

@@ -0,0 +1,169 @@
+# Development version of SEED on a Mac
+
+These instructions are for installing and running SEED on Mac OSX in development mode.
+
+## Prerequisites
+
+These instructions assume you have/use [Macports](https://www.macports.org/).
+
+Although you _could_ install Python packages globally, the easiest way to install Python packages is with [virtualenv](https://virtualenv.pypa.io/en/latest/) and [virtualenvwrapper](https://virtualenvwrapper.readthedocs.org/en/latest/). Setting these up first will help avoid polluting your base Python installation and make it much easier to switch between different versions of the code.
+
+Once you have these installed, creating and entering a new virtualenv called "``seed``" for SEED development is as easy as:
+
+    mkvirtualenv --python=python2.7 seed
+
+
+## Install PostgreSQL 9.4
+
+Perform the following commands as 'root'
+
+    sudo su - root
+
+Install Postgres 9.4
+
+    port install postgresql94-server postgresql94 postgresql94-doc
+    # init db
+    mkdir -p /opt/local/var/db/postgresql94/defaultdb
+    chown postgres:postgres /opt/local/var/db/postgresql94/defaultdb
+
+Finish initializing the DB
+
+    sudo su postgres - c '/opt/local/lib/postgresql94/bin/initdb -D /opt/local/var/db/postgresql94/defaultdb'
+
+At this point, you may want to add start/stop scripts or aliases to ~/.bashrc or your virtualenv ``postactivate`` script (in ``~/.virtualenvs/{env-name}/bin/postactivate``).
+
+    alias pg_start='sudo su postgres -c "/opt/local/lib/postgresql94/bin/pg_ctl \
+        -D /opt/local/var/db/postgresql94/defaultdb \
+        -l /opt/local/var/db/postgresql94/defaultdb/postgresql.log start"'
+    alias pg_stop='sudo su postgres -c "/opt/local/lib/postgresql94/bin/pg_ctl \
+        -D /opt/local/var/db/postgresql94/defaultdb stop"'
+
+Start Postgres
+
+    pg_start
+
+Switch to postgres user
+
+      sudo su - postgres
+      PATH=$PATH:/opt/local/lib/postgresql94/bin/
+      
+Configure Postgresql. Replace 'seeddb', 'seeduser' with desired db/user.
+seedpass
+
+    createdb seeddb
+    createuser -P seeduser
+    psql -c 'GRANT ALL PRIVILEGES ON DATABASE "seeddb" TO seeduser;'
+
+Now exit any root environments, becoming just yourself (even though it's not that easy being green..), for the remainder of these instructions.
+
+## Install Python packages
+
+Run these commands as your normal user id.
+
+Change to a virtualenv (using virtualenvwrapper) or do the following as a superuser. A virtualenv is usually better for development.
+
+Make sure PostgreSQL command line scripts are in your PATH
+
+    export PATH=$PATH:/opt/local/lib/postgresql94/bin
+
+Install requirements with `pip`
+
+    pip install -r requirements.txt
+
+Install library with `setup.py`
+
+    python setup.py install
+
+## Install Javascript libraries
+
+### Install nodejs/npm
+
+First, install [npm](https://www.npmjs.com/) if you haven't already. You can do this by installing from [nodejs.org](http://nodejs.org/), or use Macports:
+
+    sudo port install npm
+
+### Install libraries
+
+Then run, from the top-level, a script to install the JS libraries:
+
+    ./bin/install_javascript_dependencies.sh
+
+## Configure Django and its back-end DBs
+
+In the `BE/settings` directory, there must be a file called `local_untracked.py` that sets up databases and a number of other things. To create and edit this file, start by copying over the template
+
+    cd BE/settings
+    cp local_untracked.py.dist local_untracked.py
+
+### Edit `local_untracked.py`
+
+Then open the file you created in your favorite editor.
+Edit the postgresql config to look like this:
+
+    # postgres DB config
+    DATABASES = {
+        'default': {
+            'ENGINE': 'django.db.backends.postgresql_psycopg2',
+            'NAME': 'seeddb',
+            'USER': 'seeduser',
+            'PASSWORD': 'seedpass',
+            'HOST': 'localhost',
+            'PORT': '5432',
+        }
+    }
+
+You may want to comment out the AWS settings.
+
+For Redis, edit the `CACHES` and `BROKER_URL` values to look like this:
+
+    CACHES = {
+        'default': {
+            'BACKEND': 'redis_cache.cache.RedisCache',
+            'LOCATION': "127.0.0.1:6379",
+            'OPTIONS': {'DB': 1},
+            'TIMEOUT': 300
+        }
+    }   
+    BROKER_URL = 'redis://127.0.0.1:6379/1'
+
+### Run Django migrations
+
+Change back to the root of the repository. Now run the migration script to set up the database tables
+
+    export DJANGO_SETTINGS_MODULE=BE.settings.dev
+    ./manage.py syncdb --migrate
+
+### Create Django admin user
+
+You need a Django admin (super) user.
+
+    ./manage.py create_default_user [email protected] --organization=lbnl --password=badpass
+
+Of course, you need to save this user/password somewhere, since this is what you will use to login to the SEED website.
+
+### Install Redis
+
+You need to manually install Redis for Celery to work.
+
+    sudo port install redis
+
+## Run the development server
+
+You should put the following statement in ~/.bashrc or add it to the virtualenv post-activation script (e.g., in `~/.virtualenvs/seed/bin/postactivate`).
+
+    export DJANGO_SETTINGS_MODULE=BE.settings.dev
+
+If you haven't already, you need to start the Redis server. For development, it is convenient to run it as yourself, logging to /tmp
+
+    # run in background
+    redis-server >/tmp/redis-server.log 2>&1 &
+
+Next start Celery, also in the background and logging to /tmp
+
+    ./manage.py celery worker -B -c 2 --loglevel=INFO -E --maxtasksperchild=1000 >/tmp/celery.log 2>&1 &
+
+Finally, run the Django standalone server
+
+    ./manage.py runserver --settings=BE.settings.dev
+
+Login with the user/password you created before, e.g., `[email protected]` and `badpass`.

requirements.txt

@@ -51,8 +51,10 @@ salad==0.4.14
 simplejson==3.3.0
 six==1.5.2
 South==0.8.4
-Sphinx==1.2.2
+Sphinx==1.2.3
 unicodecsv==0.9.4
 xlwt==0.7.5
 xmltodict==0.9.0
 uwsgi==2.0.9
+unittest2
+pytest

scripts/seedutil

@@ -0,0 +1,58 @@
+#!/usr/bin/env python
+"""
+Run utility functions from a command-line driver
+"""
+import argparse
+import logging
+import sys
+from seed.common import util, mapper
+
+DEBUG = True
+
+_log = logging.getLogger("seed.util")
+_log.addHandler(logging.StreamHandler())
+if DEBUG:
+    _log.setLevel(logging.DEBUG)
+else:
+    _log.setLevel(logging.WARN)
+
+def error(msg):
+    _log.error(msg)
+    if DEBUG:
+        import traceback
+        traceback.print_exc()
+
+def create_map(args):
+    util.create_map(args.input, args.output)
+
+def apply_map(args):
+    out_fp = sys.stdout if args.output == '-' else open(args.output, "w")
+    util.apply_map(args.mapping, args.data, out_fp)
+
+def main():
+    p = argparse.ArgumentParser()
+    subp = p.add_subparsers()
+    # command: create => create a mapping
+    p1 = subp.add_parser("create", help="Create mapping from CSV file")
+    p1.add_argument("input")
+    p1.add_argument("output")
+    p1.set_defaults(func=create_map)
+    # command: map => apply a mapping to a data file
+    p2 = subp.add_parser("map", help="Apply a mapping to data")
+    p2.add_argument("mapping", help="JSON mapping file")
+    p2.add_argument("data", help="CSV data file")
+    p2.add_argument("output", help="CSV output file")
+    p2.set_defaults(func=apply_map)
+    # parse & run
+    args = p.parse_args()
+    result = 0
+    try:
+        args.func(args)
+    except Exception, err:
+        error("In '{}': {}".format(args.func.__name__, err))
+        result = -1
+    return result
+
+
+if __name__ == '__main__':
+    sys.exit(main())

seed/common/__init__.py

@@ -0,0 +1,6 @@
+"""
+Description.
+"""
+__author__ = 'Dan Gunter <[email protected]>'
+__date__ = '2/13/15'
+