SpectrumX Data System | Gateway

Metadata management and web interface for SDS.

• SpectrumX Data System | Gateway
  • Development environment
  • Local deploy
  • Debugging tips
  • Production deploy

Development environment

System dependencies

# dev bindings for python and postgres
sudo apt install python3-dev libpq-dev # on ubuntu
sudo dnf install python3-devel postgresql-devel # on RHEL

# get psql, createdb, etc.
sudo apt install postgresql-client
sudo dnf install postgresql

Python dependencies

pip can be used, but the easiest and fastest way is to use uv (installing uv). If you still want to use pip, consider the compatible and faster alternative uv pip (e.g. alias pip=uv pip).

uv sync --frozen --extra local
# --frozen does not upgrade the dependencies
# --extra local installs the required dependencies + 'local' ones (for local development)

Note

When using uv, all base, local, and production dependencies are described in the pyproject.toml file.

If you're using pip, refer to the requirements/ directory.

Install pre-commit hooks to automatically run linters, formatters, etc. before committing:

uv run pre-commit install

Local deploy

The project is divided into two configuration modes: local and prod/production. The local mode has better responsiveness thanks to bind mounts, easier debugging, and shorter restart times. The production mode focuses on performance with caching, minified assets; and security with enforced HTTPS and separate secrets.

For the local deploy:

1. Set secrets:

   rsync -aP ./.envs/example/ ./.envs/local
   # manually set the secrets in .envs/local/*.env files

   [!NOTE] In minio.env, AWS_SECRET_ACCESS_KEY == MINIO_ROOT_PASSWORD;

   In django.env, to generate the API_KEY get it running first, then navigate to localhost:8000/users/generate-api-key. Copy the generated key to that file. The key is not stored in the database, so you will only see it at creation time.

2. Deploy with Docker (recommended):

   Either create an sds-network-local network manually, or run the Traefik service that creates it:

   docker network create sds-network-local --driver=bridge --name=sds-network-local

   Then, run the services:

   docker compose -f compose.local.yaml --env-file .envs/production/opensearch.env up

   Add -d to run in detached mode.

   If you have issues with static files, you can check which ones are being generated by the node service:

   http://localhost:3000/webpack-dev-server

3. Make Django migrations and run them:

   docker exec -it sds-gateway-local-app python manage.py makemigrations
   docker exec -it sds-gateway-local-app python manage.py migrate

4. Create the first superuser:

   docker exec -it sds-gateway-local-app python manage.py createsuperuser

5. Access the web interface:

   Open the web interface at localhost:8000. You can create regular users by signing up there.

   You can sign in with the superuser credentials at localhost:8000/admin to access the admin interface.

6. Create the MinIO bucket:

   Go to localhost:9001 and create a bucket named spectrumx with the credentials set in minio.env.

7. Run the test suite:

   docker exec -it sds-gateway-local-app python manage.py test --force-color

   Tests that run:

   • test_authenticate
   • test_create_file
   • test_retrieve_file
   • test_retrieve_latest_file
   • test_update_file
   • test_delete_file
   • test_file_contents_check
   • test_download_file
   • test_minio_health_check
   • test_opensearch_health_check

8. Run template checks:

   docker exec -it sds-gateway-local-app python manage.py validate_templates

Debugging tips

• Where are my static files served from?
  • See localhost:3000/webpack-dev-server.
• What is the URL to X / how to see my routing table?
  • docker exec -it sds-gateway-local-app python manage.py show_urls.
  • show_urls is provided by django-extensions.

Production deploy

Tip

The production deploy uses the same host ports as the local one, just prefixed with 1: (8000 → 18000).

This means you can deploy both on the same machine e.g. dev/test/QA/staging and production as "local" and "prod" respectively. This works as they also use different docker container, network, volume, and image names. Traefik may be configured to route e.g. sds.example.com and sds-dev.example.com to the respective the prod and local services respectively, using different container names and ports.

Keep this in mind, however:

Caution

Due to the bind mounts of the local deploy, it's still recommended to use different copies of the source code between a local and production deploy, even if they are on the same machine.

1. Set secrets:

   rsync -aP ./.envs/example/ ./.envs/production
   # manually set the secrets in .envs/production/*.env files

   [!NOTE]

   Follow these steps to set the secrets:

   • Set most secrets, passwords, tokens, etc. to random values. You can use the following one-liner and adjust the length as needed:

     echo $(head /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 64)

   • In minio.env, AWS_SECRET_ACCESS_KEY must be equal to MINIO_ROOT_PASSWORD;

   • In django.env, the DJANGO_ADMIN_URL must end with a slash /.

   • In django.env, to generate the API_KEY get it running first, then navigate to localhost:18000/users/generate-api-key (or this path under your own domain).

     • Copy the generated key to that env file. The key is not stored in the database, so you will only see it at creation time.

   • In django.env, configure OAuth in Auth0's dashboard and set the CLIENT_ID and CLIENT_SECRET accordingly.

   • In django.env, set the SVI_SERVER_EMAIL and SVI_SERVER_API_KEY to match the values in the SVI's environment variables.

   • In postgres.env, don't forget to set DATABASE_URL to match the user, password, and database name in that file.

2. Deploy with Docker (recommended):

   Either create an sds-network-prod network manually, or run the Traefik service that creates it:

   docker network create sds-network-prod --driver=bridge --name=sds-network-prod

   Generate the OpenSearch certificates:

   opensearch/generate_certs.sh
   ls -alh ./opensearch/data/certs/
   ls -alh ./opensearch/data/certs-django/

   Set stricter permissions to config

   chmod -v 600 compose/*/opensearch/opensearch.yaml

   Build the OpenSearch service with the right env vars to avoid permission errors in opensearch:

   # edit `opensearch.env` with the UID and GID of the host
   "${EDITOR:nano}" .envs/production/opensearch.env
   
   # build the modified opensearch image
   docker compose -f compose.production.yaml --env-file .envs/production/opensearch.env build opensearch

   Then, run the services:

   docker compose -f compose.production.yaml --env-file .envs/production/opensearch.env up

   [!TIP]

   When restarting, don't forget to re-build it, as this deploy doesn't use a bind mount to the source code:

   export COMPOSE_FILE=compose.production.yaml; docker compose build && docker compose down; docker compose up -d; docker compose logs -f
   # tip: save this command for repeated use (alias) as you get everything set up

3. Make Django migrations and run them:

   Optionally, just run them in case you have a staging deploy and would like to test new migrations first.

   docker exec -it sds-gateway-prod-app bash -c "python manage.py makemigrations && python manage.py migrate"

4. Create the first superuser:

   docker exec -it sds-gateway-prod-app python manage.py createsuperuser
   
   # if you forget or lose the superuser password, you can reset it with:
   docker exec -it sds-gateway-prod-app python manage.py changepassword <email>

5. Try the web interface and admin panel:

   Open the web interface at localhost:18000. You can create regular users by signing up there.

   You can sign in with the superuser credentials at localhost:18000/<admin path set in django.env> to access the admin interface.

6. Create the MinIO bucket:

   Go to localhost:19001 and create a bucket named spectrumx with the credentials set in .envs/production/minio.env.

7. Set correct permissions for the media volume:

   The app container uses a different pair of UID and GID than the host machine, which prevents the app from writing to the media volume when users upload files. To fix this, run the following command:

   # check the uid and gid assigned to the app container
   docker exec -it sds-gateway-prod-app id
   
   # change the ownership of the media volume to those values
   docker exec -it -u 0 sds-gateway-prod-app chown -R 100:101 /app/sds_gateway/media/

8. OpenSearch adjustments

   If you would like to modify the OpenSearch user permissions setup (through the security configuration), see the files in compose/production/opensearch/config (and reference the OpenSearch documentation for these files):

   • internal_users.yml: In this file, you can set initial users (this is where the OPENSEARCH_USER and admin user are set).

   • roles.yml: Here, you can set up custom roles for users. The extensive list of allowed permissions can be found here.

   • roles_mapping.yml: In this file, you can map roles to users defined in internal_users.yml. It is necessary to map a role directly to a user by adding them to the users list when using HTTP Basic Authentication with OpenSearch and not an external authentication system.

   Run the following command to confirm changes:

   docker exec -it sds-gateway-prod-opensearch /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
       -cd /usr/share/opensearch/config/opensearch-security/ -icl -nhnv \
       -cacert /usr/share/opensearch/config/certs/root-ca.pem \
       -cert /usr/share/opensearch/config/certs/admin.pem \
       -key /usr/share/opensearch/config/certs/admin-key.pem

   Or you can restart the OpenSearch Docker container to reflect the changes. This option takes longer as you must wait for the container to start.

   [!TIP] If you want to reserve users or permissions so they cannot be changed through the API and only through running the securityadmin.sh script, set a parameter on individual entries: reserved: true.

   If you would like to preserve changes to your .opendistro_security (e.g. users or roles you have added through the API), add the -backup flag before running the script. Use the -f flag instead of the -cd flag if you would like to only update one of the config files. See the OpenSearch documentation on the nuances of this script for more information.

9. Run the test suite:

   docker exec -it sds-gateway-prod-app python manage.py test

10. Don't forget to approve users to allow them to create API keys.

    You can do this by logging in as a superuser in the admin panel and enabling the is_approved flag in the user's entry.

OpenSearch Query Tips

The API gives the user the ability to search captures using their metadata properties indexed in OpenSearch. To do so, you must add metadata_filters to your request to the capture listing endpoint.

The metadata_filters parameter is a JSON encoded list of dictionary objects which contain: + field_path: The path to the document field you want to filter by. + query_type: The OpenSearch query type defined in the OpenSearch DSL + filter_value: The value, or configuration of values, you want to filter for.

For example:

{
    "field_path": "capture_props.<field_name>",
    "query_type": "match",
    "filter_value": "<field_value_to_match>"
}

Note

You do not have to worry about building nested queries. The API handles nesting based on the dot notation in the field_path. Only provide the inner-most filter_value, the actual filter you want to apply to the field, when constructing filters for requests.

To ensure your filters are accepted by OpenSearch, you should reference the OpenSearch query DSL documentation for more details on how filters are structured. The API leaves this structure up to the user to construct to allow for more versatility in the search functionality.

Here are some useful examples of advanced queries one might want to make to the SDS:

1. Range Queries

   Range queries may be performed both on numerical fields as well as on date fields.

   Let's say you want to search for captures with a center frequency within the range 1990000000 and 2010000000. That filter would be constructed like this:

       {
           "field_path": "capture_props.center_freq",
           "query_type": "range",
           "filter_value": {
               "gte": 1990000000,
               "lte": 2010000000
           }
       }

   Or, let's say you want to look up captures uploaded in the last 6 months:

       {
           "field_path": "created_at",
           "query_type": "range",
           "filter_value": {
               "gte": "now-6M"
           }
       }

   [!Note] now is a keyword in OpenSearch that refers to the current date and time.

   More information about range queries can be found here.

2. Geo-bounding Box Queries

   Geo-bounding box queries are useful for finding captures based on the GPS location of the sensor. They allow you to essentially create a geospatial window and query for captures within that window. This type of filter can only be performed on geo_point fields. The SDS creates coordinates fields from latitude and longitude pairs found in the metadata.

   For example, the following filter will show captures with a latitude that is between 20° and 25° north, and a longitude that is between 80° and 85° west:

       {
           "field_path": "capture_props.coordinates",
           "query_type": "geo_bounding_box",
           "filter_value": {
               "top_left": {
                   "lat": 25,
                   "lon": -85,
               },
               "bottom_right": {
                   "lat": 20,
                   "lon": -80
               }
           }
       }

   More information about geo_bounding_box queries can be found here.

3. Geodistance Queries

   Geodistance queries allow you to filter captures based on their distance to a specified GPS location. Another useful query for GPS data.

   The following filter looks for captures with 10 mile radius of the University of Notre Dame campus, main building (approximately: 41.703, -86.243):

       {
           "field_path": "capture_props.coordinates",
           "query_type": "geo_distance",
           "filter_value": {
               "distance": "10mi",
               "capture_props.coordinates": {
                   "lat": 41.703,
                   "lon": -86.243
               }
           }
       }

   More information about geo_distance queries can be found here.