Skip to content

Commit

Permalink
Documentation: Cloud Storage Utility (move-coop#350)
Browse files Browse the repository at this point in the history
* Add docs.

* Add links.

* Lint.
  • Loading branch information
jburchard authored Aug 16, 2020
1 parent 1b8dd54 commit 8a7cb4a
Show file tree
Hide file tree
Showing 3 changed files with 52 additions and 4 deletions.
1 change: 1 addition & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,7 @@ Indices and tables
dbsync
table
notifications
utilities

.. toctree::
:maxdepth: 1
Expand Down
47 changes: 47 additions & 0 deletions docs/utilities.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
*********
Utilities
*********

.. _cloud-storage:
=============
Cloud Storage
=============
The Parsons cloud storage utility was created to interact with APIs that require access to files
to run an asynchronous process. While this utility currently only works with ``S3``, the goal
is add functionality for additional cloud storage services in the future.

The cloud storage utility is currently being utilitized primarily by the NGPVAN class
methods such as :func:`~parsons.ngpvan.van.Scores.upload_scores` and Bulk Import methods.

These methods have arguments specific their method, but all also contain the following cloud
storage arguments:

* ``url_type`` - The type of cloud storage to utilize. Currently only ``S3``.

* ``**url_kwargs`` - These are arguments specific to the cloud storage type in order to initialize.

**S3**

When you select the ``url_type`` S3, the Parsons table will be converted to a csv and compressed. The file will be posted to an S3 bucket. A presigned public url will be generated and returned. The url will be active by default for 60 minutes, however you may adjust that time.

.. list-table::
:widths: 25 25 100
:header-rows: 1

* - Argument
- Required
- Description
* - ``bucket``
- Yes
- The S3 bucket to post the file
* - ``aws_access_key``
- No
- Required if ``AWS_ACCESS_KEY_ID`` env variable not set.
* - ``aws_secret_access_key``
- No
- Required if ``AWS_SECRET_ACCESS_KEY`` env variable not set.
* - ``public_url_expires``
- No
- Defaults is 60 minutes.


8 changes: 4 additions & 4 deletions parsons/ngpvan/scores.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,8 @@ def upload_scores(self, tbl, config, url_type, id_type='vanid', email=None, auto
{'score2_id' : int, score2_column': str}]
url_type: str
The cloud file storage to use to post the file. Currently only ``S3``.
The cloud file storage to use to post the file.
See :ref:`Cloud Storage <cloud-storage>` for more details.
email: str
An email address to send job load status updates.
auto_approve: boolean
Expand All @@ -158,9 +159,8 @@ def upload_scores(self, tbl, config, url_type, id_type='vanid', email=None, auto
The deviation from the average scores allowed in order to automatically
approve the score. Maximum of .1.
**url_kwargs: kwargs
Arguments to configure your cloud storage url type.
* S3 requires ``bucket`` argument and, if not stored as env variables
``aws_access_key`` and ``aws_secret_access_key``.
Arguments to configure your cloud storage url type. See
:ref:`Cloud Storage <cloud-storage>` for more details.
`Returns:`
int
The score load job id.
Expand Down

0 comments on commit 8a7cb4a

Please sign in to comment.