forked from move-coop/parsons
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Documentation: Cloud Storage Utility (move-coop#350)
* Add docs. * Add links. * Lint.
- Loading branch information
Showing
3 changed files
with
52 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -211,6 +211,7 @@ Indices and tables | |
dbsync | ||
table | ||
notifications | ||
utilities | ||
|
||
.. toctree:: | ||
:maxdepth: 1 | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
********* | ||
Utilities | ||
********* | ||
|
||
.. _cloud-storage: | ||
============= | ||
Cloud Storage | ||
============= | ||
The Parsons cloud storage utility was created to interact with APIs that require access to files | ||
to run an asynchronous process. While this utility currently only works with ``S3``, the goal | ||
is add functionality for additional cloud storage services in the future. | ||
|
||
The cloud storage utility is currently being utilitized primarily by the NGPVAN class | ||
methods such as :func:`~parsons.ngpvan.van.Scores.upload_scores` and Bulk Import methods. | ||
|
||
These methods have arguments specific their method, but all also contain the following cloud | ||
storage arguments: | ||
|
||
* ``url_type`` - The type of cloud storage to utilize. Currently only ``S3``. | ||
|
||
* ``**url_kwargs`` - These are arguments specific to the cloud storage type in order to initialize. | ||
|
||
**S3** | ||
|
||
When you select the ``url_type`` S3, the Parsons table will be converted to a csv and compressed. The file will be posted to an S3 bucket. A presigned public url will be generated and returned. The url will be active by default for 60 minutes, however you may adjust that time. | ||
|
||
.. list-table:: | ||
:widths: 25 25 100 | ||
:header-rows: 1 | ||
|
||
* - Argument | ||
- Required | ||
- Description | ||
* - ``bucket`` | ||
- Yes | ||
- The S3 bucket to post the file | ||
* - ``aws_access_key`` | ||
- No | ||
- Required if ``AWS_ACCESS_KEY_ID`` env variable not set. | ||
* - ``aws_secret_access_key`` | ||
- No | ||
- Required if ``AWS_SECRET_ACCESS_KEY`` env variable not set. | ||
* - ``public_url_expires`` | ||
- No | ||
- Defaults is 60 minutes. | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters