Skip to content

Commit

Permalink
New MySQL metadata stores (zenml-io#580)
Browse files Browse the repository at this point in the history 
* first iteration on the metadata stores

* second iteration

* Improved secrets management CLI commands

* allow secret key/values to be passed as command line arguments
* allow secret values to be loaded from files
* enforce secret schema validation during registration and updates

* formatting

* adding a schema for mysql

* registering the new mysql schema on default

* next iteration on the mysql mlmd

* new mysql metadata stores

* docstrings

* Apply suggestions from code review

Co-authored-by: Michael Schuster <[email protected]>
Co-authored-by: Alexej Penner <[email protected]>

* Apply code review suggestions

* Fix zenml secret CLI example in the CLI docs

* returning the secret [ci skip]

* changed the field back to username

* handling the ssl keys [ci skip]

* ignored error for click 8.0.1

* fixed the path for the secrets

* minor formatting

* removed the ignore

* fixing the missing import

* small fixes

* formatting

* Fix username retrieval from secret

* imports formatted

Co-authored-by: Stefan Nica <[email protected]>
Co-authored-by: Michael Schuster <[email protected]>
Co-authored-by: Alexej Penner <[email protected]>
  • Loading branch information
@bcdurak @stefannica
@schustmi @AlexejPenner
4 people authored May 13, 2022
1 parent ccf337f commit 8afa627
Show file tree
Hide file tree
Showing 10 changed files with 582 additions and 275 deletions.
18 changes: 14 additions & 4 deletions docs/book/advanced_guide/secrets.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,16 +58,26 @@ wish to do so interactively, simply passing the secret name in as an argument
(as in the following example) will initiate an interactive process:

```shell
zenml secret register SECRET_NAME
zenml secret register SECRET_NAME -i
```

If you wish to specify a single key-value pair and pass it in as a
non-interactive process, you can type:
If you wish to specify key-value pairs using command line arguments, you can do
so instead:

```shell
zenml secret register SECRET_NAME -k KEY -v VALUE
zenml secret register SECRET_NAME --key1=value1 --key2=value2
```

For secret values that are too big to pass as a command line argument, or have
special characters, you can also use the special `@` syntax to indicate to ZenML
that the value needs to be read from a file:

```bash
zenml secret register SECRET_NAME --attr_from_literal=value \
--attr_from_file=@path/to/file.txt ...
```


## Using Secrets in a Kubeflow environment

ZenML will handle passing secrets down through the various stages of a Kubeflow
Expand Down
104 changes: 44 additions & 60 deletions examples/seldon_deployment/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -359,36 +359,25 @@ save any explicit AWS credentials in the ZenML secret. You just have to set the
as is:

```bash
$ zenml secret register -s seldon_s3 s3-store
You have supplied a secret_set_schema with predefined keys. You can fill these
out sequentially now. Just press ENTER to skip optional secrets that you do not
want to set
Secret value for rclone_config_s3_type:
Secret value for rclone_config_s3_provider:
Secret value for rclone_config_s3_env_auth: True
Secret value for rclone_config_s3_access_key_id:
Secret value for rclone_config_s3_secret_access_key:
Secret value for rclone_config_s3_session_token:
Secret value for rclone_config_s3_region:
Secret value for rclone_config_s3_endpoint:
$ zenml secret register -s seldon_s3 s3-store --rclone_config_s3_env_auth=True
The following secret will be registered.
┏━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┓
┃ SECRET_NAME │ SECRET_KEY │ SECRET_VALUE ┃
┠─────────────┼───────────────────────────┼──────────────┨
┃ seldon_aws │ rclone_config_s3_type │ *** ┃
┃ seldon_aws │ rclone_config_s3_provider │ *** ┃
┃ seldon_aws │ rclone_config_s3_env_auth │ *** ┃
┗━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┛
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┓
┃ SECRET_KEY │ SECRET_VALUE ┃
┠───────────────────────────┼──────────────┨
┃ rclone_config_s3_type │ *** ┃
┃ rclone_config_s3_provider │ *** ┃
┃ rclone_config_s3_env_auth │ *** ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┛

$ zenml secret get s3-store
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
┏━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┓
┃ SECRET_NAME │ SECRET_KEY │ SECRET_VALUE ┃
┠─────────────┼───────────────────────────┼──────────────┨
┃ seldon_aws │ rclone_config_s3_type │ s3 ┃
┃ seldon_aws │ rclone_config_s3_provider │ aws ┃
┃ seldon_aws │ rclone_config_s3_env_auth │ True ┃
┗━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┛
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┓
┃ SECRET_KEY │ SECRET_VALUE ┃
┠───────────────────────────┼──────────────┨
┃ rclone_config_s3_type │ s3 ┃
┃ rclone_config_s3_provider │ aws ┃
┃ rclone_config_s3_env_auth │ True ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┛
```

##### AWS Authentication with Explicit Credentials
Expand All @@ -398,45 +387,40 @@ configure it, you will need to set up credentials explicitly in the ZenML secret
e.g.:

```bash
$ zenml secret register -s seldon_s3 s3-store
You have supplied a secret_set_schema with predefined keys. You can fill these
out sequentially now. Just press ENTER to skip optional secrets that you do not
want to set
Secret value for rclone_config_s3_type:
Secret value for rclone_config_s3_provider:
Secret value for rclone_config_s3_env_auth: False
Secret value for rclone_config_s3_access_key_id: ASAK2NSJVO4HDQC7Z25F
Secret value for rclone_config_s3_secret_access_key: AhkFSfhjj23fSDFfjklsdfj34hkls32SDfscsaf+
Secret value for rclone_config_s3_session_token: AFdfsaSf2SDFfdaWAsfCacs...ASDFsfdfs23sc==
Secret value for rclone_config_s3_region: us-east-1
Secret value for rclone_config_s3_endpoint:
$ zenml secret register -s seldon_s3 s3-store \
--rclone_config_s3_env_auth=False \
--rclone_config_s3_access_key_id='ASAK2NSJVO4HDQC7Z25F' \ --rclone_config_s3_secret_access_key='AhkFSfhjj23fSDFfjklsdfj34hkls32SDfscsaf+' \
--rclone_config_s3_session_token=@./aws_session_token.txt \
--rclone_config_s3_region=us-east-1
Expanding argument value rclone_config_s3_session_token to contents of file ./aws_session_token.txt.
The following secret will be registered.
┏━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┓
┃ SECRET_NAME │ SECRET_KEY │ SECRET_VALUE ┃
┠─────────────┼────────────────────────────────────┼──────────────┨
┃ s3-store │ rclone_config_s3_type │ *** ┃
┃ s3-store │ rclone_config_s3_provider │ *** ┃
┃ s3-store │ rclone_config_s3_env_auth │ *** ┃
┃ s3-store │ rclone_config_s3_access_key_id │ *** ┃
┃ s3-store │ rclone_config_s3_secret_access_key │ *** ┃
┃ s3-store │ rclone_config_s3_session_token │ *** ┃
┃ s3-store │ rclone_config_s3_region │ *** ┃
┗━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┛
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┓
┃ SECRET_KEY │ SECRET_VALUE ┃
┠────────────────────────────────────┼──────────────┨
┃ rclone_config_s3_type │ *** ┃
┃ rclone_config_s3_provider │ *** ┃
┃ rclone_config_s3_env_auth │ *** ┃
┃ rclone_config_s3_access_key_id │ *** ┃
┃ rclone_config_s3_secret_access_key │ *** ┃
┃ rclone_config_s3_session_token │ *** ┃
┃ rclone_config_s3_region │ *** ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┛
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials

$ zenml secret get s3-store
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
┏━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ SECRET_NAME │ SECRET_KEY │ SECRET_VALUE ┃
┠─────────────┼───────────────────────────────┼───────────────────────────────┨
┃ s3-store │ rclone_config_s3_type │ s3 ┃
┃ s3-store │ rclone_config_s3_provider │ aws ┃
┃ s3-store │ rclone_config_s3_env_auth │ False ┃
┃ s3-store │ rclone_config_s3_access_key_… │ ASAK2NSJVO4HDQC7Z25F ┃
┃ s3-store │ rclone_config_s3_secret_acce… │ AhkFSfhjj23fSDFfjklsdfj34hkl… ┃
┃ s3-store │ rclone_config_s3_session_tok… │ AFdfsaSf2SDFfdaWAsfCacssDsfA… ┃
┃ s3-store │ rclone_config_s3_region │ us-east-1 ┃
┗━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ SECRET_KEY │ SECRET_VALUE ┃
┠────────────────────────────────────┼────────────────────────────────────────┨
┃ rclone_config_s3_type │ s3 ┃
┃ rclone_config_s3_provider │ aws ┃
┃ rclone_config_s3_env_auth │ False ┃
┃ rclone_config_s3_access_key_id │ ASAK2NSJVO4HDQC7Z25F ┃
┃ rclone_config_s3_secret_access_key │ AhkFSfhjj23fSDFfjklsdfj34hkls32SDfscs… ┃
┃ rclone_config_s3_session_token │ FwoGZXIvYXdzEG4aDHogqi7YRrJyVJUVfSKpA… ┃
┃ │ ┃
┃ rclone_config_s3_region │ us-east-1 ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```

### 🏃️Run the code
Expand Down
35 changes: 21 additions & 14 deletions src/zenml/cli/__init__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -407,21 +407,31 @@
`aws_session_token`). If you do not have a specific secret type you wish to use,
ZenML will use the `arbitrary` type to store your key-value pairs.
To register a secret, use the `register` command:
To register a secret, use the `register` command and pass the key-value pairs
as command line arguments:
```bash
zenml secret register SECRET_NAME
zenml secret register SECRET_NAME --key1=value1 --key2=value2 --key3=value3 ...
```
If you wish to register a single secret non-interactively, you can
pass in a key-value pair at the command line, as in the following example:
Note that the keys and values will be preserved in your `bash_history` file, so
you may prefer to use the interactive `register` command instead:
```shell
zenml secret register SECRET_NAME -k KEY -v VALUE
zenml secret register SECRET_NAME -i
```
As an alternative to the interactive mode, also useful for values that
are long or contain newline or special characters, you can also use the special
`@` syntax to indicate to ZenML that the value needs to be read from a file:
```bash
zenml secret register SECRET_NAME --schema=aws \
--aws_access_key_id=1234567890 \
--aws_secret_access_key=abcdefghij \
--aws_session_token=@/path/to/token.txt
```
Note that the key and value will be preserved in your `bash_history` file, so
you may prefer to use the interactive `register` command instead.
To list all the secrets available, use the `list` command:
Expand All @@ -438,19 +448,16 @@
To update a secret, use the `update` command:
```bash
zenml secret update SECRET_NAME
zenml secret update SECRET_NAME --key1=value1 --key2=value2 --key3=value3 ...
```
If you wish to update a single secret non-interactively, you can
pass in a key-value pair at the command line, as in the following example:
Note that the keys and values will be preserved in your `bash_history` file, so
you may prefer to use the interactive `update` command instead:
```shell
zenml secret update SECRET_NAME -k KEY -v NEW_VALUE
zenml secret update SECRET_NAME -i
```
Note that the key and value will be preserved in your `bash_history` file, so
you may prefer to use the interactive `update` command instead.
Finally, to delete a secret, use the `delete` command:
```bash
Expand Down
Loading

0 comments on commit 8afa627

Please sign in to comment.