[EPIC] - Implement Nebari Secret Management

[tylerpotts]

Summary

Currently there is not a robust way of exposing secrets for given users/groups in kuberentes jupyterlab.

Proposed implementation

Using hashicorp vault within the QHub deployment. I see some docs on service accounts and sidecars. Still need to research this implementation further.

[1] https://www.vaultproject.io/docs/platform/k8s/injector
[2] https://www.vaultproject.io/docs/auth/kubernetes

Acceptance Criteria

Tasks to complete

Related to

[viniciusdc]

Vault status update

Kubernetes application pods that rely on Vault to manage their secrets can retrieve them directly via network requests or maintained on a mounted file system through the Vault Injector service via annotations or attached as ephemeral volumes. This approach of employing ephemeral volumes to store secrets is a feature of the Secrets Store extension to the Kubernetes Container Storage Interface (CSI) driver.

Installing Vault:

Vault already provides a very customizable Helm chart for direct installation. There are two ways of providing the necessary configuration:

• Using helm configuration parameters
• Vault also has an interesting override configuration that can be passed as well through override-values.yaml, an example of such configuration can be seen here

Management of secrets:

Vault has two different ways of creating and managing the secrets:

• Sidecard Agent Injector

The sidecar method requires init and/or sidecar containers to retrieve secrets. This is done either by adding pod annotations or using configuration maps defining the Vault role and the path to the secret. It increases the total number of containers running in your cluster but provides a Vault agent to template secrets into configuration files and easily enables rotation of those secrets.

• Vault CSI provider

The CSI method simplifies this architecture since it does not require any sidecar containers. The Vault provider is deployed as a DaemonSet and renders secrets before the pod starts. It also provides a method to sync secrets into environment variables and Kubernetes secrets.
We will need to be sure to have hostPath volumes enabled, as it`s needed to communicate with the CSI driver.

Vault authentication

The primary method of authentication with Vault when using the Vault Agent Injector or the CSI Driver is the service account attached to the pod. For Kubernetes authentication, the service account must be bound to a Vault role and a policy granting access to the secrets desired. When the pod is started it will authenticate with the Kubernetes Auth Method using the service account identity provided in your pod manifest.

• It is not recommended to bind Vault roles to the default service account provided to pods if no service account is defined.

Extras

• There are also other auth methods provided by Vault.
• Vault also has a terraform extension read from, write to, and configure Vault itself.

[viniciusdc]

Installing Vault with Helm and executing using Kubectl

Performing a quick install of Vault can be done by following:

# Add Vault repo to Helm
helm repo add hashicorp https://helm.releases.hashicorp.com

# Add CSI Helm repo
helm repo add secrets-store-csi-driver \
 https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/master/charts

Install the latest version of the Vault Helm chart running in development mode with the injector service disabled and CSI enabled. Adjust server.enabled and injector.enabled if you want to change the deployment 'mode' and the Vault Agent injector instead of the driver:

helm install vault hashicorp/vault \
  --set "erver.dev.enabled=true" \
  --set "injector.enabled=false" \
  --set "csi.enabled=true"

We should be able to inspect the new pods in K9s vault-0 and vault-csi-provider-....

To set a new secret and inspect the generated secret volume you can follow How to set a secret in Vault

With the secret stored in Vault, the authentication configured and role created (below an example from Vault docs):

vault write auth/kubernetes/role/database \
    bound_service_account_names=webapp-sa \
    bound_service_account_namespaces=default \
    policies=internal-app \
    ttl=20m

Proceed to install the vault CSI provider extension

helm install csi secrets-store-csi-driver/secrets-store-csi-driver

and defines the SecretProviderClass.

it is finally time to create a pod that mounts the desired secret:

• Create a service account named webapp-sa: kubectl create serviceaccount webapp-sa
• Define the webapp pod that mounts the secrets volume.
• Create the webapp pod: kubectl apply --filename webapp-pod.yaml
• Display the password secret written to the file system at /mnt/secrets-store/db-password on the webapp pod: kubectl exec webapp -- cat /mnt/secrets-store/db-password

[viniciusdc]

Status report:

• I`ve successful deployed Vault and the CSI driver provider with Terraform
• I was able to enable the dynamic auto-provision of secrets and the secret engine within this deployment

Currently, I'm trying to achieve a programmatic creation of secrets within Vault, I`m able to create the secret directly using:

• The Vault web UI
• by the vault client (SSH into the pod)
• By including it in the manifest generation

Still uncertain of how Vault will be unsealed when deploying the production server and how we expect to include its secrets, we might need to change the deployment logic a bit.

I WIll include a brief report soon

[viniciusdc]

Summarizing the differences between the Sidecar injector and the CSI driver:

• The sidecar injection method requires additional sidecar containers and is set up via pod annotations. With this method, the secret stays within the context of the pod.

• The CSI method does not require sidecar containers and is set up via a CRD, which puts the secret into the context of the node.

In order to CSI driver work, we need to:

• setup the secret that our application will use, beforehand (See comment above for ways of doing that);
• Create a set of policies and a SecretProviderClass resource for each namespace (service account might be needed), some examples of SecretProviderClass;
• Have the application use the CSI driver and the SecretProviderClass you created, you just need to mount the volume at a defined path:

kind: Pod
apiVersion: v1
metadata: ...
spec:
 containers:
 - image: ...
   volumeMounts:
   - name: secrets-store-inline
     mountPath: "/mnt/secrets-store"
     readOnly: true
 serviceAccountName: myapp1-sa # be aware that we might need to provision the same service account to CSI
 volumes:
   - name: secrets-store-inline
     csi:
       driver: secrets-store.csi.k8s.io
       readOnly: true
       volumeAttributes:
         secretProviderClass: "vault-database"

How it all works

When the pod is started it will authenticate with the Kubernetes Auth Method using the service account identity provided in your pod manifest. After successful authentication, the secret paths defined in your SecretProviderClass will be retrieved and written to a tmpfs volume and mounted to your pod.

From Vault/CSI docs:

@costrouc

[viniciusdc]

Due to recent updates on Qhub this was moved to future work and will be added to a future roadmap.

[iameskild]

This is great information! Thanks @viniciusdc!

Today I spent sometime researching another proposed alternative, SOPS. SOPS is a much lighter weight option but doesn't come with many of the features that Vault does.

SOPS appears to used for decryption during the deployment process, ie decrypt the secret using SOPS and then these returned values can be stored as a Kubernetes Secret . There does exist this terraform provider which will simplify that process but it doesn't fundamentally change how or when SOPS is used.

I need to circle back to Vault to see how it compares.

cc @trallard @costrouc

[iameskild]

Today I spent sometime researching and playing with Vault. @viniciusdc provided a lot of great information above but I wanted to summarize a few of my high-level thoughts and findings.

Vault's core features

For our purposes, the Vault server(s) would run on Kubernetes. This server is responsible for managing and decrypting secrets. Vault is also very often configured to take advantage of some of the following features:

• Authentication
  • Authenticate users by outsourcing this to an identify provider such as Keycloak.
• Secret Engine
  • Manages static (key-value) or dynamic (ephemeral) secrets
    • Dynamic keys are those that are generated when they are needed and generated for particular external application or service, reducing the amount of time these secrets are usable for. For instance, if you want to connect to a data source on AWS (regardless where you cluster is hosted), you can use the AWS secret engine which can dynamically generate credentials based on IAM roles and after a specified amount of time, these AWS keys will be automatically revoked.
• Access Control List (ACL)
  • Determine which users have access to which secrets.
    • This solves the request raised in issue [ENH] - Connect to external data sources #1547.
• Storage options
  • The static secrets need to be stored (encrypted) somewhere, options include Consul (Hashicorp), ectd, MySQL, etc.
    • It is generally best practice for these storage services to live outside of the Kubernetes cluster but it is possible to run them on the cluster as well.
• User Interface
  • Vault can be enabled with a slick UI, reducing the potential pain of managing secrets from an administrator's point-of-view.
    • We could add an additional link to the main Nebari nav bar "Secret Management".

Impressions

I have to say I'm quite impressed with what I've seen so far. This is obviously a heavyweight application but makes up for it with a lot of awesome features.

To me it feels like Vault is to secret management as Keycloak is to identity and access management.

That said, this is definitely going to be a heavier lift in terms of development compared with SOPS. But, if done correctly, this will not only harden Nebari from a security perspective but it will also enable greater control over access to external data sources.

[costrouc]

@iameskild my opinion has changed on this. I certainly think that vault is extremely useful but will require a lot of effort to deploy properly and most importantly will only solve the "how to expose secrets to users" not "how to use secrets with a deployment". SOPS has the opportunity to solve both and will be significantly easier to implement. I'm not saying that we never add Vault support but SOPS will be enough to solve the current issues that users have raised with us so I'm in favor of us implementing SOPS.

[viniciusdc]

I do agree with @costrouc as well. While I still think Vault could be much more accessible for the user to manage (and we could try exposing the secrets using a mix of CSI drives with user-specific mounts), it will be undeniably a pain to integrate with all the Nebari inner services (more refactoring than anything else). Still, the main problem I can foresee with Vault right now is that it will become another "Keycloak" case where we integrate the feature, but we will not be using it to its fullest (while also taking lots of releases to make it stable).

So, I am also in favor of implementing SOPS (mainly because it has a terraform provider as well XD)

[costrouc]

This issue is currently being discussed in two RFDs.

[viniciusdc]

Another options that arised recently, and seems really promissing as we would be able to use keycloaks IAM system to mange user permissions to secrets:
https://github.com/Infisical/infisical

They are working on providing support to OIDC, but if they do implement it, could be a good solution for the problems above, as there will be an UI to manage all secrets (kuberntes included) and insert them into user pods/services