fix: Updating akv cert provider to use getSecret (ratify-project#957)

Signed-off-by: Susan Shi <[email protected]>
binbin-li · Aug 15, 2023 · dcfa9b5 · dcfa9b5
1 parent 1051142
commit dcfa9b5
Show file tree

Hide file tree

Showing 12 changed files with 309 additions and 205 deletions.
diff --git a/.github/workflows/build-pr.yml b/.github/workflows/build-pr.yml
@@ -163,3 +163,30 @@ jobs:
           name: e2e-logs
           path: |
             logs-*.json
+
+  test-cleanup:
+    env:
+      AZURE_SUBSCRIPTION_ID: daae1e1a-63dc-454f-825d-b39289070f79
+      AZURE_CLIENT_ID: 814e6e97-120c-4534-b8a9-f1645bc99500
+      AZURE_TENANT_ID: 72f988bf-86f1-41af-91ab-2d7cd011db47
+    needs: ['build_test_aks_e2e']
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write
+      contents: read
+    steps:
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
+      - name: Set up Go 1.20
+        uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1
+        with:
+          go-version: '1.20'
+
+      - name: Az CLI login
+        uses: azure/login@92a5484dfaf04ca78a94597f4f19fea633851fa2 # v1.4.7
+        with:
+          creds: '{"clientId":"${{ env.AZURE_CLIENT_ID }}","clientSecret":"${{ secrets.AZURE_CLIENT_SECRET }}","subscriptionId":"${{ env.AZURE_SUBSCRIPTION_ID }}","tenantId":"${{ env.AZURE_TENANT_ID }}"}'
+
+      - name: clean up
+        run: |
+          make e2e-cleanup AZURE_SUBSCRIPTION_ID=${{ env.AZURE_SUBSCRIPTION_ID }}
diff --git a/Makefile b/Makefile
@@ -548,6 +548,8 @@ e2e-helm-deploy-ratify-replica: e2e-helm-deploy-redis e2e-notation-setup e2e-bui
 e2e-aks:
 	./scripts/azure-ci-test.sh ${KUBERNETES_VERSION} ${GATEKEEPER_VERSION} ${TENANT_ID} ${GATEKEEPER_NAMESPACE} ${CERT_DIR}
 
+e2e-cleanup:
+	./scripts/azure-ci-test-cleanup.sh ${AZURE_SUBSCRIPTION_ID}
 ##@ Development
 
 .PHONY: manifests

diff --git a/docs/quickstarts/ratify-on-azure.md b/docs/quickstarts/ratify-on-azure.md
@@ -108,28 +108,22 @@ Configure user-assigned managed identity and enable `AcrPull` role to the worklo
 
 ## Configure access policy for AKV 
 
-> Note: Azure Key Vault integration currently only support self signed certificate, if you are working with a certificate chain, please specify the public root certificate value inline using the [inline certificate provider](../reference/crds/certificate-stores.md#inline-certificate-provider).
-
 1. Set the environmental variable for Azure Key Vault URI.
 
     ```bash
     export VAULT_URI=$(az keyvault show --name ${AKV_NAME} --resource-group ${GROUP_NAME} --query "properties.vaultUri" -otsv)
     ```
 
-2. Import your own private key and certificates. You can import it on the portal as well.
+2. Ratify requires secret permissions to retrieve the public certificates for the entire certificate chain,
+ please set private keys to Non-exportable at certificate creation time to avoid security risk. Learn more about non-exportable keys [here](https://learn.microsoft.com/en-us/azure/key-vault/certificates/how-to-export-certificate?tabs=azure-cli#exportable-and-non-exportable-keys)
+
+> Note: If you were unable to configure certificate policy, please consider specifying the public root certificate value inline using the [inline certificate provider](../reference/crds/certificate-stores.md#inline-certificate-provider) to reduce risk of exposing private key.
 
-    ```bash
-    az keyvault certificate import \
-    --vault-name ${AKV_NAME} \
-    -n ${KEY_NAME} \
-    -f ${CERT_PATH}
-    ```
-
-3. Configure policy for user-assigned managed identity:
+   Configure policy for user-assigned managed identity:
 
     ```bash
     az keyvault set-policy --name ${AKV_NAME} \
-    --certificate-permissions get \
+    --secret-permissions get \
     --object-id ${IDENTITY_OBJECT_ID}
     ```
 

diff --git a/docs/reference/crds/certificate-stores.md b/docs/reference/crds/certificate-stores.md
@@ -19,7 +19,7 @@ status: # supported in version >= config.ratify.deislabs.io/v1beta1
 
 # Certificate Store Provider
 ## AzureKeyVault Certificate Provider
-See notation integration example [here](../../developer/verifier.md#section-6-built-in-verifiers)
+See notation integration example [here](../../reference/verifier.md#section-6-built-in-verifiers)
 ```yml
 apiVersion: config.ratify.deislabs.io/v1beta1
 kind: CertificateStore
@@ -57,10 +57,14 @@ status:
 Use command `kubectl get certificatestores.config.ratify.deislabs.io` to see a overview of `certificatestores` status.
 Use command `kubectl get certificatestores.config.ratify.deislabs.io/certstore-akv` to see full details on each certificate.
 ### Limitation
-Azure Key Vault integration currently only works for self signed certificate, we are following up on Azure Key Vault specific limitations so we could support certificate chains in the future, please use [issue 695](https://github.com/deislabs/ratify/issues/695) for tracking. If you are working with a certificate chain, please specify the public root certificate value inline using the [inline certificate provider](certificate-stores.md#inline-certificate-provider). 
+Azure keyvault Certificates are built on top of keys and secrets. When a certificate is created, an addressable key and secret are also created with the same name. Ratify requires secret permissions to retrieve the public certificates for the entire certificate chain, please set private keys to Non-exportable at certificate creation time to avoid security risk. Learn more about non-exportable keys [here](https://learn.microsoft.com/en-us/azure/key-vault/certificates/how-to-export-certificate?tabs=azure-cli#exportable-and-non-exportable-keys)
 
 Please also ensure the certificate is in PEM format, PKCS12 format with nonexportable private keys can not be parsed due to limitation of Golang certificate library.
 
+Akv set up guide in ratify-on-azure [quick start](https://github.com/deislabs/ratify/blob/main/docs/quickstarts/ratify-on-azure.md#configure-access-policy-for-akv).
+
+> Note: If you were unable to configure certificate policy, please consider specifying the public root certificate value inline using the [inline certificate provider](../reference/crds/certificate-stores.md#inline-certificate-provider) to reduce risk of exposing private key.
+
 ## Inline Certificate Provider
 ```
 apiVersion: config.ratify.deislabs.io/v1beta1

diff --git a/docs/reference/oras-auth-provider.md b/docs/reference/oras-auth-provider.md
@@ -81,149 +81,11 @@ Both the above modes uses a k8s secret of type ```dockerconfigjson``` that is de
 
 ### 2. Azure Workload Identity
 
-Ratify pulls artifacts from a private Azure Container Registry using Workload Federated Identity in an Azure Kubernetes Service cluster. For an overview on how workload identity operates in Azure, refer to the [documentation](https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation). You can use workload identity federation to configure an Azure AD app registration or user-assgined managed identity. The following workflow will include steps of both.
-
-#### User steps to set up Workload Identity with AKS, ACR and AKV:
+Ratify pulls artifacts from a private Azure Container Registry using Workload Federated Identity in an Azure Kubernetes Service cluster. For an overview on how workload identity operates in Azure, refer to the [documentation](https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation). You can use workload identity federation to configure an Azure AD app registration or user-assgined managed identity. 
 
+Please refer to [quick start](https://github.com/deislabs/ratify/blob/main/docs/quickstarts/ratify-on-azure.md#configure-workload-identity-for-acr) to configure workload identity for ACR.
 The official steps for setting up Workload Identity on AKS can be found [here](https://azure.github.io/azure-workload-identity/docs/quick-start.html).  
 
-1. Configure environment variables.
-```shell
-export IDENTITY_NAME=<Identity Name>
-export GROUP_NAME=<Azure Resource Group Name>
-export SUBSCRIPTION_ID=<Azure Subscription ID>
-export TENANT_ID=<Azure Tenant ID>
-export ACR_NAME=<Azure Container Registry Name>
-export AKS_NAME=<Azure Kubernetes Service Name>
-export KEYVAULT_NAME=<Azure Key Vault Name>
-export RATIFY_NAMESPACE=<Namespace where Ratify deployed, defaults to "gatekeeper-system">
-export NOTATION_PEM_NAME=<Name of the certificate file uploaded to Azure Key Vault>
-```
-
-2. Create a Workload Federated Identity.
-
-Create ACR AAD application:
-```shell
-az ad sp create-for-rbac --name ${IDENTITY_NAME}
-
-export IDENTITY_CLIENT_ID="$(az ad sp list --display-name "${IDENTITY_NAME}" --query '[0].appId' -otsv)"
-export IDENTITY_OBJECT_ID="$(az ad app show --id ${IDENTITY_CLIENT_ID} --query id -otsv)"
-```
-
-Or create user-assigned managed identity:
-```shell
-az identity create --name "${IDENTITY_NAME}" --resource-group "${GROUP_NAME}" --location "${LOCATION}" --subscription "${SUBSCRIPTION_ID}"
-
-export IDENTITY_OBJECT_ID="$(az identity show --name "${IDENTITY_NAME}" --resource-group "${GROUP_NAME}" --query 'principalId' -otsv)"
-export IDENTITY_CLIENT_ID=$(az identity show --name ${IDENTITY_NAME} --resource-group ${GROUP_NAME} --query 'clientId' -o tsv)
-```
-3. Create ACR and enable acrpull role to the workload identity.
-```shell
-az acr create --name "${ACR_NAME}" \
-  --resource-group "${GROUP_NAME}" \
-  --sku Standard
-
-# AAD application:
-az role assignment create \
-  --assignee ${IDENTITY_CLIENT_ID} \
-  --role acrpull \
-  --scope subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${GROUP_NAME}/providers/Microsoft.ContainerRegistry/registries/${ACR_NAME}
-
-# User-assigend managed identity:
-az role assignment create \
-  --assignee-object-id ${IDENTITY_OBJECT_ID} \
-  --role acrpull \
-  --scope subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${GROUP_NAME}/providers/Microsoft.ContainerRegistry/registries/${ACR_NAME}
-```
-
-4. Create OIDC enabled AKS cluster by following the [doc](https://learn.microsoft.com/en-us/azure/aks/cluster-configuration#oidc-issuer) or below steps:
-```shell
-# Install the aks-preview extension
-az extension add --name aks-preview
-
-# Register the 'EnableWorkloadIdentityPreview' feature flag
-az feature register --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview"
-az provider register --namespace Microsoft.ContainerService
-
-az aks create \
-    --resource-group "${GROUP_NAME}" \
-    --name "${AKS_NAME}" \
-    --node-vm-size Standard_DS3_v2 \
-    --node-count 1 \
-    --generate-ssh-keys \
-    --enable-workload-identity \
-    --attach-acr ${ACR_NAME} \
-    --enable-oidc-issuer
-
-# Connect to the AKS cluster:
-az aks get-credentials --resource-group ${GROUP_NAME} --name ${AKS_NAME}
-
-export AKS_OIDC_ISSUER="$(az aks show -n ${AKS_NAME} -g ${GROUP_NAME} --query "oidcIssuerProfile.issuerUrl" -otsv)"
-```
-
-5. [optional] Install Mutating Admission Webhook onto AKS cluster by following steps [here](https://azure.github.io/azure-workload-identity/docs/installation/mutating-admission-webhook.html) if you didn't add the `--enable-workload-identity` option while creating the AKS cluster.
-
-6. Establish federated identity credential. On AZ CLI `${RATIFY_NAMESPACE}` is where you deploy Ratify:
-
-Using AAD application:
-```shell
-cat <<EOF > body.json
-{
-  "name": "kubernetes-federated-credential",
-  "issuer": "${AKS_OIDC_ISSUER}",
-  "subject": "system:serviceaccount:${RATIFY_NAMESPACE}:ratify-admin",
-  "description": "Kubernetes service account federated credential",
-  "audiences": [
-    "api://AzureADTokenExchange"
-  ]
-}
-EOF
-
-az rest --method POST --uri "https://graph.microsoft.com/beta/applications/${APPLICATION_OBJECT_ID}/federatedIdentityCredentials" --body @body.json
-```
-
-Using user-assigned managed identity:
-```shell
-# Establish feerated identity credentials between the managed identity, the service account issuer and the subject.
-az identity federated-credential create \
-  --name ratify-federated-credential \
-  --identity-name "${IDENTITY_NAME}" \
-  --resource-group "${GROUP_NAME}" \
-  --issuer "${AKS_OIDC_ISSUER}" \
-  --subject system:serviceaccount:"${RATIFY_NAMESPACE}":"ratify-admin"
-```
-
-7. Create an Azure Key Vault and set an access policy.
-```shell
-az keyvault create \
-  --resource-group ${GROUP_NAME} \
-  --name ${KEYVAULT_NAME}
-
-export VAULT_URI=$(az keyvault show --name ${KEYVAULT_NAME} --resource-group ${GROUP_NAME} --query "properties.vaultUri" -otsv)
-
-# Import your own private key and certificates. You can import it on the portal as well.
-az keyvault certificate import \
-  --vault-name ${KEYVAULT_NAME} \
-  -n <Certificate Name> \
-  -f /path/to/certificate
-
-# Grant permission to access the certificate.
-az keyvault set-policy --name ${KEYVAULT_NAME} \
-  --certificate-permissions get \
-  --object-id ${IDENTITY_OBJECT_ID}
-```
-
-8. Deploy from local helm chart. Follow the [Quick Start](https://github.com/deislabs/ratify/blob/main/README.md#quick-start) to deploy Gatekeeper and Ratify.
-
-Notes: add below options while installing Ratify
-```shell
---set azureWorkloadIdentity.clientId=${IDENTITY_CLIENT_ID} \
---set akvCertConfig.enabled=true \
---set akvCertConfig.vaultURI=${VAULT_URI} \
---set akvCertConfig.cert1Name=${NOTATION_PEM_NAME} \
---set akvCertConfig.tenantId=${TENANT_ID}
-```
-
 ### 3. Kubernetes Secrets
 
 Ratify resolves registry credentials from [Docker Config Kubernetes secrets](https://kubernetes.io/docs/concepts/configuration/secret/#docker-config-secrets) in the cluster. Ratify considers kubernetes secrets in two ways: