Deliver Secure Access to Azure Key Vault from AKS Powered by Tanzu

Deliver Secure Access to Azure Key Vault from AKS Powered by Tanzu Overview I’ve been in engaged in several discussions recently related to secrets management for Kubernetes. Particularly in use cases for applications that require credentials to access downstream services (e.g. databases). Sure, Kubernetes has the Secret resource. And for many small or bespoke implementations this may be the initial and perhaps final solution. However, it is common for enterprises to use an Enterprise Secret Store for credentials. And an enterprise may have 100s of k8s clusters. It can be overly burdensome for the credential owners to have to put the credentials in secrets in the right clusters, and maintain those secrets over time. There may be many ways to solve this challenge. However, for a solution to be reasonable it must be presented within a given context. That context defines a set of constraints and resources available to the solution. I’ve enjoyed learning, researching, and exploring this technology space and wanted to share my findings. This blog presents a complete solution that demonstrates different aspects to this challenge. As a Kubernetes platform operator, I want a secure and scalable way to provide my app team tenants access to the credentials they require to access their downstream dependent services (aka database). The key enabling technology is the External Secret Operator, which is a k8s operator that synchronizes external secrets into the k8s cluster. To constrain the solution, I’ve chosen Azure Kubernetes Service and Azure Key Vault, however External Secret Operator works with any k8s distribution and a wide variety of secret stores. I also have available to me VMware’s Tanzu Mission Control and Tanzu Application Platform. I gained a lot of initial learning through this excellent blog Tutorial: How to Set External-Secrets with Azure KeyVault. It would be helpful for you to review this blog prior to continuing because I would consider this an extension of the solution presented in there. I wanted to take it a step further. Tanzu Application Platform packages External Secret Operator, and Tanzu Mission Control delivers OPA Gatekeeper and policy management. So Tanzu can aid in the setup. The blog didn’t necessarily address the topic of External Service Operator’s Multi Tenancy. I also wanted a solution where Azure client secrets were not maintained in my cluster. For this, I adjusted the solution to use Azure AD Workload Identity. Multi Tenancy Requirements Acme offers its app teams shared Kubernetes clusters where tenancy is managed by namespace. All secrets for different services and app combinations are to be managed in single enterprise secret store. Azure Key Vault does not provide RBAC among the secrets within in the vault. However the enterprise secret solution has been built on top of a single Azure Key Vault instance. An automation process has been put in place for the enterprise secret store so that service providers can lifecycle manage the secrets in the secret store. A strict tokenization process allows for the definition of regex based constraints against the key names. Format: ${APP} — ${SERVICE} — ${SECRET_NAME}. Service providers can only LCM secrets for their secret name. App owners should only be able to access secrets that start with their app name. Validation Approach Three app teams, each with their own k8s namespace. app-1: enabled for ExternalSecrets and must start with app-1 — . app-2: enabled for ExternalSecrets and must start with app-2 — . app-3: not enabled for ExternalSecrets. Each app team has one secret in the Enterprise Secret Store. app-1 — postgres972 — connectionstring app-2 — rabbitmq361 — password app-3 — gemfire912 — password Following condition should be tested. app-1 should be able to retrieve app-1 — postgres972 — connectionstring. app-1 should NOT be able to retrieve app-2 — rabbitmq361 — password. app-2 should be able to retrieve app-2 — rabbitmq361 — password. app-3 should NOT be able to retrieve any ExternalSecrets. Solution Implementation The complete series of commands can be found at the supporting git repo akv-external-secret. https://github.com/doddatpivotal/akv-external-secret Azure Setup There is a Single ClusterSecretStore on the k8s cluster to reference the single Azure Key Vault. I need to establish access for a ClusterSecretStore to access Azure Key Vault, but I don’t want to store credentials in the cluster at all, so I leverage the External Secret Operator’s Azure Provider with Azure AD Workload Identity. This requires that I enable oidc issuer capability on the AKS cluster. I need a Azure AD App Client and Service Account to represent k8s cluster access. I need to grant the Azure AD App read access on the Azure Key Vault. Tanzu Mission Control (TMC) Setup How to manage ESO and OPA Gatekeeper in my k8s cluster? How to maintain a library to OPA Policy Templates for my enterprise? How to manage namespaces, policy, access control across all k8s clusters in my enterprise? Use Tanzu Mission Control to manage Kubernetes operations for my cluster. Attach the cluster to my TMC organization. My practice is for each cluster under TMC management, to have a platform-ops namespace. For this I have a platform-ops workspace in TMC, and platform-ops namespaces for each cluster. I want to deploy External Secret Operator to my k8s cluster. External Secret Operator is a package available in the TAP Package Repository. I have access to the TAP Package Repository through TanzuNet and the credentials I have established. Use TMC to Add TanzuNet registry credentials to the platform-ops namespace on the k8s cluster as K8s Secret and export to all other namespaces. Use TMC to Add TAP Package Repository to the k8s cluster. Use TMC to Deploy the External Secret Operator to the k8s clsuter. I want to restrict the ability for k8s users to access External Secrets, unless I explicitly grant the permission to the namespace. And even then, only to the secrets within the Enterprise Secret Store that belong to that app. Use TMC to create a Custom Policy Template that defines these constraints. Use TMC to create a Custom Policy assigned to the k8s cluster based upon this template for any ExternalService resource and specifying the label acme.com/es-regex. My practice is to provision namespaces and manage RBAC for app team use through TMC. For this I have a app-X workspace in TMC, and app-X namespace for applicable clusters. Using TMC Add label acme.com/es-regex=app-1 — to app-1 namespace on my k8s cluster. Using TMC Add label acme.com/es-regex=app-2 — to app-2 namespace on my k8s cluster. Final Workload Identity Setup I need to create a k8s ServiceAccount in the platform-ops namespace with the appropriate annotations and labels to associate it with the Azure AD App Client and Azure Workload Identity. Create an Azure App Federated Credential referring to the k8s ServiceAccount, the k8s cluster OIDC Issuer URL and the Azure AD App. Results The output below validates that policy is correctly restricting which External Secrets can be created in which namespaces. It also shows that ESO has synchronised the ExternalSecret into a k8s Secret. # Condition 1 - Successfully create secret that matches regex for app 1 ns $ ytt -f $PARAMS_YAML -f app-1-good-secret.yaml | kubectl apply -f - externalsecret.external-secrets.io/app-1-good-secret created $ kubectl get externalsecret,secret -n app-1 NAME STORE REFRESH INTERVAL STATUS READY externalsecret.external-secrets.io/app-1-good-secret akv-demo-dpfeffer-eso-vault 1h SecretSynced True NAME TYPE DATA AGE secret/app-1--postgres972--connectionstring Opaque 1 4m18s # Condition 2 - Policy blocks attempt to create secret that does not match $ ytt -f $PARAMS_YAML -f app-1-bad-secret.yaml | kubectl apply -f - Error from server (Forbidden): error when creating "STDIN": admission webhook "validation.gatekeeper.sh" denied the request: [tmc.cp.es-access-by-label] Data key remote ref app-2--rabbitmq361--password not allowed in namespace app-1 # Condition 3- Successfully create secret that matches regex for app 2 ns $ ytt -f $PARAMS_YAML -f app-2-good-secret.yaml | kubectl apply -f - externalsecret.external-secrets.io/app-2-good-secret created $ kubectl get externalsecret,secret -n app-2 NAME STORE REFRESH INTERVAL STATUS READY externalsecret.external-secrets.io/app-2-good-secret akv-demo-dpfeffer-eso-vault 1h SecretSynced True NAME TYPE DATA AGE secret/app-2--rabbitmq361--password Opaque 1 7m14s # Condition 4 - Policy blocks attempt to create secret for ns without regex $ ytt -f $PARAMS_YAML -f app-3-good-secret.yaml | kubectl apply -f - Error from server (Forbidden): error when creating "STDIN": admission webhook "validation.gatekeeper.sh" denied the request: [tmc.cp.es-access-by-label] Namespace app-3 not allowed to receive secrets (missing labels acme.com/es-regex) Conclusion Apps running in Kubernetes naturally like to leverage Kubernetes API and common resource model for processing. Kubernetes Secrets provide this for credentials. The External Service Operator provides an excellent means of synchronizing your external secrets into your Kubernetes clusters so that Apps can access those secrets. Azure AD Workload Identity provides a secure way to bridge the Kubernetes security model (service accounts) with Azure AD Apps. Tanzu provides curated, packaged open source solutions like External Service Operator through is Package Catalogs. Tanzu Mission Control allows you to easily manage namespaces, access control, enterprise service deployment and policies for your clusters.