The project addressed a Redis deployment issue on Azure Kubernetes Service (AKS), where the Redis pod failed to start due to a missing authentication secret. The pod remained in a ContainerCreating state with the error FailedMount: secret 'redis-auth' not found, blocking the deployment of Redis using a Helm chart that required an existing secret for authentication.

Issue

The Redis pod could not start because the Helm chart expected an authentication secret named redis-auth containing a key password. This secret was not present in the Kubernetes cluster, despite the use of Azure Key Vault as the secrets backend. The deployment was blocked until this secret was properly provisioned and synced with the cluster.

Culprit

The issue had multiple interconnected causes:

Missing Secret Synchronization: While Azure Key Vault was configured as the secrets backend, no SecretProviderClass was set up to sync the Redis password from Key Vault to Kubernetes.
Manual Secret Temptation: The team initially considered manually creating the secret using kubectl, which would have bypassed the intended Key Vault integration, creating technical debt.
Resource Constraints: Previous CPU allocation issues masked the secret problem, delaying its discovery.

Solution

Recommended Approach: Azure Key Vault CSI Driver This approach leveraged the existing AKS Key Vault CSI driver integration to automatically sync secrets from Key Vault to Kubernetes.

Implementation Steps

Create Secret in Key Vault: A secure password was stored in Azure Key Vault for Redis authentication.
Create SecretProviderClass Manifest: This configuration enabled the CSI driver to sync the Redis password from Key Vault to Kubernetes, creating the required redis-auth secret.
Update HelmRelease: The HelmRelease was configured to reference the newly synced secret, ensuring proper authentication.

Alternative Approach: External Secrets Operator For more complex scenarios requiring secret rotation or multi-cloud support, the External Secrets Operator could be used instead of the CSI driver. This approach would involve installing the operator and configuring it to fetch secrets from Azure Key Vault.

Key Technical Details

CSI Driver: The AKS cluster was configured with the Key Vault CSI driver addon, enabling secure access to Key Vault secrets.
Secret Requirements: The redis-auth secret needed to contain a key named password that matched the Helm chart’s expectation.
Namespace: The secret was created in the same namespace as the Redis deployment (redis-system).
Authentication: The CSI driver used AKS managed identity to access Key Vault, ensuring secure communication.
Auto-sync: Secrets in Kubernetes were automatically updated when changes were made in Key Vault.

Verification

CSI Driver Pods: Verified that CSI driver pods were running correctly.
Secret Creation: Confirmed that the redis-auth secret was successfully created in the Kubernetes cluster.
Pod Status: Ensured that the Redis pod was successfully deployed and operational.