Resolved

Redis pod stuck in ContainerCreating with secret redis-auth not found error

Redis
Kubernetes
Azure
Key Vault
CSI Driver
Secrets Management
Helm
DevOps

Redis Helm chart failed to start due to missing Kubernetes secrets for authentication, resolved by integrating Azure Key Vault CSI Driver for dynamic secret management.

Issue

Redis pods remained stuck in the ContainerCreating state for an extended period due to a missing Kubernetes secret (redis-auth) required for authentication. The Helm chart was configured to use an existing secret, but this secret did not exist in the cluster. This created a dependency cycle where:

  • The pod could not start without the secret.
  • The CSI driver only creates the secret when a pod mounts the volume.
  • Helm releases became corrupted due to prolonged timeouts, resulting in failed deployments.

Culprit

Three interconnected issues caused the problem:

  1. Missing Secret Configuration: The Redis Helm chart relied on an existing Kubernetes secret (redis-auth) that was never created upfront.
  2. CSI Driver Timing Issue: The Azure Key Vault CSI driver creates Kubernetes secrets dynamically only when pods mount the volume, but the pods could not start without the secret.
  3. Helm Release Corruption: Extended ContainerCreating state led to Helm upgrade timeouts, leaving the release in a corrupted state.

Solution

Phase 1: Azure Key Vault CSI Driver Integration

  1. Secure Secret Creation: Created a secure secret in Azure Key Vault to avoid credential exposure.
  2. RBAC Configuration: Assigned the appropriate role to the Kubernetes identity to access the Key Vault.
  3. SecretProviderClass: Configured a SecretProviderClass to bridge Azure Key Vault secrets with Kubernetes pods dynamically, ensuring secrets are only created when needed.

Phase 2: Pod Configuration for CSI Mount

  1. InitContainer: Added an init container to ensure the secret is mounted before the main Redis container starts.
  2. CSI Volume Mount: Configured the pod to use the CSI driver for secret mounting, resolving the dependency cycle.

Phase 3: Helm Release Recovery

  • Cleanup: Removed corrupted secrets and Helm release state to allow a fresh reconciliation.

Security Implementation

  • Ephemeral Secrets: Secrets exist only while the pod is running.
  • No Credential Storage: Used AKS managed identities instead of static credentials.
  • Encryption: All communications between Kubernetes and Key Vault use TLS 1.3.
  • Audit Trail: Maintained a 7-year retention of Key Vault access logs.
  • Least Privilege: Kubelet identity has only necessary permissions (get/list).

Verification

The solution was verified by:

  • Confirming the CSI driver created the required Kubernetes secret.
  • Ensuring the init container completed successfully.
  • Validating the SecretProviderClass configuration.

This approach resolved the secret management issue while adhering to banking-grade security standards and ensuring compliance-ready audit trails.