Redis authentication failures due to password rotation and persistence encryption

Issue

Redis pods failed authentication after a password rotation, returning WRONGPASS invalid username-password pair errors despite successful updates to the Kubernetes secret. This issue occurred even though the infrastructure components (e.g., CSI driver, Kubernetes secrets, pods) reported healthy status and the password was correctly synced from Azure Key Vault. The root cause was tied to Redis persistence mechanisms, which encrypted existing data with the old password.

Culprit

The failure arose because Redis persistence files (AOF/RDB) were encrypted with the original password. The architecture flow revealed:

1. Initial Deployment: Redis was initialized with an auto-generated password, and persistence files were encrypted using this password.
2. Password Rotation: The password was updated via the CSI driver, syncing the new password from Azure Key Vault to the Kubernetes secret.
3. Pod Restart: Redis attempted to load persisted data (encrypted with the old password) using the new password.
4. Result: Authentication failed because Redis could not decrypt the existing persistence files with the new password.

Redis persistence files store not only data but also the encryption password hash, making them incompatible with password changes without reinitialization.

Solution

Option 1: Data Migration (For Critical Environments)

For environments with critical data that cannot be lost:

1. Connect with the Old Password: Access Redis using the original password to ensure data integrity.
2. Update Password In-Memory: Change the password in Redis’s runtime configuration.
3. Force Rewrite Persistence Files: Trigger Redis to recreate persistence files with the new password.
4. Restart Redis: Ensure the new configuration is applied and data remains accessible.

This approach preserves data integrity while enabling password rotation.

Option 2: PVC Reset (For Non-Critical/ Fresh Deployments)

For non-production or fresh deployments where data loss is acceptable:

1. Force Terminate Redis Pods: Prevent graceful termination to avoid loading old persistence files.
2. Delete the Persistent Volume Claim (PVC): Remove the PVC containing the old, password-encrypted persistence files.
3. Trigger Reconciliation: Use GitOps (e.g., FluxCD) to redeploy Redis with a fresh PVC, ensuring persistence files are encrypted with the new password.

This approach ensures a clean slate, avoiding compatibility issues with old passwords while maintaining security.

Verification

After applying the solution:

• New PVC Creation: Confirm that the new PVC is created with fresh persistence files encrypted with the new password.
• Authentication Enforcement: Verify that Redis enforces authentication and rejects connections without the correct password.
• CSI Sync Confirmation: Ensure that the CSI driver successfully syncs the updated password from the key vault to the Kubernetes secret.

Security Considerations

• No Manual Secret Exposure: Avoid manual handling of secrets to prevent exposure in logs or history.
• Automated Rotation: Leverage automated secrets management (e.g., CSI driver) to rotate passwords securely.
• Managed Identities: Use managed identities or secure authentication methods to access cloud-based key vaults.

This solution ensures a secure and reliable Redis deployment while addressing the root cause of password rotation failures.