Redis StatefulSet deployment failure due to incorrect storage class configuration

Issue

Redis StatefulSet pods were stuck in Pending state for over 55 minutes due to persistent volume claim (PVC) provisioning failures. The error indicated that the storage class standard (common in GKE/EKS) was not available in the current environment. This caused:

• A complete Redis service outage.
• Blocked dependent applications relying on Redis.
• Failed GitOps reconciliation processes.
• Infrastructure health checks reporting critical status.

Culprit

The issue originated from a misconfigured HelmRelease for Redis, where the storage class standard was hardcoded. This class is specific to Google Kubernetes Engine (GKE) and Amazon Elastic Kubernetes Service (EKS) but does not exist in Azure Kubernetes Service (AKS). The configuration likely originated from generic Kubernetes documentation or default Helm chart values not tailored to the cloud provider’s offerings.

Solution

Storage Class Selection

The standard storage class was replaced with the appropriate AKS storage class (managed), which provides:

• Cost-effective HDD-backed storage suitable for development/test environments.
• Optimal binding mode (WaitForFirstConsumer) for co-located storage and efficient pod scheduling.
• CSI driver support for modern Azure Disk integration.

Resolution Workflow

1. Diagnosis: Identified the missing storage class and its impact on PVC provisioning.
2. Configuration Update: Modified the Helm chart to reference the correct storage class (managed).
3. Validation: Monitored PVC and pod transitions to Bound and Running states, respectively.
4. Verification: Confirmed Redis functionality post-deployment using Redis CLI commands.

Storage Class Strategy

A standardized approach was implemented:

• Redis (Dev/Test): managed (HDD-backed, cost-effective).
• Redis (Production): managed-premium (SSD-backed, performance-critical).
• Other Services: Tailored storage classes based on performance and cost requirements.

Lessons Implemented

1. Environment-Specific Configurations: Ensured storage class selections align with the target cloud provider’s offerings.
2. Debugging Checklist: Added diagnostic commands for quick troubleshooting of storage-related issues.
3. Documentation Updates: Clarified storage class strategies for Redis, Prometheus, and logging services.

This fix resolved the outage and ensured future deployments account for cloud-specific storage configurations.