Resolved

Kubernetes ConfigMap Override Issue Leading to Database Connectivity Failure

Kubernetes
DevOps
Database
Security
Debugging

PostgreSQL connectivity failure due to ConfigMap overriding Secret environment variables in Kubernetes. Debugging steps and architectural fixes.

Issue

A production API service experienced a total loss of PostgreSQL database connectivity, with health checks failing due to an empty DATABASE_URL environment variable. Despite PostgreSQL pods operating normally, the application reported critical status and silent connection failures. This outage impacted all database-dependent operations, though fallback mechanisms ensured basic business continuity.

Culprit

The issue stemmed from an architectural violation: a sensitive database credential (DATABASE_URL) was incorrectly included in both a ConfigMap and a Secret resource. Kubernetes processes envFrom sources in the order they are declared, and the ConfigMap—containing an empty value for DATABASE_URL due to a failed template substitution—overwrote the correct Secret value during environment variable injection. This resulted in the application receiving an empty connection string, causing silent failures.

The empty value originated from an improperly formatted template in the ConfigMap, where a substitution placeholder (${DATABASE_URL}) rendered as an empty string instead of the intended credential.

Solution

1. Credential Separation

The DATABASE_URL credential was removed from the ConfigMap template entirely, ensuring all sensitive data is stored exclusively in Secrets. Non-sensitive configuration (e.g., VLLM_URL) remained in the ConfigMap.

2. Template Fixes

The ConfigMap template was reprocessed using environment variables to ensure proper substitution:

  • Source environment variables from a secure script.
  • Apply the corrected template to Kubernetes using kubectl.

3. Configuration Reload

API pods were restarted to reload the corrected configuration, restoring database connectivity.

4. Validation Enhancements

Pre-deployment checks were added to prevent credential leakage:

  • Static validation: Scan ConfigMap templates for accidental inclusion of sensitive keywords (e.g., PASSWORD, SECRET).
  • Runtime verification: Execute checks on running pods to confirm environment variables do not expose sensitive data.

The fix resolved the outage within 45 minutes, with no impact on business operations. The solution enforces a clear separation between ConfigMaps (non-sensitive configuration) and Secrets (credentials), adhering to Kubernetes best practices.