Resolved

Kubernetes pods show empty environment variables despite correct config

Kubernetes
DevOps
Environment Variables
CI/CD
Debugging
Configuration
Secrets
Deployment

Critical environment variables appear empty in running Kubernetes pods even when properly set in .env files and configmaps/secrets | Causes complete system failures including database disconnections and service unavailability

Working with Kubernetes deployments where environment variables are managed through .env files, configmaps, and secrets during CI/CD pipelines. The system relies on critical service URLs (database, vLLM, MCP) being properly injected into pod environments for full functionality.

Issue

Kubernetes pods exhibited empty environment variables for critical service URLs despite these values being correctly defined in the .env file and deployment configurations. The system manifested complete failure with:

  • Database connection errors.
  • vLLM integration failures.
  • MCP tool unavailability.
  • Email response pipeline returning ‘critical’ health status.

Culprit

The root cause was a faulty secret generation script that failed to properly load environment variables from the .env file during deployment. While the variables existed in the source .env file, they were being stored as empty strings in the resulting Kubernetes secrets, causing pods to receive empty values despite correct configurations.

Solution

Secret Regeneration

  • Reran the secret generation process to properly load values from the .env file.

Deployment Restart

  • Triggered a rolling restart of the affected deployments to pick up the corrected environment variables.

Verification

  • Confirmed proper variable injection by inspecting environment variables within the pod.

The solution restored all services to operational status, ensuring proper database connectivity, vLLM integration, and MCP tool availability. Preventive measures were implemented to verify environment variables immediately after deployment.