Fixing Kubernetes Pods Losing Environment Variables After Deployment

Issue

Kubernetes pods consistently lost AWS-related environment variables (AWS_REGION, AWS_ACCESS_KEY_ID, S3_EMAIL_BUCKET) after pipeline deployments, resulting in complete failures for:

• S3 bucket operations (errors like Invalid endpoint: https://example.com).
• SES email processing.
• SNS webhook handling.
• Database email record creation.

The issue followed a daily pattern where manual fixes temporarily restored functionality until the next pipeline run. Inspection of containers revealed empty environment variables despite correctly configured .env files.

Culprit

The root cause was in the deploy-versioned.sh script, where environment variables were loaded but not exported to subprocesses. Specifically:

• source "${PROJECT_ROOT}/.env" loaded variables into the current shell process only.
• Without export, these variables were invisible to child processes.
• generate-secrets.sh (responsible for creating Kubernetes secrets) ran as a subprocess.
• The subprocess received empty variables, creating empty Kubernetes secrets.
• Containers inherited these empty secrets as environment variables.

Solution

Standardized Environment Variable Export

A standardized approach was implemented in common.sh to ensure environment variables were properly exported to subprocesses:

1. Created load_env_vars() function in common.sh that:

   • Loads variables from the .env file.
   • Exports them to the environment for use in subprocesses.

   load_env_vars() {
     while IFS='=' read -r key value; do
       export "$key=$value"  # Critical export statement
     done < "${PROJECT_ROOT}/.env"
   }

2. Updated the deployment script to use the standardized approach:

   source "${SCRIPT_DIR}/common.sh"
   load_env_vars  # Both loads AND exports variables
   "${SCRIPT_DIR}/generate-secrets.sh"  # Now inherits all variables

Verification Process

Verification steps were added to ensure AWS environment variables were correctly propagated:

• Kubernetes Secret Validation:

  kubectl get secret api-secret -n name -o jsonpath='{.data.AWS_REGION}' | base64 -d
  # Expected Output: eu-north-1 (previously returned empty)

• CI/CD Pipeline Integration: Added secret validation to the pipeline to ensure environment variables were correctly deployed.

Architecture Impact

• Pipeline Flow: pipeline.sh → build-images-versioned.sh → deploy-versioned.sh → load_env_vars() → generate-secrets.sh → Kubernetes.
• Process Isolation: Proper parent-child process variable inheritance.
• Consistency: Standardized environment loading across all deployment scripts.

Results:

• ✅ AWS endpoints properly formed (https://example.com).
• ✅ Email processing restored (SNS + SES + S3).
• ✅ Eliminated the need for daily manual fixes.
• ✅ Container startup logs show clean AWS client initialization.