API pods entered a CrashLoopBackOff state after server restarts due to unexpanded environment variable placeholders in ConfigMaps. Specifically, the pods failed with an error related to unresolved template strings like ${REDIS_URL}, which were treated as literal values instead of resolved configuration entries.

Issue

API pods crashed immediately after startup due to unresolved ${VARIABLE} placeholders in ConfigMaps, e.g., REDIS_URL, ENVIRONMENT, and MCP_URL were empty strings.
The error manifested as limits.errors.ConfigurationError: unknown storage scheme : ${REDIS_URL}, preventing pod initialization.

Culprit

The root cause was an incomplete set of environment variable exports in the centralized script (common.sh), which only exported a subset of required variables:

Missing exports: REDIS_URL, API_PORT, ENVIRONMENT, VLLM_URL, MCP_URL, and SES_REPLY_TO were not included in the export list.
Undefined variables: When the ConfigMap template was processed by envsubst, undefined variables resolved to empty strings, breaking the configuration.
Restart dependency: Previous deployments worked because variables were present in the shell session, but restarts cleared this state, exposing the missing exports.

Solution

Centralized Environment Exports

Updated common.sh: Added exports for all required variables, ensuring all necessary configurations were available for envsubst processing.
Validation process: Verified that ConfigMaps contained resolved values (e.g., REDIS_URL: redis://redis-service:6379, ENVIRONMENT: production).

Prevention Strategy

Checklist for new variables: Ensure all variables are added to both the .env file and common.sh exports, and validated after a clean restart.
Template safety: Introduced a verification step to ensure substitutions are resolved before applying the ConfigMap:

envsubst < api-configmap.yaml | kubectl apply -f -


This resolved the issue, allowing API pods to restart successfully and ensuring all environment variables were properly resolved in the ConfigMaps.