Resolved

Kubernetes deployments failed due to envsubst compatibility with bash parameter expansion

Kubernetes
DevOps
Bash
CI/CD
Deployment
Template Processing
envsubst
YAML

Kubernetes deployments broke when envsubst processed bash-style `${VAR:-default}` syntax in manifests | Resulted in invalid image tags and deployment failures

In a Kubernetes CI/CD pipeline, deployments failed due to envsubst processing manifests containing bash-style parameter expansion syntax (${VAR:-default}). This caused invalid YAML output, where image tags appeared as empty strings, leading to deployment failures. The issue occurred consistently during automated deployments but worked intermittently with scripts that bypassed template processing.

Issue

Kubernetes deployments failed with errors related to invalid container specifications, where image tags were malformed. The root cause was that envsubst, a POSIX-compliant tool, could not process bash-style parameter expansion syntax in YAML templates. This resulted in partial substitutions, leaving variables like image: name/api: incomplete, which caused deployment failures.

Culprit

The issue stemmed from a toolchain compatibility mismatch:

  1. Bash syntax in templates: Manifests used bash parameter expansion (${VAR:-default}), which envsubst does not recognize.
  2. Missing variable resolution: Deployment scripts did not ensure variables had resolved values before template processing.
  3. Inconsistent deployment paths: Some scripts applied manifests directly without template processing, leading to inconsistent behavior.

envsubst only supports simple variable references ($VAR or ${VAR}) and treats the colon and fallback values as literal text, resulting in partial substitution failures.

Solution

Two-Stage Variable Resolution

A separation of concerns was introduced between bash and envsubst processing:

  1. Bash Stage: Resolved fallback values using bash syntax before envsubst was invoked.
  2. Template Stage: Used POSIX-compatible variable references in YAML templates.

Deployment Script Updates

The deployment pipeline was updated to enforce template processing consistently:

  • All manifests were routed through the envsubst pipeline.
  • Environment variables were resolved in bash before template substitution.

Verification Process

A test command was implemented to validate substitution:

  • Environment variables were exported with fallback values.
  • envsubst was applied to templates, and the output was verified to ensure correct substitution.

This solution ensures consistent behavior across all deployment paths by strictly separating bash logic from template processing, thereby preventing toolchain conflicts.