Resolved

Kubernetes pod stuck in Init CrashLoopBackOff due to missing script

Kubernetes
DevOps
Docker
GitOps
FluxCD
Debugging
Azure
PVC

Init container fails with CrashLoopBackOff when trying to execute non-existent script `/app/scripts/index-default-repo.sh` | Pod remains stuck despite successful data indexing in Azure PVC

Issue

A Kubernetes deployment encountered a persistent init container failure, preventing the main container from starting. The init container, responsible for executing a data indexing script, repeatedly crashed due to a missing script file, despite the data being pre-indexed and stored in a persistent volume. This resulted in the pod entering a CrashLoopBackOff state, blocking deployment.

Culprit

The issue stemmed from an obsolete init container configuration that:

  1. Referenced a non-existent script: The Docker image no longer included the script required for data indexing.
  2. Was redundant: Data was pre-indexed locally and deployed via a persistent volume, rendering the init container unnecessary.
  3. Blocked pod startup: Kubernetes requires all init containers to complete successfully before proceeding to the main container.

The init container was originally designed for first-time deployments but became redundant after adopting a pre-indexed data workflow.

Solution

1. Removed the Obsolete Init Container

  • Eliminated the init container configuration entirely from the deployment manifest.
  • Added an explanatory comment to document the removal and rationale.

2. Followed GitOps Best Practices

  • Committed the configuration change to version control.
  • Triggered an automated reconciliation via the GitOps tool to ensure consistency.
  • Forced a pod replacement to apply the updated configuration.

3. Verification Process

  • Confirmed pod status: Verified the pod transitioned to a Running state with all containers ready.
  • Validated data accessibility: Ensured pre-indexed data was correctly mounted and accessible.
  • Checked configuration integrity: Confirmed the absence of init containers and verified the integrity of the data collections.

The solution resolved the deployment blocker while adhering to Kubernetes’ immutable deployment principles, ensuring that configuration changes trigger rolling updates without disrupting workflows.