Resolved

Resolving K3s Pods Stuck in ImagePullBackOff Due to Storage Layer Mismatch

K3s
Kubernetes
DevOps
Containerization
Debugging
ImagePullBackOff

Kubernetes pods failing to start with ImagePullBackOff status due to storage discrepancies between Docker and K3s containerd. Debugging and solution for local development deployments.

Issue

Kubernetes pods entered a persistent ImagePullBackOff state during deployment, preventing application startup despite successful Docker image builds and valid manifests. This occurred despite:

  • Correctly tagged Docker images.
  • Proper template substitution in Kubernetes manifests.
  • No network connectivity issues to registries.

The pod status consistently showed:

kubectl get pods -n name
# NAME                     READY   STATUS             RESTARTS   AGE
# api-6c4c8955cb-268g4     0/1     ImagePullBackOff   0          3m37s

Culprit

The root cause was a storage layer mismatch between the Docker daemon (where images were built) and K3s’s containerd runtime (where Kubernetes attempted to pull images). A misconfigured imagePullPolicy in the deployment manifest exacerbated the issue:

  • Docker daemon: Built images were stored in Docker’s local storage.
  • K3s containerd: Had no visibility of Docker-built images due to separate storage layers.

The deployment manifest specified:

imagePullPolicy: Never

This forced K3s to rely exclusively on pre-existing local images, which were inaccessible to containerd. No automatic synchronization existed between the storage layers.

Solution

Updated Image Pull Policy

The deployment manifest was modified to allow conditional image pulling:

  • Changed from imagePullPolicy: Never to imagePullPolicy: IfNotPresent.
  • This policy allows K3s to check for locally available images first and pull only if necessary.

Automated Image Import

A post-build script was added to synchronize images between Docker and K3s containerd:

  • Exported Docker images to K3s containerd using docker save and k3s ctr images import.
  • Ensured all required images were available for K3s to use without manual intervention.

Verification Process

  • Image availability check: Confirmed images were accessible in K3s using sudo k3s ctr images list.
  • Deployment monitoring: Used kubectl rollout status deployment/api -n name to track progress.

Impact:

  • Eliminated manual image import requirements.
  • Maintained local development workflows without relying on external registries.
  • Reduced deployment time from 2+ minutes to <30 seconds.
  • Preserved security by avoiding Always pull policies in development environments.