Docker Build Performance Optimization: Reducing Build Times from Minutes to Seconds

Issue

Docker builds consistently took 3-5 minutes for every deployment due to the use of the --no-cache flag, which forced complete rebuilds. This resulted in redundant downloads of Python packages and unnecessary rebuilding of layers from scratch, even when source code remained unchanged. The inefficiency slowed deployment cycles and wasted infrastructure resources.

Culprit

The primary causes of slow builds included:

1. Forced cache invalidation: The build process defaulted to --no-cache, disregarding existing cache layers.
2. Poor layer separation: The Dockerfile did not optimize layer ordering, leading to frequent cache misses for frequently changing components.
3. Lack of change detection: No mechanism existed to intelligently determine whether source code changes justified a full rebuild.

As a result, the build command docker build --no-cache -t name/api:latest k3s/api/ forced Docker to:

• Redownload all Python dependencies (pip install) every time.
• Rebuild every layer from scratch.
• Ignore cache layers, even for unchanged dependencies.

Solution

Layer Optimization

The Dockerfile was restructured to prioritize optimal layer separation:

• Layer 1: System dependencies (rarely change, e.g., apt-get install for tools like gcc).
• Layer 2: Python dependencies (change occasionally, e.g., pip install -r requirements.txt).
• Layer 3: Source code (changes frequently, e.g., COPY src/ /app/).

This structure minimized cache misses by isolating frequently changing components.

Smart Cache Detection

A git-based change detection mechanism was implemented in the build script:

• If no changes were detected in the source directory (git diff --quiet), the build leveraged Docker cache.
• If changes were detected, the build selectively rebuilt only the necessary layers.

This approach reduced unnecessary rebuilds and improved build performance.

Versioned Image Tagging

Builds were enhanced with fingerprinting for traceability:

• Git hash and build timestamp were incorporated into image tags (e.g., GIT_HASH-build_timestamp).

BuildKit Integration

Docker BuildKit was enabled to leverage improved caching and build performance:

• Used DOCKER_BUILDKIT=1 for optimized cache handling.
• Improved build progress tracking with --progress=plain.

Performance Results:

Scenario	Before (No Cache)	After (Optimized)	Improvement
No source changes	3-5 minutes	5-10 seconds	30x
Source changes	3-5 minutes	30-60 seconds	5x
Dependency changes	3-5 minutes	~3 minutes	Appropriate
Cache hit rate	N/A	95%+	-

The solution maintains full rebuild capability when explicitly required (e.g., via --force flag) while optimizing the common case of incremental changes. This approach ensures efficient use of caching while accommodating necessary rebuilds.