Resolved

Kubernetes pods fail with namespace not found and missing codebase after migration

Kubernetes
DevOps
Migration
Debugging
Redis
Docker
Python
CI/CD

Migration deleted all application code without replacements while deployment scripts targeted wrong cluster context | Redis protected-mode blocked connections and email-worker failed due to environment variable mismatches

Working with a Kubernetes-based system using Fast-Agent (Python 3.13) deployed on local K3s, where the migration to FastMCP 2.0 resulted in the deletion of critical application files and deployment scripts incorrectly targeting an Azure cluster instead of the local context. The system included API pods, Redis, PostgreSQL, email workers, and vLLM components.

Issue

The system experienced complete failure with 0/7 pods running after a migration attempt. Key issues included:

  • Namespace not found errors during deployment due to incorrect cluster context usage.
  • Empty codebase in critical directories, including missing Dockerfiles, requirements, and source files.
  • Email-worker pod failures due to environment variable mismatches.
  • API pod crashes caused by Redis protected-mode blocking connections.
  • Build failures due to missing Dockerfiles and source code.

Culprit

1. Codebase Deletion Without Replacement

The migration script inadvertently deleted the entire application codebase, including the Dockerfile, dependencies, and source files. This was due to a migration approach that deleted files without verifying replacements.

2. Cluster Context Mismanagement

Deployment scripts lacked explicit cluster context flags, causing them to default to an Azure cluster instead of the local K3s cluster. This led to namespace and connection errors.

3. Environment Variable Mismatches

The email-worker deployment configuration used incorrect environment variable names, leading to connection and configuration errors.

4. Redis Protected-Mode Blocking Connections

Redis was configured in protected-mode, which blocked all non-localhost connections. The API pods failed to connect due to this configuration.

Solution

1. Codebase Restoration

  • Recovered the Fast-Agent 0.3.15 codebase from Git history.
  • Restored all necessary files, including Dockerfile, requirements, source code, and configuration files.

2. Cluster Context Enforcement

  • Modified all scripts to explicitly use the correct cluster context.
  • Added explicit context flags to ensure deployment scripts targeted the local K3s cluster.

3. Environment Variable Simplification

  • Replaced individual environment variable entries with consolidated envFrom references.
  • This reduced the risk of key mismatches and simplified configuration management.

4. Redis Configuration Fix

  • Temporarily disabled Redis protected-mode for cluster-internal traffic.
  • Implemented network policies to restrict Redis access to internal pods only.

5. Architecture Decisions

  • Postponed FastMCP 2.0 migration to maintain a stable and proven Fast-Agent 0.3.15 codebase.
  • Standardized environment variable management to prevent future key mismatches.
  • Mandated explicit cluster contexts in all deployment scripts to avoid mis-targeting.

Verification: All 7 pods were successfully running without restarts after the fixes were applied.