Kubernetes deployment shows old container image despite new build

Issue

The Kubernetes deployment was stuck using an outdated container image (name-api-pythonpath-fixed:latest) while a newer version (566a8e3-20250929-161135) had been built. This prevented the updated components from being deployed. Additionally, the Cloudflared tunnel pod failed, blocking all email webhook traffic and halting end-to-end email processing. Database schema mismatches further complicated the issue, as the users table lacked critical columns (phone, bio, years_experience, etc.), causing silent failures in agent workflows.

Culprit

Deployment Script Flaw

The deployment script used sed to update image tags in manifests but failed to apply these changes to the running deployment, leaving the old image in place. The workflow lacked a critical step to enforce image updates via kubectl set image.

Missing Image Update Command

The absence of explicit commands to update container images meant that even after building new versions, the deployment continued using outdated containers.

Unvalidated Infrastructure

The Cloudflared tunnel was deployed without health checks or automatic restart policies, leading to unnoticed failures. The email processing pipeline was marked as ‘production-ready’ without end-to-end validation, including testing of the email-to-AI-response flow.

Schema Drift

Database migrations for new columns (required by MCP resources) were not applied, causing agent components to fail silently when querying non-existent fields in the users table.

Solution

Deployment Fix

The outdated image was replaced by manually enforcing the correct image using kubectl set image. The deployment script was updated to include explicit image updates in the workflow.

Tunnel Recovery

The Cloudflared tunnel pod was restarted, and a restart policy was applied to ensure automatic recovery from failures.

Validation Workflow

A manual testing process was implemented to verify email processing:

• Test emails were sent to the webhook endpoint.
• Logs were monitored for successful SNS deliveries and AI processing.
• Database queries confirmed the presence of processed email records.

Schema Repair

Missing columns were added to the users table to align with agent requirements. A pre-deployment schema validation step was introduced to prevent future mismatches.

Architectural Improvements

1. Deployment Validation: Added post-deployment checks to verify image updates and container status.
2. Tunnel Monitoring: Implemented health checks and automated monitoring in the CI pipeline.
3. End-to-End Testing: Automated email tests with response validation were added to the deployment workflow.
4. Schema Management: Introduced a migration system to track and apply schema changes, along with pre-deployment validation checks.