Resolved

PostgreSQL connection missing in subprocess environment variables

PostgreSQL
Python
Kubernetes
CLI
Database
DevOps
Environment Variables
Backend

CLI tool subprocess fails to access DATABASE_URL causing conversation history feature to break | PostgreSQL connection string not inherited from Kubernetes environment in child processes

Issue

The CLI tool encountered a critical failure when attempting to access database resources during testing, specifically for the conversation history feature. While primary CLI operations, such as message generation, storage, and UI interactions, functioned correctly, any subprocess requiring database access (e.g., retrieving conversation history) resulted in connection errors. This issue surfaced exclusively during CLI testing and did not affect the production email flow, which relied on Kubernetes-managed environment variables.

Culprit

The root cause was an environment variable inheritance failure in the subprocess execution model. When the CLI tool spawned subprocesses, they did not inherit the DATABASE_URL environment variable from the parent Kubernetes pod. This occurred because:

  1. Kubernetes environment variables are not automatically propagated to child processes in subprocess executions.
  2. The subprocess logic in the CLI tool assumed the DATABASE_URL would be available, as it is in the parent environment.
  3. No fallback mechanism was implemented to handle scenarios where the subprocess required direct database access during testing.

Solution

Environment Variable Fallback

A conditional fallback mechanism was introduced to construct the PostgreSQL connection string dynamically when the DATABASE_URL was unavailable. This ensured that subprocesses could still access the database by using a predefined connection string format, leveraging the Kubernetes service DNS (postgres-service) and retrieving the password from an existing environment variable (POSTGRES_PASSWORD).

Testing Workarounds

For immediate CLI testing, two approaches were implemented:

  1. Direct Execution for Single Messages: Commands executed directly via Kubernetes (e.g., kubectl exec) retained access to the parent environment, allowing successful database interactions.
  2. Routing Conversation History Tests: Tests requiring conversation history were directed through the production email flow, where environment variables were properly inherited.

Architectural Considerations

The solution preserved the store-first pattern while introducing resilience for:

  • Environment Variable Propagation: Addressed gaps in Kubernetes environment variable inheritance for subprocesses.
  • Testing Scenarios: Enabled CLI testing with different service discovery requirements.
  • Subprocess Execution: Ensured robustness in subprocess contexts where parent environment variables were unavailable.

This approach maintained system reliability while providing flexibility for testing and production use cases.