K3s deployment script fails due to undefined path variables and missing component fallback

Issue

A Kubernetes deployment script encountered three critical failures during execution:

1. Undefined variable error: The script crashed due to an undefined environment variable (TEMP_DIR) used for locating configuration files, preventing the application of API-related configurations.
2. Component omission: The script entirely skipped the deployment of a critical component (vLLM) when no GPU was detected, despite the component supporting CPU-based execution.
3. Manual intervention required: The deployment left API replicas at zero, necessitating manual scaling and creating inconsistencies where only database-related services (PostgreSQL and Redis) were operational.

Culprit

Undefined Variable Crash

The script referenced an undeclared environment variable (TEMP_DIR) for locating configuration files. Although the correct variable (MANIFESTS_DIR) was exported earlier in the script, the incorrect reference caused path resolution failures.

All-or-Nothing Component Logic

The GPU detection logic was overly restrictive, skipping the entire vLLM deployment if no GPU was detected. This design assumption overlooked the component’s CPU compatibility, leading to partial deployments.

Missing Readiness Checks

The script lacked proper dependency verification, allowing subsequent operations to proceed before critical components (e.g., API) were fully ready. This created race conditions and deployment inconsistencies.

Solution

Path Resolution Fix

Replaced the undefined variable (TEMP_DIR) with the correctly exported variable (MANIFESTS_DIR) to ensure configuration files were accessible during deployment.

Graceful Component Fallback

Modified the GPU detection logic to deploy vLLM in all cases, with conditional GPU optimization. This ensures the component is deployed regardless of hardware availability while optimizing performance when GPUs are present.

Readiness Verification

Added explicit wait conditions for critical components, including timeouts to ensure dependencies were ready before proceeding. This prevents race conditions and ensures deployment consistency.

Enhanced Status Reporting

Implemented post-deployment verification steps to validate system health and provide observable feedback. This includes checking pod statuses and confirming operational readiness.

The changes reduce technical debt by:

1. Eliminating undefined variable references and ensuring proper path resolution.
2. Implementing fallback mechanisms for hardware-dependent components.
3. Adding explicit dependency checks and observable deployment status.
4. Maintaining existing container architecture while fixing deployment gaps.