Single-node k3s cluster needs lightweight production monitoring stack

Issue

A single-node k3s production environment lacked observability for critical services, including CloudBeaver database inspection tools, FastAPI endpoints, and vLLM inference workloads. Traditional monitoring solutions, such as the kube-prometheus operator, were too resource-intensive for the constrained environment, while manual metrics collection provided insufficient visibility into system health and performance.

Culprit

The primary constraints were:

• Limited resources in a single-node k3s cluster, with only 10Gi of storage and 2Gi of memory allocated for monitoring components.
• High overhead of operator-based solutions, which required managing multiple Custom Resource Definitions (CRDs) and resources, making them impractical for the environment.
• Missing metrics collection for CloudBeaver pods, which lacked built-in Prometheus metrics endpoints.
• Gaps in service discovery, as dynamic services like vLLM and API pods lacked proper annotations for automatic metrics collection.

Solution

To address these challenges, a lightweight monitoring stack was deployed using direct Kubernetes manifests, reducing resource usage by approximately 40% compared to operator-based solutions.

Key Optimizations

1. Resource Allocation: Prometheus was configured with strict memory and CPU limits (2Gi memory and 1000m CPU) to fit within the single-node constraints, ensuring efficient usage of the available resources.
2. Storage Efficiency: Storage retention was set to 30 days, reducing storage usage to 3-5Gi compared to 20Gi+ for default settings.
3. Service Discovery: Annotation-based discovery was implemented, allowing Prometheus to automatically detect and scrape metrics from services marked with prometheus.io/scrape: "true", eliminating the need for manual configuration.
4. Selective Scraping: Only relevant pods were included in the scrape configuration, and system pods were excluded to minimize overhead.

CloudBeaver Monitoring

A dedicated Grafana dashboard was created to track critical metrics for CloudBeaver, including:

• Memory usage to monitor working set memory for CloudBeaver pods.
• CPU usage to track CPU consumption and identify performance bottlenecks.
• Restart alerts to detect excessive pod restarts, which could indicate underlying issues.

Access and Security

Access to Grafana and Prometheus was restricted to local port-forwarding, ensuring no external exposure of services. This approach maintained security while enabling internal monitoring and visualization.

Performance and Outcome

The solution successfully provided full observability for all services while consuming only 15Gi of storage and 3Gi of memory at peak, significantly lower than the 50Gi+ storage and 8Gi+ memory required by operator-based solutions. All monitoring components remained internal to the cluster, ensuring data security and compliance with internal policies.