Resolved

K3s performance bottlenecks with HDD storage and high disk usage

K3s
Kubernetes
DevOps
Performance
Storage
NVMe
PostgreSQL
Migration

K3s cluster suffering from slow pod startups and database performance due to HDD storage limitations | Migration to NVMe storage resolved 12x sequential write bottlenecks and 98% disk capacity issues

Issue

The K3s cluster experienced severe performance degradation due to storage bottlenecks, characterized by slow pod startups, database bottlenecks, high storage capacity utilization, and deployment failures. This impacted containerized workloads such as PostgreSQL, Redis, and API services, leading to degraded system-wide performance and operational inefficiencies.

Culprit

Storage Layer Bottlenecks

The primary bottleneck was the use of HDD-based storage, which severely limited sequential write speeds, random read operations, and database transaction throughput. This resulted in slow performance for containerized workloads and storage capacity crises, with the root partition reaching 98% utilization.

Architectural Constraints

Default K3s paths were configured to HDD-based directories without relocation support, leading to performance issues. Additional architectural constraints included:

  • Containerd Shim Processes: These held mounts during shutdown, requiring forceful termination.
  • Redis Protected Mode: Default configuration blocked API pod connections.
  • Overly Restrictive Network Policies: These prevented necessary inter-pod communication.

Configuration Gaps

Missing configurations for Redis persistence, lack of pre-created namespaces, and non-customized K3s paths contributed to the performance and deployment challenges.

Solution

Storage Migration

The K3s data directories were relocated from the HDD-based /var/lib/rancher/k3s to an NVMe-backed /home partition. This migration resulted in significant performance improvements, including:

  • Sequential Writes: Increased from 196 MB/s to 2,500 MB/s.
  • Database Transaction Throughput: Improved by 45% in PostgreSQL fsync operations.
  • Random Reads: Improved by 17% in operations per second.
  • Mixed Workload Performance: Improved by 41% in overall operations per second.

Configuration Fixes

  1. Redis Persistence: Updated Redis deployment configuration to disable protected mode, ensuring proper connectivity for API pods.
  2. Namespace Pre-Creation: Pre-created necessary namespaces to avoid race conditions in deployment pipelines.
  3. Network Policy Adjustment: Temporarily disabled overly restrictive network policies and planned for re-enablement with pod-specific rules post-testing.

Verification

The migration was verified through:

  • Pod Health Checks: Ensured all components were healthy.
  • Storage Validation: Confirmed adequate free space on the new storage partition.
  • Benchmark Reproduction: Re-ran performance tests to validate improvements in sequential writes, database operations, and random reads.

The post-migration architecture ensured that data and runtime configurations were optimized for NVMe storage, significantly enhancing overall system performance and reliability.