A Kubernetes cluster (K3s) deployed on a mixed storage configuration (high-capacity HDD-based RAID with a cache layer) exhibited significant performance bottlenecks. These bottlenecks manifested as slow sequential writes, degraded PostgreSQL transaction performance, and limited throughput for mixed workloads, hindering system responsiveness and efficiency.

Issue

The storage configuration caused the following performance challenges:

Sequential write speeds were severely limited, leading to slow container image pulls and deployments.
Database transaction latency spiked during write-heavy workloads due to inefficient handling of small, random writes.
Mixed workload throughput was capped, reducing overall system responsiveness.
Storage capacity risks due to high usage (98% full) increased the likelihood of failures during critical operations.

These issues resulted in slow pod startups, high-latency database operations, and degraded system performance during peak loads.

Culprit

The primary causes of the performance bottlenecks were:

Cache inefficiency: The caching layer failed to fully mitigate HDD latency for small, random writes, which are critical for database transactional operations.
RAID overhead: The RAID configuration introduced latency due to parity calculations and stripe synchronization, particularly for mixed workloads.
Filesystem fragmentation: High storage utilization (98% capacity) led to aggressive space reclamation, exacerbating latency spikes during fsync operations.

Benchmark Data

The performance gap between the legacy and new storage configurations was stark:

Sequential writes: 11.7x improvement after migrating to high-performance storage.
Database transaction speeds: 46% faster write operations.
Random reads: 7.6% improvement in read performance.
Mixed workload throughput: 48% faster overall performance.

Solution

To resolve these issues, the cluster was migrated to a high-performance storage configuration (e.g., NVMe-based RAID). This involved:

Migration Process

Fresh installation (recommended approach):
- Created a dedicated directory for K3s data on high-performance storage.
- Installed K3s with a custom data directory to avoid migration complexity.
- Isolated K3s data from legacy storage to ensure optimal performance.
Data migration (alternative approach):
- Stopped K3s services and relocated data to high-performance storage.
- Used symlinks for compatibility during the transition.
- Restarted K3s to validate the new configuration.

Storage Allocation Rationale

The new storage configuration provided:

Sufficient capacity for container images, models, and databases.
Headroom for future growth (8-16x capacity available).
Reduced latency for critical operations (e.g., pod startups, database transactions).

Performance Gains

The migration resulted in significant improvements:

Pod startup time: Reduced from ~5 seconds to ~0.4 seconds (12.5x faster).
Database transaction speeds: 46% faster fsync operations.
Mixed workload throughput: 48% improvement in overall performance.

Validation

After migration, the system was validated to ensure:

Kubernetes node health: All nodes were operational and accessible.
Database performance: PostgreSQL transaction speeds met expectations.
Storage I/O efficiency: Monitoring confirmed improved disk I/O performance.

Key Outcomes

Elimination of storage bottlenecks: High-performance storage resolved write amplification and latency issues.
Improved system responsiveness: Reduced pod startup times and database transaction latency.
Future-proof capacity: Sufficient headroom for future workloads.

This solution transformed the storage infrastructure from a performance bottleneck into a high-speed, reliable system, enabling optimal Kubernetes and database operations.