Resolved

CUDA out of memory crashes when running multiple video processing instances

Python
CUDA
GPU
Process Management
DevOps
Video Processing
WhisperX
Memory Management

System freezes with no warning when multiple video processing scripts load WhisperX models simultaneously, exceeding GPU VRAM capacity | Implemented lockfile-based instance control, orphan process cleanup, and sequential processing to prevent GPU OOM crashes

Issue

The system experienced complete freezes and crashes when multiple instances attempted to load resource-intensive models simultaneously, leading to out-of-memory errors and unresponsive system states. This occurred due to overlapping processing attempts that exceeded available hardware limits, causing the system to become completely unresponsive to user input or remote connections.

Culprit

The issue was caused by four key factors:

  1. Lack of Multi-Instance Protection: The system did not prevent multiple instances from running simultaneously, allowing users to accidentally launch duplicate processes that competed for the same resources.

  2. Orphaned Process Accumulation: Failed or crashed processes remained active, holding onto resources and preventing new processes from starting successfully.

  3. Parallel Processing: The system processed multiple videos in parallel without any safeguards, leading to resource contention and system instability.

  4. No Resource Awareness: The system lacked mechanisms to monitor and manage resource usage, particularly in relation to memory and computational limits.

Solution

Instance Control

A locking mechanism was introduced to prevent multiple instances of the system from running simultaneously. This ensures that only one instance can operate at a time, eliminating conflicts and resource contention.

Orphan Process Cleanup

The system now includes automated cleanup procedures that remove any orphaned processes on startup. This ensures that resources are properly released and no residual processes interfere with new operations.

Sequential Processing with Resource Gaps

Processing tasks are now handled sequentially with intentional delays between operations. This allows resources to be fully released and prevents resource exhaustion by limiting concurrent operations.

Pre-Render Validation

Before starting any processing task, the system checks for existing processes that might be using resources. If such processes are detected, they are terminated to ensure a clean environment for the current operation.

Graceful Shutdown Handling

The system now includes signal trapping for clean exits, ensuring that all resources are properly released and no orphaned processes remain after termination. This improves overall system stability and reliability.

Result: The system now operates safely and efficiently, preventing resource exhaustion and ensuring stable performance even during heavy workloads.