Resolved

Python video rendering crashes with OOM despite memory monitoring

Python
Performance
Debugging
DevOps
Memory Management
FFmpeg
Multiprocessing
psutil

Video generation pipeline crashes with kernel OOM killer despite existing ResourceMonitor, with no logs identifying which phase (TTS, WhisperX, rendering) causes memory leaks | Solution implements persistent daemon monitoring with phase detection and enhanced process termination

Issue

A Python-based video generation pipeline using FFmpeg, WhisperX, and TTS components frequently crashed due to out-of-memory (OOM) errors, particularly during frame rendering (~13000 frames). Despite the presence of a resource monitor in the rendering phase, the system lacked comprehensive monitoring across all six pipeline phases (TTS generation, WhisperX alignment, rendering, encoding, upload, and idle periods). The existing monitor only operated during rendering with 30-second intervals, failing to detect rapid 20GB+ memory spikes. Additionally, orphaned FFmpeg and WhisperX processes continued consuming memory after the parent Python process terminated, exacerbating the issue.

Culprit

Monitoring Gaps

The original resource monitoring system only covered the rendering phase, leaving five other critical phases unmonitored:

  • TTS Generation: Consumed 3-6GB per video.
  • WhisperX Alignment: Involved heavy memory usage.
  • Idle Periods: Allowed memory to accumulate without intervention.
  • FFmpeg Encoding: Partially monitored but insufficient for early detection.
  • Upload Phase: Completely unmonitored.

Ineffective Process Termination

The OOM handler used os._exit(1), which only terminated the current Python process while leaving orphaned FFmpeg and WhisperX subprocesses running. These orphaned processes continued consuming memory, leading to eventual kernel OOM killer intervention.

Configuration Flaws

The monitoring system was configured with:

  • 30-second sampling intervals: Too slow to detect rapid memory spikes.
  • 100GB threshold: Allowed excessive memory consumption before intervention.

Solution

Persistent System Monitor Daemon

A dedicated daemon process was introduced to continuously monitor system resources and provide comprehensive phase-specific diagnostics. Key features include:

  • Rotating log buffer: Maintains a 10,000-line log to track memory usage trends and detect anomalies.
  • 5-second sampling interval: Enables early detection of rapid memory spikes.
  • Phase detection: Identifies which pipeline phase is causing memory leaks by scanning processes and monitoring file activity.
  • Independent operation: Runs as a separate process alongside the main pipeline, ensuring continuous monitoring.

Enhanced Phase Detection

The system now tracks six distinct phases:

  • Idle: Periods between video processing.
  • TTS Generation: Text-to-speech processing.
  • WhisperX Alignment: Audio alignment for synchronization.
  • Rendering: Frame rendering with FFmpeg.
  • Encoding: Video encoding.
  • Upload: Video upload phase.

Comprehensive Process Termination

An improved OOM handler was implemented to:

  • Terminate all Python processes within the virtual environment.
  • Explicitly kill FFmpeg and WhisperX subprocesses to prevent orphaned processes.
  • Enforce strict memory limits and rapid intervention.

Integration Architecture

The monitor is launched as a persistent daemon alongside the main pipeline processes, with:

  • Automatic log rotation: Prevents unbounded log growth.
  • Clean shutdown: Ensures the monitor is terminated gracefully during pipeline shutdown.

Configuration Improvements

  • 5-second sampling interval: Replaces the 30-second interval to detect rapid memory spikes.
  • 120GB threshold: Provides a higher but more strictly enforced memory limit.
  • Comprehensive diagnostics: Logs include phase tracking, memory trends, and status indicators.

Diagnostic Output

The monitor provides detailed logging, including:

  • Timestamp, video name, phase, progress, CPU usage, RAM usage, and status indicators (e.g., ✅ OK, ⚠️ HIGH, ❌ LIMIT).
  • Phase tracking: Links crashes to specific input videos and pipeline stages.
  • Trend analysis: Highlights memory growth rates and identifies problematic phases.

Result: The system now detects and mitigates OOM errors effectively, ensuring stable operation across all pipeline phases while providing detailed diagnostics for troubleshooting.