Performance regression in parallel video rendering with Python multiprocessing

A Python-based video rendering pipeline leveraging multiprocessing.Pool and MoviePy/FFmpeg for parallel frame processing experienced a severe performance regression. The system, designed to utilize 32 CPU cores for rendering at 80-100fps (1080p) and 20-25fps (4K), degraded to 10-15fps (1080p) and 8-10fps (4K), with CPU usage dropping from 95% to just 5-20% across all cores. The regression occurred after introducing a “streaming” approach to reduce memory usage, which inadvertently destroyed parallelism.

Issue

The performance collapse manifested in the following ways:

1. Frame rate drop: From 80-100fps (1080p) and 20-25fps (4K) to 10-15fps (1080p) and 8-10fps (4K).
2. CPU underutilization: Dropped from ~95% to 5-20% across all 32 cores, behaving as if single-threaded despite parallel execution.
3. Crash on startup: A KeyError: 'USE_PARALLEL_RENDERING' error appeared, preventing the system from running.

Culprit

1. Missing Configuration Check

The code attempted to read an environment variable (os.environ['USE_PARALLEL_RENDERING']) that no longer existed, causing immediate crashes. This variable was removed during refactoring but not updated in the configuration checks.

2. Lazy Iterator Starvation

The primary performance bottleneck was the use of pool.imap() with lazy scheduling, which:

• Only scheduled 10-20 frames at a time (based on chunksize × workers).
• Created a bottleneck where FFmpeg’s I/O-bound pipe consumption rate limited parallel execution.
• Caused worker pool starvation, as each worker completed one task and then waited for more, leaving most cores idle.
• Resulted in only 1-2 cores active at any given time.

3. Premature Optimization

The “streaming” approach attempted to address a non-existent memory issue (the system had 128GB RAM). Instead of rendering all frames upfront, it returned an unconsumed iterator, which destroyed parallelism because:

• Multiprocessing pools require upfront work to saturate workers.
• Lazy scheduling effectively forced sequential execution.
• The original approach using list(pool.imap(...)) worked because it immediately scheduled all frames across workers.

Solution

Fix 1: Remove Configuration Toggle

The unnecessary environment variable check was eliminated, and parallel rendering was hardcoded as the default (since it is always faster). This resolved the KeyError and ensured the system always used parallel processing.

Fix 2: Restore Eager Frame Rendering

The solution reverted to the original approach, which forces immediate scheduling of all work across workers:

with Pool(processes=num_workers) as pool:
    frames = list(tqdm(
        pool.imap(render_single_frame, render_args),
        total=total_frames
    ))

This ensures:

• All frames are scheduled immediately across all workers.
• Workers remain saturated with a full queue of tasks.
• CPU utilization reaches ~95% across all cores.

Fix 3: Update Caller for Frame List

The calling code was modified to handle the returned frame list (instead of an iterator):

frames = render_frames_parallel(...)  # Now returns a complete list
write_video_with_ffmpeg(frames, ...)  # Processes the full frame list

Fix 4: Clean Up Configuration

Obsolete environment variables (e.g., USE_PARALLEL, USE_PARALLEL_RENDERING) were removed from the configuration.

Performance Results

The fixes restored expected performance levels:

• 4K rendering: 8-10fps → 20fps (2.5× improvement).
• 1080p rendering: 10-15fps → 80fps (5-8× improvement).
• CPU utilization: 5-20% → 95% across all 32 cores.
• Memory usage: Remained stable (~60-76GB for 11,824 4K frames), as the “streaming” approach did not actually reduce memory consumption due to pool buffering.