A video generation pipeline using MoviePy for frame rendering encountered a critical performance bottleneck, achieving only 7.47 frames per second (FPS) on a 32-core system. This resulted in a rendering time of approximately 15 minutes for a 450-second video, significantly limiting scalability. The bottleneck was caused by MoviePy’s sequential frame rendering, which utilized only 1 out of 32 CPU cores for rendering frames, leaving the remaining cores idle.

Issue

The video generation process was constrained by the sequential execution of frame rendering, causing a significant slowdown. Despite the availability of 32 CPU cores, the system utilized only 1 core during frame rendering, leading to a utilization of 96.7% of a single core while the other cores remained unused. This limited the system’s ability to handle larger videos efficiently and left users waiting for extended periods.

Culprit

MoviePy’s architecture enforces single-threaded frame generation through its make_frame(t) callback system. This constraint arises from three interdependent factors:

Sequential callback execution: The make_frame function processes frames one at a time in strict time order, making it impossible to parallelize.
State-dependent animations: Each frame’s output depends on cumulative calculations from all previous frames, including character visibility timing, scroll position, line wrapping decisions, and syntax highlighting state.
Tight audio synchronization: The system uses precise audio timestamps to bind character appearances to exact moments in the audio, which requires maintaining frame-by-frame synchronization.

The performance breakdown highlighted the imbalance in resource utilization:

Frame rendering phase: Utilized a single core for 90% of the processing time.
FFmpeg encoding phase: Utilized all 32 cores for 6% of the processing time.
Other phases: Utilized multiple cores or GPU resources for 4% of the processing time.

Solution

A proposed refactor would enable the use of parallel processing while maintaining synchronization:

Pre-calculation phase: Generate all character positions and frame metadata sequentially but efficiently, ensuring accurate cumulative calculations.
Parallel rendering phase: Distribute frame rendering across all 32 CPU cores, leveraging pre-calculated metadata to avoid dependencies between frames.
Direct FFmpeg pipeline: Bypass MoviePy’s sequential rendering for final encoding, ensuring full utilization of available hardware resources.

This approach would significantly reduce rendering time from 900 seconds to an estimated 30-40 seconds, achieving a 7× speedup in the overall pipeline. However, the refactor would require addressing complex dependencies and ensuring pixel-perfect output matching.

Current Mitigations

To address the bottleneck without a full refactor, the following measures were implemented:

Reduced FPS: Dropped the frame rate from 60 FPS to 25 FPS, achieving a 2.4× speedup in rendering.
Multicore FFmpeg: Offloaded the encoding phase to all 32 cores, providing a 10-20% improvement in overall performance.

Despite these mitigations, frame rendering remains a single-core bottleneck, resulting in a total rendering time of 15 minutes per 7-minute video. This limitation highlights the need for a more comprehensive solution to fully utilize available hardware resources.

Alternative Approaches

Alternative solutions were evaluated but rejected due to constraints:

GPU acceleration: Not viable due to CPU-bound text rendering operations and limited benefits for non-matrix operations.
Hardware encoding: Addressed only the final encoding phase, which constitutes only 10% of the pipeline time.
Segmented rendering: Not feasible due to potential desynchronization with audio and the requirement for pixel-perfect stitching between segments.

The proposed refactor remains the most effective approach for leveraging the available hardware and significantly improving rendering performance.