Python video generation watcher fails to process existing files after restart

Issue

The video generation watcher failed to process files present before startup, requiring manual intervention to reprocess existing content. The system relied on in-memory state tracking, complex file movements across multiple directories, and timestamp-based naming, leading to inconsistent processing and operational inefficiencies. Users experienced scenarios where files added while the watcher was offline remained unprocessed until a manual restart.

Culprit

1. In-Memory State Tracking: The watcher used an in-memory processed_scripts set, which reset on restart and ignored files created before the system started.
2. Complex Intermediate Staging: Files followed a convoluted path through multiple directories, requiring extensive file management logic and increasing the risk of errors.
3. Timestamp-Based Naming: Dynamic naming conventions (e.g., output_horizontal_20251116_180457.mp4) complicated file pairing and made the system difficult to debug.
4. Overly Complex File Discovery: The system required checking both in-memory state and filesystem archives, parsing timestamps from filenames, and managing multiple file extensions, leading to maintenance challenges.

Solution

1. Stateless Filesystem-Based Processing

Replaced in-memory tracking with filesystem state to ensure robustness across restarts:

• Check for Inputs and Outputs: Only process if inputs (e.g., .md, .json, .jpg) exist and outputs (e.g., .mp4) do not.
• Single Source of Truth: Use the filesystem to track processed files, ensuring no reliance on in-memory state.

2. Direct Directory Watching

Eliminated intermediate staging directories by watching final directories (uploads/youtube/ and uploads/youtube-shorts/) directly, reducing complexity and improving efficiency.

3. In-Place Generation

Generated videos directly in their final destinations, eliminating the need for file shuffling:

• Standardized Naming: Use deterministic naming conventions (e.g., script.md → script.mp4 and script.jpg).
• Simplified File Pairing: Ensure horizontal and vertical variants share consistent naming patterns (e.g., script.mp4 for horizontal and script_shorts.mp4 for vertical).

4. Architecture Simplification

• Reduced Code Complexity: Eliminated redundant methods and file management logic, reducing the codebase by over 100 lines.
• Deterministic Processing: Filesystem state now determines processing status, ensuring stateless operation and no reliance on restart-dependent memory.
• Efficient File Handling: Reduced file movements from four directories to a single, direct path, improving performance and reducing errors.

The solution transforms the system into a stateless, filesystem-driven architecture where processing is determined solely by the presence or absence of input and output files, ensuring reliability and simplicity.