Resolved

YouTube series videos scheduled out of order after service restart

Python
YouTube API
Asyncio
Backend
Scheduling
State Management
Debugging
Data Persistence

Python asyncio scheduler with YouTube Data API v3 loses series part ordering after watcher restart | In-memory `_series_slots` dictionary not persisted, causing parts 3/18 to schedule before parts 1-2

A Python-based asyncio scheduler for YouTube video uploads was failing to maintain sequential ordering for multi-part video series after service restarts. This caused later parts (e.g., Part 3/18) to be scheduled before earlier parts (Parts 1-2), disrupting the intended upload sequence.

Issue

After a scheduler service restart, series videos were scheduled out of order, leading to critical errors like:

  • CRITICAL: Series '...' part X uploading before part Y!
  • Visually incorrect ordering in YouTube’s scheduled videos interface.

This occurred because the system relied on an in-memory dictionary (_series_slots) to track scheduled slots for each series part. When the watcher process restarted between uploading different parts, the in-memory state was lost while YouTube retained previously scheduled parts, resulting in incorrect scheduling priorities.

Culprit

The root cause was the ephemeral nature of the _series_slots dictionary, which stored series_name -> {part_num: scheduled_datetime} mappings in memory. During synchronization with YouTube (sync_with_youtube()), the system only fetched video status data, lacking the necessary snippet (title) information to:

  • Identify which scheduled videos belonged to which series.
  • Rebuild the _series_slots state from existing scheduled videos.
  • Maintain correct sequential ordering after restarts.

When Part 3 arrived after a restart, the empty _series_slots dictionary caused the system to treat it as a standalone video, scheduling it before Parts 1-2 had been uploaded.

Solution

Enhanced YouTube Data Fetching

  • Modified API requests to include both status and snippet (title) data to extract series metadata.

Series Metadata Parsing

  • Implemented regex-based parsing of video titles to extract series names, part numbers, and total parts.

State Reconstruction Logic

  • Added logic to rebuild the _series_slots dictionary from YouTube’s existing scheduled videos, prioritizing YouTube data as the source of truth.
  • Ensured the scheduler merges new parts with existing scheduled videos correctly, even after service restarts.

Improved Scheduling Fallback

  • Enhanced slot calculation to handle missing parts by determining the latest scheduled part and scheduling the new part with appropriate spacing.

Result

The scheduler now correctly maintains sequential ordering regardless of service restarts, with proper spacing between parts. For example, Part 18/18 arriving after a restart correctly schedules 15 slots (15 × 6 hours) after the latest found part (e.g., Part 3/18). The system logs state reconstruction events, such as Rebuilt series slots from YouTube: %d series with scheduled parts.