Resolved

Python heapq priority queue fails comparing dataclass objects

Python
asyncio
heapq
Priority Queue
Exponential Backoff
YouTube API
Video Processing
Debugging

Priority queue errors when comparing VideoPair dataclass instances with identical priority tuples | Global quota backoff stuck at 300s due to per-item retry counters | Video rendering timeouts at 120 minutes for long videos

Issue

The video upload watcher system encountered three critical technical issues:

  1. Priority Queue Comparison Failure: The system crashed when attempting to compare VideoPair objects in the priority queue due to a missing comparison method, halting upload processing entirely.
  2. Stuck Exponential Backoff: The retry mechanism for YouTube API quota errors consistently deferred retries for 300 seconds, failing to escalate delays exponentially (e.g., 300s → 900s → 3600s).
  3. Insufficient Timeout for Long Videos: High-resolution videos (15+ minutes) repeatedly hit a 120-minute timeout threshold before rendering completed, causing unnecessary failures.

Culprit

  1. Priority Queue Failure: The VideoPair dataclass lacked a comparison method (__lt__), causing Python’s heapq module to fail when comparing objects with identical priority tuples.
  2. Backoff Calculation Logic: The retry counter was scoped per-video-pair, resetting independently for each upload attempt. This resulted in all quota errors triggering the initial 300-second delay, as each pair’s retry count remained at zero.
  3. Timeout Configuration: The hardcoded 120-minute timeout was insufficient for long rendering tasks, which could exceed 7-8 hours for complex, high-resolution videos.

Solution

  1. Dataclass Comparison: Implemented a custom __lt__ method in the VideoPair class to ensure proper comparison logic using the existing sort key, enabling stable heap operations.
  2. Global Quota Backoff: Introduced a global retry counter for quota errors to ensure exponential backoff progression, while preserving per-pair counters for other error types. Added logic to reset the global counter upon reaching maximum retries.
  3. Extended Timeout Threshold: Increased the timeout limit from 120 minutes to 8 hours to accommodate extended rendering times for high-resolution videos.

Result: The system now correctly handles object comparisons in priority queues, implements true exponential backoff for quota errors, and supports long rendering tasks without premature timeouts.