Working with Python’s asyncio and priority queues to manage multi-platform video upload automation, where YouTube and X/Twitter uploads share a single queue but have distinct API quotas. The system prioritizes uploads using tuple-based sort keys.
Issue
When YouTube API quota limits triggered 403 quotaExceeded errors, the entire upload system stalled, blocking unrelated X/Twitter uploads that had separate quotas. The exponential backoff mechanism used await asyncio.sleep(delay), freezing the entire queue—including X uploads that could have processed normally.
Error pattern:
- HTTP 403 quotaExceeded errors from YouTube API.
- Blocking sleep for 300-3600 seconds during retries, halting all uploads.
Impact metrics:
- System throughput dropped to 0 uploads per hour during YouTube quota exhaustion.
- X uploads (500/month quota) remained idle while waiting for YouTube retries.
- Batch processing stalled for hours during peak usage periods.
Culprit
The blocking architecture treated all uploads as a single sequential pipeline:
- Single-task queue processing: Processed one upload at a time, without isolating platforms.
- Blocking sleep implementation: Used
await asyncio.sleep(delay)during retries, freezing the entire queue. - Assumption of shared quota: Designed for single-platform (YouTube-only) workflows, without platform-specific retry handling.
The root cause was conflating retry delays with queue blocking, assuming all uploads shared the same rate limits and failure modes.
Solution
Non-Blocking Deferred Retry System
Replaced blocking sleep with priority-based deferral:
- Priority structure: Normal uploads use
(0/1, part#, total, original_time), while deferred uploads use(999, 999, 999, original_time + delay). - Natural delay enforcement: Queue processing time validates timestamps, ensuring delayed retries.
- Platform isolation: X uploads process immediately during YouTube backoff.
Performance impact:
| Metric | Before | After |
|---|---|---|
| YouTube quota hit | 0 uploads/hour | 15-20 X uploads/hour |
| Queue stall time | 3-5 hours | <5 minutes |
| Platform isolation | None | Full separation |
X/Twitter Upload Pipeline
-
New watcher component: Dedicated X/Twitter watcher with:
- Vertical FHD configuration.
- Simplified logic (50% less code than YouTube watcher).
-
JSON format conversion: Transformed YouTube metadata to X-specific format, preserving essential fields like title, description, and tags while removing YouTube-specific data.
-
Multi-watcher orchestration: Parallel process management with:
- Clean shutdown handling.
- PID tracking for all three watchers.
Architectural outcome:
- Platform-agnostic upload queue.
- Independent rate limit handling for each platform.
- Full utilization of separate API quotas.
- Zero manual intervention required for X uploads.