Resolved

Python asyncio PriorityQueue blocks high-priority items during sleep

Python
asyncio
PriorityQueue
Concurrency
Backend
Performance
Debugging
Queue

Asyncio processor using PriorityQueue gets stuck in long sleep operations | Prevents high-priority items (priority -1) from being processed while waiting for future-scheduled items (priority 998)

A video upload processor using Python’s asyncio and PriorityQueue encountered a critical issue where high-priority items were blocked during long sleep operations. The system prioritized X/TikTok videos (priority -1) and YouTube backoff items (priority 998), but the latter’s scheduled delays caused the processor to stall for extended periods, preventing high-priority uploads from being processed.

Issue

High-priority X videos remained queued and unprocessed for extended periods (e.g., 41+ minutes) when the processor encountered future-scheduled YouTube backoff items. During a blocking asyncio.sleep() call lasting 300+ seconds, the processor could not dequeue new items, effectively freezing the queue until a system restart.

Culprit

The blocking sleep mechanism in the upload processor prevented the queue from operating normally. When a future-scheduled YouTube backoff item was dequeued, the processor would sleep for the remaining delay time before processing it, completely halting all other queue operations during this period. This design flaw created a coupling between future item scheduling and queue responsiveness.

Solution

The solution replaced the blocking sleep with a non-blocking re-queue pattern:

  1. Immediate Re-Queueing: Instead of sleeping, future items were re-queued for later processing.
  2. Short Poll Interval: A maximum 5-second sleep was introduced to maintain system responsiveness.
  3. Priority Preservation: Proper queue reinsertion ensured that high-priority items (e.g., X videos) remained accessible and were processed immediately.

Processing Flow Improvements

  • Dequeued a future-scheduled YouTube backoff item.
  • Immediately re-queued it for later processing.
  • Slept for a maximum of 5 seconds.
  • The next queue.get() operation correctly retrieved the high-priority X video.
  • The X video was processed immediately without waiting for the backoff period.

This change eliminated the blocking behavior, allowing high-priority videos to upload within seconds of creation, even during active YouTube quota backoff periods. The system no longer required restarts to recover from stalls.