Python asyncio PriorityQueue causes instant retries instead of exponential backoff

A Python-based asynchronous upload system using asyncio.PriorityQueue and exponential backoff was failing to respect scheduled retry delays, leading to immediate retry attempts instead of adhering to the configured backoff schedule.

Issue

Upload retries were executing instantly, despite the system being configured for exponential backoff delays (e.g., 5 minutes, 15 minutes, 1 hour). This resulted in:

• All retry attempts exhausted within less than a second.
• YouTube API being overwhelmed with excessive requests per second, resembling DDOS behavior.
• Quota limits being hit without adequate time to reset.
• Videos being prematurely archived without completing the retry process.

Log analysis revealed retries occurring within 0.1-0.5 seconds of each other instead of the scheduled delays:

18:33:12 - retry 0 → defer 300s (5 minutes)
18:33:12 - retry 1 (0.4 seconds later!) ← Should wait 5 minutes
18:33:12 - retry 2 (0.1 seconds later!) ← Should wait 15 minutes

Culprit

The root cause was a mathematical error in timestamp calculation, where the system used the file creation timestamp (from hours or days earlier) instead of the current time for scheduling future retries.

The incorrect implementation used pair.creation_time + delay to calculate the scheduled retry time. Since pair.creation_time represented an old timestamp (e.g., from the file’s creation date), adding a delay still resulted in a timestamp far in the past. This caused the timing check to always fail, leading to instant retries.

Solution

Timestamp Fix

The timestamp reference point was corrected to use the current time (time.time()) instead of the file creation time for scheduling retries. This change was applied across all four locations where retries were scheduled.

Key Changes

• Current time as reference: Retry scheduling now uses time.time() + delay, ensuring correct future timestamps.
• Maintained priority queue structure: The tuple format (is_series, series_part, series_total, scheduled_time) was preserved for consistency.
• Preserved exponential backoff: The schedule of delays (5 minutes → 15 minutes → 1 hour → 3 hours → 6 hours → 12 hours) remained unchanged.

Result

With the fix in place, retries now correctly respect scheduled delays:

• The system waits the full configured time before retrying.
• YouTube API quotas have sufficient time to reset.
• Proper exponential backoff behavior is restored.
• Console output now reflects the expected wait messages:

  16:25:00 - ⏰ Item scheduled for future - waiting 300 seconds
  [5-minute pause]
  16:30:00 - ✓ Wait complete - processing now

This resolved the issue of immediate retries and restored proper exponential backoff functionality for reliable API usage.