A YouTube upload system faced a critical issue where series videos lost their priority when re-queued after hitting quota limits, causing standalone videos to upload out of order. This disrupted the intended sequence of series videos, such as uploading standalone videos before completing a series (e.g., Part 1 before Part 2).
Issue
Series videos, which required strict ordering, were incorrectly re-queued with a low-priority value during quota backoff. This allowed non-series videos to upload before the series was complete, breaking the intended upload order. For example, a series video labeled ‘Part 1/9’ might be delayed by quota limits, allowing a standalone video like ‘Production OAuth Storage’ to upload first, thereby disrupting the series continuity.
Culprit
The root cause was the incorrect handling of video priorities during re-queueing. The system used a tuple-based priority system where:
- Series videos had a high-priority value.
- Non-series videos had a slightly lower priority value.
- Deferred videos (after quota backoff) were incorrectly assigned the same low-priority value, regardless of their original status.
This caused non-series videos to always process before deferred series videos, disrupting the intended upload order.
Solution
A priority-preserving mechanism was implemented to ensure that deferred videos retained their original priority type (series or non-series) during re-queueing. This involved:
- Checking if the video was part of a series.
- Preserving the original priority structure for deferred videos, maintaining series metadata for proper ordering.
- Only updating the scheduled time with the backoff delay, ensuring series videos maintained their upload priority.
The solution ensured that series videos were uploaded in the correct sequence, even after quota-induced delays.