Resolved

YouTube API uploads fail with quota exceeded and series parts upload out of order

Python
YouTube API
API Integration
Error Handling
Retry Logic
Sorting Algorithms
Video Processing
Backend

YouTube upload system incorrectly archived videos on temporary quota errors and uploaded series parts in wrong order due to timestamp-based sorting | Implemented exponential backoff retry logic and fixed series part sorting priority

Issue

The YouTube upload system experienced four critical failures:

  1. Permanent archival of videos due to temporary quota errors (HTTP 403 quotaExceeded), preventing automatic retries.
  2. Lack of retry mechanisms for quota or server errors, requiring manual intervention.
  3. Incorrect series upload order due to timestamp-based sorting, causing part 4 to upload before part 3.
  4. Loss of shorts files when partial uploads failed, as both videos in a pair were archived.

Culprit

Error Handling

The system lacked error classification, treating all HTTP errors as permanent failures. Without distinguishing between temporary (e.g., quota) and permanent errors, videos were immediately archived, blocking recovery.

Sorting Logic

The sort key prioritized file creation timestamps over series part order, leading to inconsistent upload sequences. Python’s tuple-based sorting evaluated creation_time first, overriding the intended series ordering.

State Management

The system did not track the upload state of individual videos in a pair separately. A failure in either the horizontal or shorts upload would archive both, causing data loss.

Solution

Error Classification and Retry Logic

Introduced an error classification system to distinguish between temporary and permanent errors, enabling automatic retries with exponential backoff. Quota errors now retry with increasing delays (5m → 15m → 1h → 3h → 6h → 12h), while server errors retry more aggressively (30s → 2m → 5m).

Series Order Fix

Restructured the sort key to prioritize series part number over creation time, ensuring chronological upload order. This resolved issues where newer files (e.g., part 4) would incorrectly upload before older series parts (e.g., part 3).

Partial Upload Handling

Enhanced the system to track upload state per video in a pair. Horizontal and shorts uploads are now managed independently, allowing the shorts file to retry while the horizontal video is published. This prevents data loss and maintains scheduling consistency.

This transformation improved reliability by introducing robust error recovery, correct series ordering, and independent state management for paired uploads.