Resolved

Authentication error in YouTube API uploads due to incompatible token format

Python
OAuth
YouTube API
Serialization
Authentication
Debugging
File Handling

YouTube video uploads failed with a UTF-8 decode error when attempting to read legacy pickle-format token files as JSON | Issue resolved by migrating to the JSON-based token storage format.

A Python-based system for YouTube video uploads encountered an authentication error when transitioning from legacy binary token storage to a newer JSON-based format. The issue arose when the system attempted to read outdated pickle-format token files using a library that expected JSON input, resulting in a UTF-8 decode error.

Issue

YouTube video uploads failed at startup with the following error:

‘utf-8’ codec can’t decode byte 0x80 in position 0: invalid start byte

This error occurred when the system tried to load a token file (youtube_token.pickle) that was created using an older version of the OAuth library. The file contained binary data in a pickle format, which was incompatible with the updated library’s expectation of JSON-formatted tokens.

Culprit

The root cause was a format mismatch between the token file and the authentication library:

  • The system contained a legacy binary pickle file (1.2KB), created using older OAuth flows.
  • The updated google-auth library exclusively supports JSON-formatted token files for security and compatibility reasons.
  • The library’s Credentials.from_authorized_user_file() function attempted to parse the binary file as UTF-8 encoded JSON, causing the UTF-8 decode error.

The error manifested when token refresh was required after expiration, as the system relied on the outdated pickle file for authentication.

Solution

Immediate Resolution

The solution involved replacing the legacy pickle file with a new JSON-formatted token file:

  1. Removed the incompatible pickle file to eliminate the source of the error.
  2. Triggered a re-authentication flow to generate a new JSON-formatted token file with proper UTF-8 encoding.
  3. Verified the new token file to ensure it was valid and compatible with the updated library.

The new token file adhered to the JSON format, ensuring compatibility with the authentication library and resolving the UTF-8 decode error.

Verification Process

The updated system was tested to confirm:

  • Successful uploads for both horizontal and vertical video formats.
  • Proper handling of metadata and thumbnail processing.
  • Correct encoding and structure of the new JSON token file.

Proposed Improvements

To prevent similar issues in the future, the system was enhanced with:

  • Explicit format validation to detect and reject legacy pickle files.
  • Automatic migration support to assist users in transitioning from binary to JSON token formats.
  • Diagnostic logging to identify and flag incompatible token formats early.

These improvements ensure backward compatibility while enforcing the use of modern token formats.

Outcome

The authentication error was resolved by migrating from legacy pickle-format token files to JSON-based tokens. This change improved system compatibility and security while maintaining functionality for YouTube API interactions.