X API rate limits cause service conflicts between engagement bot and video uploader

Issue

Two independent services—a video uploader and an engagement bot—shared the same X (Twitter) API OAuth credentials, leading to repeated rate limit conflicts. The engagement bot’s fixed 15-minute interval consumed API posting quota, blocking video uploads even when the bot was active. This resulted in stalled uploads despite successful media processing, with no mechanism to coordinate API usage between the services.

Culprit

The root cause was the lack of coordination between services sharing the same rate limit pool:

1. Shared OAuth credentials: Both services competed for the same API quota under X’s free tier restrictions.
2. No activity awareness: The engagement bot operated independently without awareness of video uploader status.
3. Fixed scheduling: The bot’s rigid 15-minute cycle ignored video upload requirements, consuming quota during uploads.

Solution

Filesystem-Based Coordination

Implemented a dynamic coordination mechanism using filesystem observation:

• Activity Detection: The system checks the modification time of files in the video uploader’s archive directory to determine if uploads are active.
• Dynamic Backoff: The engagement bot delays operation if no video upload activity is detected within a 150-minute threshold.
• Rate Limit Sharing: The bot only operates when video uploads are active or when uploads are definitively blocked, preserving quota for critical operations.

Key Features

• No Cross-Service Communication: Coordination relies solely on filesystem observation, eliminating the need for direct inter-service communication.
• Threshold-Based: A 150-minute threshold (matching the video upload cycle) ensures responsiveness while balancing API efficiency.
• Minimal Overhead: The solution requires no configuration changes and operates transparently.

Result: The engagement bot now adapts dynamically to video uploader activity, ensuring fair API quota distribution and preventing upload stalls.