X API Free tier rate limits block bot development with 100 searches per month

Issue

The X engagement bot became unusable under the Free tier API limits, encountering immediate 429 ‘Too Many Requests’ errors upon startup. The bot was designed to make one API call every 60 seconds, resulting in 1440 calls per day (43,200 per month), far exceeding the Free tier limit of 100 searches per month. This restriction enforced a single API request every 15 minutes, making it impossible to iterate on the two-lane reply system (DISCUSS/ENGAGE lanes) as each test consumed 1% of the monthly quota.

Culprit

The issue stemmed from direct X API calls in the server without any caching mechanism. The architecture assumed higher-tier API access, where calls were made every 60 seconds, but the Free tier enforces:

• A hard limit of 100 searches per month.
• A rate limit of one request per 15-minute window.
• Immediate 429 errors after the first request in the 15-minute window.

Solution

A cache-and-refresh pattern was implemented to stay within the 100-search-per-month limit, refreshing the cache every 7.3 hours. This reduced API calls from 1440 per day to approximately 3.3 per day, ensuring compliance with the Free tier restrictions.

Cache System

A caching mechanism was introduced to store tweet data locally. The cache system checks the age of stored tweets and triggers a refresh only when necessary:

• Refresh interval: Every 7.3 hours, calculated to stay within the 100-search monthly limit.
• Graceful degradation: If the API returns a rate limit error during refresh, the system continues using cached data until the next scheduled refresh.

Modified Search Logic

The search logic was updated to:

1. Check if a cache refresh is needed before making an API call.
2. Use cached tweets for subsequent reads, minimizing API usage.
3. Include fallback mechanisms to handle rate limit errors gracefully.

Key Improvements

Metric	Before	After
API calls per day	1440	3.3
API calls per month	43,200	99
Rate limit compliance	Failed (429 errors)	Compliant
Cache refresh math	N/A	365 days/12 months/100 searches * 24 hours

Additional enhancements included:

• A cache status endpoint to monitor data freshness.
• A lockfile-based automation script to prevent duplicate bot instances. The bot now operates continuously using cached data, only querying the X API three times per day for fresh data.