On Going Legit on Twitter/X (Or: How I Learned to Stop Scraping and Love the Rate Limit)

Here's the thing about deciding to do things The Official Way¹: you think you're making your life easier—cleaner code, better documentation, compliance with Terms of Service—but what you're actually doing is trading one set of problems (scraping is brittle, cookies expire, the library maintainer might abandon ship) for a completely different set of problems that somehow feel more legitimate but are actually just as annoying, except now you can't even complain about them because you signed up for this.

This is a story about three days in November 2025 when I migrated a Twitter bot from cookie-based scraping to Twitter's Official API, hit every possible rate limit and restriction, tried to be clever about it, failed, tried again, failed differently, and eventually arrived at what software engineers call "a working solution" and what everyone else calls "good enough."

Context: The code in this post is from a heavily customized fork of a very old version of the ElizaOS framework. Your mileage may vary, but the API lessons apply universally.

Act 1: The Great Migration (Nov 8-9)

The typescript bot had been using agent-twitter-client, which is one of those unofficial libraries that does the thing Twitter doesn't want you to do, which is to say it uses cookie-based authentication to call Twitter's internal GraphQL API². This had been working fine for months—until early November 2025, when I noticed the bot had stopped replying to mentions. Something broke. Whether it was cookie expiration, GraphQL endpoint changes, or Twitter's ongoing war against unofficial access didn't really matter; what mattered was that relying on an unofficial library that violated Terms of Service meant debugging mysteries with no documentation and no support. Time to go legit.

So: time to go legit. Commit 190165baf on Nov 8: "Refactor Twitter client to use Twitter API v2."

The migration itself was straightforward in that deeply tedious way where straightforward means "I know exactly what needs to happen" but tedious means "I have to change it in 47 different places":

• Ripped out the cookie-based authentication
• Installed twitter-api-v2, the official library
• Set up OAuth 1.0a (which is the version where you have four different strings—API key, API secret, access token, access token secret—and you have to keep them all straight)
• Updated environment variables across multiple character configurations because yes, this is a multi-bot system, each with different credentials, because why keep things simple

// Four strings that must all be correct or nothing works
const auth = new TwitterAuth({
  appKey: process.env.TWITTER_API_KEY,
  appSecret: process.env.TWITTER_API_SECRET_KEY,
  accessToken: process.env.TWITTER_ACCESS_TOKEN,
  accessSecret: process.env.TWITTER_ACCESS_TOKEN_SECRET,
});

First lesson learned: Official APIs come with rate limits that are documented, which sounds like an improvement until you realize documentation doesn't make the limits any less restrictive.

Act 2: Nobody Reads The Documentation Including Me (Nov 9)

Three commits on Nov 9 (dfbde35f3, 3bd430c4e, cdb62c304) were basically me realizing that Future Me and Future Other People would have absolutely no idea how this OAuth configuration worked, so I wrote:

• A 187-line README explaining the setup process
• Updates to CLAUDE.md with architectural notes
• Character-specific configuration documentation

Also discovered during this documentation spree: The free tier has a 100 post retrieval cap per month. Not 10,000. Not 1,000. One hundred. That's approximately five mention checks if each returns 20 tweets. Not rate limits per 15 minutes—which would be annoying but manageable—but a hard monthly cap, like a cell phone plan from 2005, except worse because at least those gave you more than 100 minutes. X's own post-cap doc spells it out in painful detail.

The error response calls this product_name: 'standard-basic', which sounds like it might be a paid tier but is actually X's internal SKU for the free tier, a naming choice that confused me for an embarrassing amount of time.

This will become relevant approximately one hour later.

Act 3: Hitting The Wall (Nov 9-10)

The bot kept crashing. Not in interesting ways—just boring "uncaught exception" crashes whenever it hit API limits. So I spent several commits (85a9ff6e3, 2142dab44, 6940b0116) implementing what I call "survival mechanisms" and what more disciplined engineers call "proper error handling":

1. TWITTER_INTERACTIONS_ENABLE flag - A kill switch to disable mention monitoring entirely, because sometimes the only way to win is not to play
2. DKG query timeout - Stop waiting forever for failed operations (the technical term for "forever" here is "45 seconds" but in computer time that's geological)
3. Try-catch wrapping - Around timeline population so startup doesn't crash
4. Dynamic polling intervals - Adjust check frequency when rate-limited

Here's where the math got ugly: Every interaction check cost 20 tweets from the monthly cap. The bot was polling every 120 seconds.

First check: 20 posts consumed (80 left) Second check: 20 posts consumed (60 left) Third check: 20 posts consumed (40 left) Fourth check: 20 posts consumed (20 left) Fifth check: 20 posts consumed (0 left)

Monthly cap exhausted. Time elapsed: under one hour.

You see the problem.

Act 4: Getting Smart (Nov 10 morning)

Commit 2c1d35b29: "Optimize Twitter API usage for free tier."

This is where I actually read the API documentation instead of just skimming it with the confidence of someone who has "done this before"³. Three optimizations that collectively saved 95% of API usage:

1. The since_id parameter - This is the genius one. Instead of fetching the same 20 most recent tweets every check and re-checking if they're new, you pass the ID of the last tweet you've seen and Twitter only returns newer tweets. This is a native Twitter API v2 parameter specifically designed for the "polling pattern"—the filtering happens server-side, not in your code. Transformed the operation from O(n) re-scanning to O(new). Like magic except it's just basic API design that I should have been using from day one.

// Before: Wasting 20 posts from the cap every check
const mentions = await client.fetchSearchTweets(
  `@${username}`,
  20,
  SearchMode.Latest
);
// Returns the same 20 tweets. Every. Single. Time.

// After: Only fetching new tweets
const mentions = await client.fetchSearchTweets(
  `@${username}`,
  20,
  SearchMode.Latest,
  undefined, // cursor
  lastCheckedTweetId  // since_id - the magic parameter
);
// Only returns tweets newer than lastCheckedTweetId

2. The -from:username query modifier - Excludes your own tweets from search results, which sounds obvious but I'd been paying API costs to fetch and filter my own tweets for two days.

// Adding the second optimization
const mentions = await client.fetchSearchTweets(
  `@${username} -from:${username}`,  // Exclude own tweets from results
  20,
  SearchMode.Latest,
  undefined,
  lastCheckedTweetId
);
// Went from 20 posts/check to ~0-5 posts/check with both optimizations

3. Increased polling interval - From 120 seconds to 300 seconds (5 minutes). Less frequent checks, fewer API calls, still responsive enough for a bot that doesn't need to respond instantly because we're not running an emergency service here.

New math (optimistic scenario assuming since_id actually works and deduplication helps): Maybe 5 new unique mentions per day × 20 posts to fetch them = 100 posts per month.

Exactly at the cap. Barely sustainable. Problem... mostly solved? Technically viable in the same way that technically you could survive on 800 calories a day.

Except I also fixed pagination handling (PaginationState type to track v2 vs v1.1 cursors properly) and photo/video filters that were getting dropped in fallback scenarios, because when you're in there fixing things you might as well fix all the things.

Act 5: The v1.1 Gambit (Nov 10 mid-day)

Here's where I got clever, which is always dangerous.

I thought: "Wait, maybe v1.1 endpoints have different limits than v2? Maybe that's the escape hatch?"

Commits 241b6c80e and b4d5fb56f implemented a full v1.1 fallback system:

• Added TwitterApiv1 client alongside the v2 client
• Built a fallback chain: Try v2 → if 429 (rate limit exceeded), try v1.1
• v1.1 support for search, user timeline, home timeline, tweet creation

// The beautiful fallback that didn't work
export async function* searchTweets(
  query: string,
  maxTweets: number,
  searchMode: SearchMode,
  auth: TwitterAuth,
  sinceId?: string,
): AsyncGenerator<Tweet, void> {
  const client = auth.getV2Client();

  try {
    // Try v2 first
    const searchIterator = await client.v2.search(query, {
      max_results: Math.min(maxTweets, 100),
      // ... other params
    });

    for await (const tweet of searchIterator) {
      yield convertToTweet(tweet);
    }
  } catch (error) {
    console.warn("v2 search failed, trying v1.1 fallback:", error);

    // Fall back to v1.1
    const v1Client = auth.getV1Client();
    const searchResults = await v1Client.get("search/tweets.json", {
      q: query,  // Same query string, different API version
      count: Math.min(maxTweets, 100),
      since_id: sinceId,
    });

    for (const tweet of searchResults.statuses) {
      yield convertToTweet(tweet);
    }
  }
}

The hypothesis was sound. The implementation was clean. The testing was thorough.

The results were crushing.

Free tier blocks v1.1 read endpoints entirely. HTTP 403, error code 453, message: "You currently have access to a subset of X API V2 endpoints and limited v1.1 endpoints (e.g. media post, oauth) only."

{
  "errors": [{
    "message": "You currently have access to a subset of X API V2 endpoints and limited v1.1 endpoints (e.g. media post, oauth) only.",
    "code": 453
  }]
}

What v1.1 actually allows on free tier:

• ✅ POST tweet (media upload, OAuth)
• ❌ GET search/tweets
• ❌ GET statuses/home_timeline
• ❌ GET statuses/user_timeline

So: Can post, can't read. The v1.1 "fallback" was a beautiful piece of engineering that had absolutely no operational value.

Act 6: Acceptance (Nov 10 afternoon)

This is the part where you stop fighting reality and start working with it.

Final commit b4d5fb56f: "Enhanced error handling and logging."

Implemented graceful degradation instead of clever workarounds:

1. v1AccessDisabled flag - After the first 403/453 error, permanently disable v1.1 attempts. Don't retry in a loop like an optimistic fool.
2. getV1ClientIfAvailable() - Safe accessor that returns null if v1.1 is disabled, preventing crashes.
3. handleV1AccessError() - Detects access denial errors, logs appropriately, prevents infinite retry loops.
4. Return empty results - Instead of throwing exceptions when limits are hit, just return empty arrays and let the bot continue functioning.

handleV1AccessError(error: any, context: string): boolean {
  const isAccessError =
    error?.code === 403 ||
    error?.code === 453 ||
    error?.errors?.some?.(err => err?.code === 453);

  if (isAccessError) {
    if (!this.v1AccessDisabled) {
      this.v1AccessDisabled = true;  // Never try again this session
      elizaLogger.warn(
        `Twitter API v1.1 access disabled after "${context}" ` +
        `(insufficient account permissions). ` +
        `Future v1.1 fallbacks will be skipped.`
      );
    }
    return true;  // Stop retrying
  }
  return false;  // Different error, might be worth retrying
}

Final architecture:

• v2 API for all reads (100/month cap)
• v1.1 for posting (unlimited)
• Smart retry logic (no infinite loops)
• since_id optimization (only fetch new tweets)
• Graceful degradation when limits hit
• Aggressive deduplication (same post = counts once per day)
• Prayers

Does it work? Yes. Is it elegant? No. Is it the best possible solution? Probably not. Is it good enough? Absolutely.

The Scorecard

Twitter Free Tier Capabilities

Capability	Status	API Version	Notes
Posting tweets	✅ Works	v1.1	Unlimited
Reading posts	✅ Works	v2	100/month cap
Mention monitoring	✅ Works	v2	Only with since_id optimization
v1.1 search/timeline	❌ Blocked	v1.1	HTTP 403, error code 453
>100 reads/month	❌ Blocked	v2	Hard monthly cap
Real-time streaming	❌ Blocked	—	Not available on free tier

Optimization Results

Metric	Before	After	Change
Posts consumed per check	20 (re-fetching same tweets)	0-5 (only new tweets)	-95%
Monthly cap exhaustion	Under 1 hour	~1 month (barely)	Sustainable
Polling interval	120 seconds	300 seconds	+150%
Key optimization	None	since_id + -from:username	Server-side filtering
Engineering time	—	3 days	—

Lessons Learned

1. Read the docs - The free tier limitations were documented. I just didn't read carefully enough because who reads documentation carefully anymore?

2. since_id is magical - One parameter, 95% savings. Sometimes the solution is embarrassingly simple.

3. v1.1 is NOT an escape hatch - On free tier, only posting works. Reading is blocked. The clever workaround doesn't exist.

4. Graceful degradation beats crashes - Bot stays running even when limits are hit. Boring reliability is underrated.

5. Monthly caps ≠ rate limits - They require completely different optimization strategies. Rate limits you wait out; monthly caps you avoid hitting in the first place.

6. Going legit has costs - But they're predictable costs with documented limits, which beats the chaos of scraping even if it's more restrictive.

After 10 commits over three days, I built a system that technically works within Twitter's free tier by accepting that the free tier is essentially a trial mode masquerading as a feature. Which sounds like defeat but is actually just engineering: You work with the constraints you have, not the constraints you wish you had.

The bot posts. The bot reads mentions (if there aren't too many). The bot stays under 100 post retrievals per month (barely). It's not elegant, it's not robust, but it technically functions, which is what matters when you've spent three days optimizing API usage instead of building actual features.

The real lesson: The free tier exists so X can say they have a free tier. For anything beyond "hello world," you need Basic ($100/month, 15,000 posts) or higher. Everything I built was elaborate infrastructure to work within constraints that were designed to be unworkable.

And now it's documented, which means the next person—possibly Future Me—won't have to learn these lessons again.

Probably.

Epilogue: The Road Not Taken

After writing this post, I discovered TwitterInternalAPIDocument, a repository that documents Twitter's internal API endpoints. Apparently there are people who've been studying these patterns for years, reverse-engineering how the web client works, cataloging the internal architecture that Twitter doesn't officially expose.

Which raises interesting questions about alternative approaches to the problem I just spent three days solving.

The conventional wisdom says maintaining unofficial access requires significant ongoing effort—estimates range from 10-15 hours per month. But those estimates might come from building everything from scratch rather than leveraging existing research and automation. Hard to say without actually testing it.

Factor	Official API (chosen)	Unofficial/Internal API
Read limits	100 posts/month (documented)	Potentially unlimited (undocumented)
Stability	Predictable, stable	Can break without notice
Maintenance	Zero overhead	10-15 hours/month estimated
Compliance	Fully compliant	ToS gray area
Support	Official documentation	Community reverse-engineering
Risk	Known constraints	Unknown failure modes

I went with the official API because I needed something working today, not something that might work better after two weeks of experimentation. The 100 posts/month limit is barely sustainable, but it's a known constraint.

But I'm curious about what's possible with modern tooling and automation. The conventional wisdom might be right. Or it might be outdated assumptions that nobody's bothered to challenge. Can't know without trying.

For now, the bot works within official channels.

That's enough.

---

Footnotes

1. As opposed to The Way That Works But Violates ToS, which is what I'd been doing. ↩

2. Twitter's internal GraphQL API that powers the web interface—the one that's not documented, not officially supported, and definitely not meant for third-party use. The historical context: Twitter moved to GraphQL internally around 2019-2020 but kept it locked down. For years, libraries like agent-twitter-client reverse-engineered this access using cookie authentication, creating a "golden era" (pre-2023) where bots could operate without API keys. Then Twitter/X started a systematic crackdown: killed the free API tier (mid-2023), destabilized password logins (fall 2023), began rotating GraphQL doc_ids every 2-4 weeks, implemented TLS fingerprinting, tightened IP reputation scoring, and bound guest tokens to browser fingerprints. By 2025, maintaining GraphQL scrapers required 10-15 hours/month just to keep up with anti-scraping changes. My bot worked fine for months using this approach—until early November 2025, when it silently stopped working, joining the long list of casualties in Twitter's war against unofficial access. It's the API equivalent of sneaking in through the kitchen entrance, except the locks keep changing and eventually they welded the door shut. ↩

3. I had not, in fact, "done this before" with v2 of the Twitter API, which is different enough from v1.1 that experience is almost a liability because you think you know how it works. ↩