CrawlForge MCP v5.1.0: Search Reddit Without the API

Ask any scraper developer what the hardest mainstream site is in 2026 and Reddit is a safe answer. We would know — reddit.com blocks every direct access path CrawlForge has, from a plain fetch to a full stealth browser. So CrawlForge MCP v5.1.0 stops fighting the wall and goes around it: the new reddit_search tool — our 28th — searches Reddit posts and comments and reads entire threads through community-run archives, with no Reddit API key, no OAuth app, and no credentials of any kind.

Table of Contents

• What Shipped
• Why reddit.com Cannot Be Scraped
• Meet reddit_search
• The Archives: Arctic Shift and PullPush
• How Routing Works
• Honest Caveats
• Using It From MCP and REST
• What About the Reddit Thread Template
• Also Since v5.0.4
• Credit Cost
• How to Upgrade

What Shipped

v5.1.0 is a focused minor release with one headline:

• reddit_search — search Reddit posts (title + selftext) and comments, or read a post plus its nested comment tree, via the Arctic Shift and PullPush community archives. Three modes, scoped filters, normalized output. 2 credits per call.
• Tool count goes from 27 to 28, and the MCP server card, tool instructions, and getting-started prompt all know about the newcomer.
• 31 new unit tests (1,018 total in the suite) and 100% MCP protocol compliance across all 28 tools, live-verified over MCP stdio returning real r/ClaudeAI posts.

Why reddit.com Cannot Be Scraped

We did not reach for archives out of preference — we measured the wall first. Before building the tool we ran every direct path against reddit.com, live:

• fetch_url with a full browser User-Agent: 403
• scrape against old.reddit.com: 403
• scrape_template with the reddit-thread template: 403
• stealth_mode at its most advanced level: 403

The block is layered — datacenter IP reputation first, TLS fingerprinting at the handshake, then a JavaScript challenge — which is why even a stealth browser on a server fails. And the sanctioned door narrowed too: Reddit closed self-service API signup in November 2025 under its Responsible Builder Policy, so new applications for official API credentials go through an approval request rather than a signup form. Most Reddit MCP servers wrap that official API, which now makes "get credentials" the hardest step of their setup.

reddit_search sidesteps all of it. It never sends a single request to reddit.com.

Meet reddit_search

One tool, three modes:

• posts (default) — keyword search across post titles and selftext, optionally scoped to a subreddit or author. Supports "quoted phrases", OR, and -exclusion.
• comments — full-text search across comment bodies, something the official Reddit API cannot do at all.
• thread — hand it a post ID and get the post plus its nested comment tree, with reddit-style collapse markers for branches too deep to expand.

Filters cover subreddit, author, after/before dates (ISO 8601, epoch seconds, or offsets like "7d"), limit (up to 100), and sort. Results come back normalized: full reddit.com permalinks, ISO dates, scores, comment counts, and text capped at 2,000 characters with truncation flags so a 100-result payload stays friendly to an LLM context window.

A typical call from an MCP client looks like this:

{
  "tool": "reddit_search",
  "arguments": {
    "query": "best mcp servers",
    "subreddit": "ClaudeAI",
    "mode": "posts",
    "limit": 10
  }
}

Take any post id from the results, pass it back as link_id with mode: "thread", and you have the whole discussion — ready for summarize_content or analyze_content to turn into sentiment, topics, and a briefing.

The Archives: Arctic Shift and PullPush

If you remember Pushshift — the beloved Reddit research archive that lost its API access in 2023 — these two projects are its successors, and they are what reddit_search queries:

• Arctic Shift ingests Reddit in near-real-time. During live testing it returned a post created the same day, and it serves proper nested comment trees. Its one documented constraint: keyword search must be scoped to a subreddit or author.
• PullPush is Pushshift-compatible and does what Arctic Shift will not — full-text search across all of Reddit. The trade-offs: documented gaps in its post-2023 archive and aggressive rate limits.

Both are free, both are community-run, and neither needs credentials. Every response includes provenance notes naming the archive that answered, so downstream consumers always know where the data came from.

How Routing Works

You never pick a backend unless you want to. In the default auto mode:

• Scoped searches (a subreddit or author filter) and thread reads go to Arctic Shift — the fresher archive — with PullPush as an error-only fallback. When the fallback fires, the response says so in a fallback_used field.
• Unscoped keyword searches across all of Reddit go to PullPush, because that is the only archive that supports them.

The plumbing absorbs the archives' quirks: every request carries an identifying User-Agent (live testing showed Arctic Shift throttles anonymous clients into a shared bucket), transient throttle responses get one bounded retry, and PullPush's rate-limit messages pass through verbatim so you see the real reason instead of a generic error. A REDDIT_SEARCH_TIMEOUT_MS environment variable overrides the 30-second per-request cap if your pipeline needs a different budget.

Honest Caveats

Archive data has edges, and we would rather document them than let you find them:

• Very fresh content reads low. The archives capture posts the moment they appear, so scores and comment counts of content less than ~36 hours old often read 0 or 1. The content is there; the vote tallies have not caught up.
• PullPush has post-2023 gaps. An empty unscoped search does not prove the content does not exist. Scope to a subreddit or author when you can — it routes you to the more complete archive.
• Deleted content may persist in archives. That is inherent to how archives work, and it cuts both ways: it is also why researchers use them.

Using It From MCP and REST

In an MCP client — Claude Desktop, Claude Code, Cursor — just ask: "Search r/webdev for discussions about scraping infrastructure from the last month and summarize the top complaints." The agent picks reddit_search, scopes it, and chains the summary itself.

From the REST API, it is one authenticated POST:

const response = await fetch('https://crawlforge.dev/api/v1/tools/reddit_search', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    query: 'best mechanical keyboard',
    subreddit: 'MechanicalKeyboards',
    mode: 'posts',
    limit: 10,
  }),
});

const { data } = await response.json();
console.log(`${data.count} posts via ${data.source}`);

You can also try it with zero setup in the playground — reddit_search is live there alongside the other 27 tools.

What About the Reddit Thread Template

scrape_template still ships its reddit-thread template, but it points at reddit.com — the very wall this release routes around — so it no longer works reliably. The scrape_template docs now say exactly that and point Reddit work at reddit_search instead. The other nine templates (Amazon, GitHub, YouTube, Hacker News, and friends) are unaffected.

Also Since v5.0.4

If you skipped the patch between releases, v5.0.5 fixed two numbers serp_rank had been asserting wrongly: the DataForSEO request timeout (now 120 seconds by default, tunable via DATAFORSEO_TIMEOUT_MS, because live Google scrapes at depth 100 routinely take 30-60+ seconds) and the documented upstream cost (the old figure was the depth-10 price, 10× lower than the depth-100 default actually bills). It also fixed the Smithery listing, which now derives its tool table from the live registry instead of a hand-written card. The full story of the v5.0.x hardening cycle is in the v5.0.4 live-testing post.

Credit Cost

reddit_search costs 2 credits per call — search or full thread read, same price. For scale: the free plan's 1,000 starter credits cover 500 Reddit searches, and a search plus a thread read plus summarize_content on the result is 8 credits end to end. As always, failed requests are not charged.

How to Upgrade

npm install -g crawlforge-mcp-server@latest
crawlforge --version   # 5.1.0

Or, if your MCP client launches the server with npx, it picks up 5.1.0 automatically on the next restart. No schema, output-shape, or credit-cost changes to any existing tool — this is a drop-in upgrade.

Ready to mine Reddit discussions without an API key? Start free with 1,000 credits — enough for 500 searches — and read the reddit_search API reference for every parameter and mode.