On this page
Zed is fast. Zed AI is powerful. But Zed AI cannot see the web -- until you plug in an MCP server. CrawlForge adds 20 scraping tools that Zed AI can invoke directly from the Assistant panel.
This guide walks you through every step to scrape websites in Zed AI, with real code and a full troubleshooting section.
Table of Contents
- Why Web Scraping in Zed AI Matters
- Prerequisites
- Step 1: Install CrawlForge MCP
- Step 2: Configure Zed's Context Server
- Step 3: Restart Zed and Verify
- Step 4: First Scrape from the Assistant
- Full Working Example: Scrape an API Reference
- Advanced: Stealth and Structured Extraction
- Troubleshooting
- FAQ
Why Web Scraping in Zed AI Matters
Zed AI lives in the editor. When you can scrape websites in Zed AI without leaving the Assistant, patterns like these become cheap:
- Read a third-party SDK doc, then generate a wrapper in your project's style.
- Fetch a competitor's blog post, then summarize the architecture decisions.
- Pull a sitemap, then flag pages your crawler has not visited.
Zed's MCP support (called "Context Servers" in settings) launched in 2024 and uses the same open protocol as Claude Desktop and Cursor. CrawlForge exposes 20 scraping tools over MCP, so they appear in Zed's slash-command menu with zero glue code. For protocol background see our MCP protocol explainer.
Prerequisites
- Zed 0.148+ -- download from zed.dev
- Node.js 18+
- CrawlForge account -- free at crawlforge.dev/signup, 1,000 credits included
Step 1: Install CrawlForge MCP
Step 2: Configure Zed's Context Server
Open Zed settings with Cmd+, (macOS) or Ctrl+, (Linux/Windows). Zed settings live at ~/.config/zed/settings.json. Add a context_servers block:
Replace cf_live_your_key_here with the key from crawlforge.dev/dashboard/api-keys.
Step 3: Restart Zed and Verify
- Fully quit Zed (
Cmd+Q). - Reopen and open the Assistant panel (
Cmd+?). - Type
/in the Assistant prompt. You should see CrawlForge tools in the suggestion list:/fetch_url,/scrape_structured,/search_web, and others.
Step 4: First Scrape from the Assistant
In the Assistant panel, try:
Use CrawlForge to fetch https://news.ycombinator.com and return the top 5 story titles.
Zed calls fetch_url (1 credit), streams the HTML back, then Zed AI extracts the titles. Response appears inline in the Assistant panel, ready to paste into a buffer.
Full Working Example: Scrape an API Reference
Say you are integrating with the Stripe API and want the full PaymentIntent field list as TypeScript. Paste this prompt:
Use CrawlForge's extract_content on https://docs.stripe.com/api/payment_intents/object.
Then write a TypeScript interface that matches every documented field.
Zed issues this call under the hood:
Cost: 2 credits. Zed AI reads the markdown and produces:
Paste it into a buffer and you have type-safe Stripe integration in seconds.
Advanced: Stealth and Structured Extraction
Hit a Cloudflare wall? Swap fetch_url for stealth_mode:
Use CrawlForge's stealth_mode on https://locked-down-site.example.com and return the main article text.
Cost jumps from 1 to 5 credits but bypasses most anti-bot systems.
Want CSS-selector precision? Use scrape_structured:
Use scrape_structured on https://ycombinator.com/companies with:
- name: .company-name
- batch: .batch-tag
- description: .company-description
Zed AI returns a typed array. 2 credits per run.
Credit Reference
| Tool | Credits | Use when |
|---|---|---|
fetch_url | 1 | Static HTML |
extract_content | 2 | Readable article / doc |
scrape_structured | 2 | CSS-selector fields |
search_web | 5 | URL unknown |
stealth_mode | 5 | Anti-bot protection |
scrape_with_actions | 5 | SPA with interactions |
deep_research | 10 | Multi-source synthesis |
Troubleshooting
Tools missing from slash menu -- Context server config not loaded. Check ~/.config/zed/logs/Zed.log for context_servers parse errors. A trailing comma in JSON silently disables the whole block.
"Failed to spawn crawlforge-mcp-server" -- Zed cannot find the binary. Replace "path": "crawlforge-mcp-server" with the absolute path from which crawlforge-mcp-server, for example "path": "/usr/local/bin/crawlforge-mcp-server".
401 Unauthorized on every call -- API key is wrong or unset. Verify with curl -H "Authorization: Bearer $CRAWLFORGE_API_KEY" https://crawlforge.dev/api/v1/credits/balance. Expected response includes your remaining credits.
Assistant picks the wrong tool -- Explicitly name it: "Use CrawlForge's scrape_structured..." instead of "scrape this page." Zed AI defaults to the cheapest match when the intent is ambiguous.
Zed freezes on long scrapes -- Deep research can run for 60+ seconds. Zed shows a spinner but input is still responsive; wait for completion rather than canceling.
Next Steps
- Explore all 20 CrawlForge tools you can call from Zed
- Read the complete MCP web scraping guide for architecture patterns
- See getting started docs for API reference
- Compare to Firecrawl alternative if you are evaluating vendors
Start free at crawlforge.dev/signup -- 1,000 credits, no credit card.
