How to Use CrawlForge with OpenAI Agents SDK

OpenAI's Agents SDK provides a production-ready framework for building autonomous AI agents with tool use, handoffs, and guardrails. CrawlForge adds the missing piece: live web access. By connecting CrawlForge's 18 scraping tools to your OpenAI agents, you enable them to search the web, extract structured data, read documentation, and conduct multi-source research -- all within the Agents SDK's orchestration framework.

This guide shows you how to define CrawlForge tools as OpenAI agent functions and build agents that act on real-time web data.

Table of Contents

• Prerequisites
• Architecture: CrawlForge + OpenAI Agents
• Step 1: Create the CrawlForge Tool Functions
• Step 2: Build a Web Research Agent
• Step 3: Add Structured Data Extraction
• Advanced: Multi-Agent Web Pipeline
• Credit Cost Breakdown
• Best Practices
• Next Steps

Prerequisites

Get your CrawlForge API key at crawlforge.dev/signup -- 1,000 free credits included.

Architecture: CrawlForge + OpenAI Agents

The OpenAI Agents SDK uses a tool pattern similar to the function calling API but with richer orchestration. You define tools as functions with JSON Schema parameters, and the agent decides when and how to call them.

User Query -> OpenAI Agent -> Tool Selection -> CrawlForge API -> Results -> Agent Response

CrawlForge's REST API at https://crawlforge.dev/api/v1/tools/ maps cleanly to the Agents SDK's tool definition format. Each tool becomes a function the agent can invoke.

Step 1: Create the CrawlForge Tool Functions

First, create a reusable CrawlForge client and tool definitions:

Step 2: Build a Web Research Agent

Create an agent that uses CrawlForge tools to research topics:

The agent will autonomously:

1. Call search_web to find relevant articles (5 credits)
2. Call extract_content on the top results (2 credits each)
3. Synthesize a cited summary

Step 3: Add Structured Data Extraction

Build a data extraction agent that pulls specific fields from web pages:

Advanced: Multi-Agent Web Pipeline

The Agents SDK supports handoffs between specialized agents. Build a pipeline where a researcher finds sources and hands off to an analyst:

This pipeline separates concerns: the collector gathers data (using CrawlForge credits), and the analyst processes it (no credits needed). Total cost depends on sources fetched -- typically 15-25 credits for a 3-source comparison.

Credit Cost Breakdown

The CrawlForge Free tier (1,000 credits) supports roughly 90 search-and-extract workflows per month. The Professional plan ($99/month, 50,000 credits) handles production agent workloads.

Best Practices

Choose the cheapest tool first. The agent's instructions should guide it toward fetch_url (1 credit) when full HTML is acceptable, and extract_content (2 credits) only when clean text is needed. Reserve deep_research (10 credits) for complex multi-source queries.

Limit agent steps. Set a maximum number of tool invocations to control costs. Most research tasks complete in 3-5 tool calls.

Use handoffs for complex pipelines. Rather than one agent with many tools, split responsibilities. The collector agent handles web access (credits), while the analyst agent processes data (no credits).

Cache tool outputs. If your agent repeatedly accesses the same URL, implement response caching to avoid duplicate credit charges.

Monitor usage. Check your credit consumption in the CrawlForge dashboard and set alerts for unexpected spikes.

Next Steps

You now have OpenAI agents that can access live web data. Continue building:

• 18 CrawlForge tools overview -- register more tools for your agents
• Stealth mode scraping -- access sites with anti-bot protection
• Deep research automation -- use the 10-credit deep_research tool for comprehensive reports
• CrawlForge Quick Start -- full MCP setup guide

---

Give your OpenAI agents eyes on the web. Start free with 1,000 credits -- no credit card required.