LlamaIndex

Integración con LlamaIndex

Integre CrawlForge MCP con LlamaIndex para crear conectores de datos, índices y motores de consulta con capacidades de web scraping. Perfecto para aplicaciones de RAG y bases de conocimiento.

Casos de uso

Conectores de datos web

Cree conectores de datos que obtienen e indexan contenido web automáticamente

Bases de conocimiento

Cree bases de conocimiento consultables a partir de páginas web y documentos

Motores de consulta

Cree motores de consulta con recuperación de datos web en tiempo real

Procesamiento de documentos

Extraiga y procese documentos desde URLs para indexarlos

Instalación

Instale LlamaIndex y el adaptador de CrawlForge MCP.

Bash

npm install llamaindex
npm install @crawlforge/llamaindex-adapter

También necesitará una API key de CrawlForge desde el panel.

Conector de datos web

Use CrawlForge como conector de datos para obtener y cargar documentos web.

Typescript

import { CrawlForgeReader } from '@crawlforge/llamaindex-adapter';
import { Document } from 'llamaindex';

// Initialize the reader
const reader = new CrawlForgeReader({
  apiKey: process.env.CRAWLFORGE_API_KEY!,
  tool: 'extract_content' // or 'extract_text', 'fetch_url'
});

// Load a single document
const documents = await reader.loadData(['https://example.com']);

console.log(documents[0].text);      // Document content
console.log(documents[0].metadata);  // URL, title, credits

// Load multiple documents
const urls = [
  'https://example.com/page1',
  'https://example.com/page2',
  'https://example.com/page3'
];

const allDocuments = await reader.loadData(urls);
console.log(`Loaded ${allDocuments.length} documents`);

Consejo: Use extract_content para una extracción limpia de artículos o extract_text para el texto completo de la página.

Índice de almacén vectorial

Cree un índice de almacén vectorial a partir de documentos web para búsqueda semántica.

Typescript

import { CrawlForgeReader } from '@crawlforge/llamaindex-adapter';
import { VectorStoreIndex } from 'llamaindex';
import { OpenAIEmbedding } from 'llamaindex';

// 1. Load documents from web
const reader = new CrawlForgeReader({
  apiKey: process.env.CRAWLFORGE_API_KEY!,
  tool: 'extract_content'
});

const documents = await reader.loadData([
  'https://example.com/doc1',
  'https://example.com/doc2',
  'https://example.com/doc3'
]);

// 2. Create embeddings
const embedModel = new OpenAIEmbedding({
  apiKey: process.env.OPENAI_API_KEY!,
  model: 'text-embedding-3-small'
});

// 3. Build vector index
const index = await VectorStoreIndex.fromDocuments(documents, {
  embedModel
});

// 4. Query the index
const queryEngine = index.asQueryEngine();
const response = await queryEngine.query(
  'What are the main topics covered?'
);

console.log(response.toString());

Motor de consulta con herramientas

Cree un motor de consulta que pueda obtener datos web en tiempo real bajo demanda.

Typescript

import { CrawlForgeTool } from '@crawlforge/llamaindex-adapter';
import { OpenAIAgent } from 'llamaindex';

// Create CrawlForge tools
const tools = [
  new CrawlForgeTool({
    name: 'web_search',
    description: 'Search the web for information',
    apiKey: process.env.CRAWLFORGE_API_KEY!,
    tool: 'search_web'
  }),
  new CrawlForgeTool({
    name: 'fetch_content',
    description: 'Fetch and extract content from a URL',
    apiKey: process.env.CRAWLFORGE_API_KEY!,
    tool: 'extract_content'
  }),
  new CrawlForgeTool({
    name: 'deep_research',
    description: 'Perform comprehensive research on a topic',
    apiKey: process.env.CRAWLFORGE_API_KEY!,
    tool: 'deep_research'
  })
];

// Create agent with tools
const agent = new OpenAIAgent({
  tools,
  verbose: true
});

// Query with tool access
const response = await agent.chat(
  'Research the latest developments in quantum computing'
);

console.log(response.toString());

Consejos para agentes: El agente elegirá automáticamente qué herramientas usar según la consulta. Establezca verbose=true para ver la selección de herramientas.

Recuperador web personalizado

Cree un recuperador personalizado que obtenga datos web según las consultas.

Typescript

import { CrawlForgeReader } from '@crawlforge/llamaindex-adapter';
import { BaseRetriever } from 'llamaindex';
import type { NodeWithScore } from 'llamaindex';

export class WebRetriever extends BaseRetriever {
  private reader: CrawlForgeReader;

  constructor(apiKey: string) {
    super();
    this.reader = new CrawlForgeReader({
      apiKey,
      tool: 'search_web'
    });
  }

  async retrieve(query: string): Promise<NodeWithScore[]> {
    // 1. Search for URLs
    const searchResults = await this.reader.search(query);

    // 2. Fetch content from top results
    const urls = searchResults.slice(0, 3).map(r => r.url);
    const documents = await this.reader.loadData(urls);

    // 3. Convert to nodes with scores
    return documents.map((doc, i) => ({
      node: doc,
      score: 1.0 - (i * 0.1) // Simple scoring
    }));
  }
}

// Use the custom retriever
const retriever = new WebRetriever(process.env.CRAWLFORGE_API_KEY!);
const nodes = await retriever.retrieve('latest AI news');

console.log(`Retrieved ${nodes.length} documents`);

Procesamiento por lotes con async

Procese múltiples URLs de forma eficiente con operaciones por lotes asíncronas.

Typescript

import { CrawlForgeReader } from '@crawlforge/llamaindex-adapter';
import { VectorStoreIndex } from 'llamaindex';

const reader = new CrawlForgeReader({
  apiKey: process.env.CRAWLFORGE_API_KEY!,
  tool: 'batch_scrape' // Use batch tool for efficiency
});

// Define URL batches
const urlBatches = [
  ['https://example.com/1', 'https://example.com/2'],
  ['https://example.com/3', 'https://example.com/4'],
  ['https://example.com/5', 'https://example.com/6']
];

// Process in parallel
const allDocuments = await Promise.all(
  urlBatches.map(urls => reader.loadData(urls))
);

const documents = allDocuments.flat();

console.log(`Loaded ${documents.length} documents`);

// Build index from all documents
const index = await VectorStoreIndex.fromDocuments(documents);

console.log('Index created successfully');

Consejo de rendimiento: Use batch_scrape para procesar múltiples URLs: está optimizado para la ejecución en paralelo y cuesta solo 1 credit por URL.

Buenas prácticas

Elija herramientas eficientes — Use batch_scrape para múltiples URLs, extract_content para texto limpio
Implemente caché — Almacene en caché los documentos indexados para evitar obtenciones redundantes y ahorrar credits
Use operaciones asíncronas — Aproveche async/await para el procesamiento en paralelo y acelerar las operaciones masivas
Monitoree los credits — Rastree el uso de credits en los metadatos de los documentos y configure alertas en su panel

¿Listo para crear con LlamaIndex?

Explore las 23 herramientas de CrawlForge o consulte otras integraciones.

Ver todas las herramientas Integración con LangChain

import { CrawlForgeReader } from '@crawlforge/llamaindex-adapter'; import { Document } from 'llamaindex'; // Initialize the reader const reader = new CrawlForgeReader({ apiKey: process.env.CRAWLFORGE_API_KEY!, tool: 'extract_content' // or 'extract_text', 'fetch_url' }); // Load a single document const documents = await reader.loadData(['https://example.com']); console.log(documents[0].text); // Document content console.log(documents[0].metadata); // URL, title, credits // Load multiple documents const urls = [ 'https://example.com/page1', 'https://example.com/page2', 'https://example.com/page3' ]; const allDocuments = await reader.loadData(urls); console.log(`Loaded ${allDocuments.length} documents`);

import { CrawlForgeReader } from '@crawlforge/llamaindex-adapter'; import { VectorStoreIndex } from 'llamaindex'; import { OpenAIEmbedding } from 'llamaindex'; // 1. Load documents from web const reader = new CrawlForgeReader({ apiKey: process.env.CRAWLFORGE_API_KEY!, tool: 'extract_content' }); const documents = await reader.loadData([ 'https://example.com/doc1', 'https://example.com/doc2', 'https://example.com/doc3' ]); // 2. Create embeddings const embedModel = new OpenAIEmbedding({ apiKey: process.env.OPENAI_API_KEY!, model: 'text-embedding-3-small' }); // 3. Build vector index const index = await VectorStoreIndex.fromDocuments(documents, { embedModel }); // 4. Query the index const queryEngine = index.asQueryEngine(); const response = await queryEngine.query( 'What are the main topics covered?' ); console.log(response.toString());

import { CrawlForgeTool } from '@crawlforge/llamaindex-adapter'; import { OpenAIAgent } from 'llamaindex'; // Create CrawlForge tools const tools = [ new CrawlForgeTool({ name: 'web_search', description: 'Search the web for information', apiKey: process.env.CRAWLFORGE_API_KEY!, tool: 'search_web' }), new CrawlForgeTool({ name: 'fetch_content', description: 'Fetch and extract content from a URL', apiKey: process.env.CRAWLFORGE_API_KEY!, tool: 'extract_content' }), new CrawlForgeTool({ name: 'deep_research', description: 'Perform comprehensive research on a topic', apiKey: process.env.CRAWLFORGE_API_KEY!, tool: 'deep_research' }) ]; // Create agent with tools const agent = new OpenAIAgent({ tools, verbose: true }); // Query with tool access const response = await agent.chat( 'Research the latest developments in quantum computing' ); console.log(response.toString());

import { CrawlForgeReader } from '@crawlforge/llamaindex-adapter'; import { BaseRetriever } from 'llamaindex'; import type { NodeWithScore } from 'llamaindex'; export class WebRetriever extends BaseRetriever { private reader: CrawlForgeReader; constructor(apiKey: string) { super(); this.reader = new CrawlForgeReader({ apiKey, tool: 'search_web' }); } async retrieve(query: string): Promise<NodeWithScore[]> { // 1. Search for URLs const searchResults = await this.reader.search(query); // 2. Fetch content from top results const urls = searchResults.slice(0, 3).map(r => r.url); const documents = await this.reader.loadData(urls); // 3. Convert to nodes with scores return documents.map((doc, i) => ({ node: doc, score: 1.0 - (i * 0.1) // Simple scoring })); } } // Use the custom retriever const retriever = new WebRetriever(process.env.CRAWLFORGE_API_KEY!); const nodes = await retriever.retrieve('latest AI news'); console.log(`Retrieved ${nodes.length} documents`);

import { CrawlForgeReader } from '@crawlforge/llamaindex-adapter'; import { VectorStoreIndex } from 'llamaindex'; const reader = new CrawlForgeReader({ apiKey: process.env.CRAWLFORGE_API_KEY!, tool: 'batch_scrape' // Use batch tool for efficiency }); // Define URL batches const urlBatches = [ ['https://example.com/1', 'https://example.com/2'], ['https://example.com/3', 'https://example.com/4'], ['https://example.com/5', 'https://example.com/6'] ]; // Process in parallel const allDocuments = await Promise.all( urlBatches.map(urls => reader.loadData(urls)) ); const documents = allDocuments.flat(); console.log(`Loaded ${documents.length} documents`); // Build index from all documents const index = await VectorStoreIndex.fromDocuments(documents); console.log('Index created successfully');