# CrawlForge MCP - Full Documentation
> Enterprise web scraping MCP server with 27 specialized tools for AI applications

## Overview

CrawlForge MCP is a Model Context Protocol server providing 27 web scraping and data extraction tools. It integrates natively with Claude, Cursor, LangChain, and any MCP-compatible AI framework.

### v4.2.2 Highlights

The latest release adds Ollama-default LLM extraction (run `extract_with_llm` against a local Ollama model with no API key required, while OpenAI and Anthropic remain optional), a standalone CLI with 15 commands for running any tool from the terminal, MCP Resources and Prompts for richer client surfaces, and `scrape_template` pre-built scrapers for 10 popular sites (GitHub, Amazon, YouTube, and more).

- Website: https://www.crawlforge.dev
- Documentation: https://www.crawlforge.dev/docs
- API Reference: https://www.crawlforge.dev/docs/api-reference
- Pricing: https://www.crawlforge.dev/pricing
- Getting Started: https://www.crawlforge.dev/docs/getting-started
- Concise Summary: https://www.crawlforge.dev/llms.txt

### Localized Versions

CrawlForge is available in English (default), Spanish, Simplified Chinese, and Malay. The site mirrors the English structure under locale path prefixes:

- English (default): https://www.crawlforge.dev/
- Español: https://www.crawlforge.dev/es/
- 简体中文 (zh-Hans): https://www.crawlforge.dev/zh/
- Bahasa Melayu (ms-SG): https://www.crawlforge.dev/ms/

Marketing, pricing, legal, and documentation pages are fully translated (blog posts are English-only). Every page declares hreflang alternates; the XML sitemap at https://www.crawlforge.dev/sitemap.xml lists all localized URLs.

## Authentication

All API requests require an API key sent via the `X-API-Key` header.

```
curl -X POST https://www.crawlforge.dev/api/v1/tools/fetch_url \
  -H "X-API-Key: cf_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

API key formats:
- `cf_test_...` - Test/development environment
- `cf_live_...` - Production environment

## Pricing Plans

| Plan | Price | Monthly Credits |
|------|-------|----------------|
| Free | $0 | 1,000 one-time credits |
| Hobby | $19/mo | 5,000 |
| Professional | $99/mo | 50,000 |
| Business | $399/mo | 250,000 |
| Enterprise | Custom | Unlimited |

One-time credit packs available from $3 (1,000 credits) with no subscription.

Credit add-ons:
- 1K credits: $3
- 5K credits: $14
- 10K credits: $27
- 25K credits: $62.50

## Tools Reference

### fetch_url
- **Cost:** 1 credit
- **Description:** Fetch and return the raw content of a webpage
- **Parameters:**
  - `url` (string, required): The URL to fetch
  - `timeout` (number, optional): Request timeout in ms (default: 30000)
  - `headers` (object, optional): Custom request headers
- **Use when:** You need the raw HTML or text content of a single page

### extract_text
- **Cost:** 1 credit
- **Description:** Extract clean, readable text from a URL, stripping HTML tags and boilerplate
- **Parameters:**
  - `url` (string, required): The URL to extract text from
  - `include_links` (boolean, optional): Include hyperlink URLs in output
- **Use when:** You need human-readable text content without HTML markup

### extract_links
- **Cost:** 1 credit
- **Description:** Discover and return all links found on a webpage
- **Parameters:**
  - `url` (string, required): The URL to extract links from
  - `filter` (string, optional): Regex pattern to filter links
  - `include_external` (boolean, optional): Include external links (default: true)
- **Use when:** You need to discover navigation structure or find specific URLs on a page

### extract_metadata
- **Cost:** 1 credit
- **Description:** Extract page metadata including title, description, Open Graph, and structured data
- **Parameters:**
  - `url` (string, required): The URL to extract metadata from
- **Use when:** You need SEO data, social media tags, or structured data from a page

### scrape_structured
- **Cost:** 2 credits
- **Description:** Extract specific data from a page using CSS selectors
- **Parameters:**
  - `url` (string, required): The URL to scrape
  - `selectors` (object, required): Map of field names to CSS selectors
  - `multiple` (boolean, optional): Return multiple matches per selector
- **Use when:** You need specific data points from a known page structure (prices, titles, etc.)

### scrape_template
- **Cost:** 1 credit
- **Description:** Run a pre-built scraper template for 10 popular sites (GitHub, Amazon, YouTube, Hacker News, Reddit, Stack Overflow, Wikipedia, eBay, IMDb, Product Hunt)
- **Parameters:**
  - `url` (string, required): URL on the supported site
  - `template` (string, required): Template name (e.g. "github_repo", "amazon_product", "youtube_video")
  - `fields` (string[], optional): Subset of template fields to return
- **Use when:** Target site is one of the supported templates and you want clean structured data without writing selectors

### extract_content
- **Cost:** 2 credits
- **Description:** Smart content extraction that identifies and returns the main content of a page
- **Parameters:**
  - `url` (string, required): The URL to extract content from
  - `format` (string, optional): Output format - "markdown", "text", or "html" (default: "markdown")
- **Use when:** You need the primary article/content from a page, ignoring navigation and ads

### map_site
- **Cost:** 2 credits
- **Description:** Map the structure of a website by crawling and analyzing its link structure
- **Parameters:**
  - `url` (string, required): The starting URL to map from
  - `max_depth` (number, optional): Maximum crawl depth (default: 3)
  - `max_pages` (number, optional): Maximum pages to visit (default: 100)
- **Use when:** You need to understand a website's structure before targeted scraping

### process_document
- **Cost:** 2 credits
- **Description:** Process PDF and other document formats, extracting text and metadata
- **Parameters:**
  - `url` (string, required): URL of the document to process
  - `extract_tables` (boolean, optional): Attempt to extract tables (default: false)
  - `extract_images` (boolean, optional): Extract image descriptions (default: false)
- **Use when:** You need to extract content from PDFs, Word docs, or other document formats

### search_web
- **Cost:** 5 credits
- **Description:** Search the web using Google or DuckDuckGo and return results
- **Parameters:**
  - `query` (string, required): Search query
  - `num_results` (number, optional): Number of results to return (default: 10)
  - `engine` (string, optional): Search engine - "google" or "duckduckgo" (default: "google")
  - `country` (string, optional): Country code for localized results
- **Use when:** You need to find relevant web pages for a topic or query

### serp_rank
- **Cost:** 5 credits
- **Description:** Check where a domain ranks in Google's organic search results for a keyword — the real SERP position, not Custom Search order. Returns the target's organic rank, the ranking URL, and every position it holds. Powered by DataForSEO.
- **Parameters:**
  - `keyword` (string, required): The search query to check ranking for
  - `target` (string, required): Domain or URL to locate in the results
  - `depth` (number, optional): How many results to scan, 10-200 (default: 100; 100 = 1 page of cost)
  - `device` (string, optional): Device to emulate - "desktop" or "mobile" (default: "desktop")
  - `location_name` (string, optional): Location, e.g. "United States"
  - `location_code` (number, optional): Numeric DataForSEO location code (overrides location_name)
  - `language_code` (string, optional): Language code, e.g. "en"
- **Use when:** You need to track a domain's real organic Google position for target keywords

### crawl_deep
- **Cost:** 4 credits
- **Description:** Deep crawl a website, following links and extracting content from multiple pages
- **Parameters:**
  - `url` (string, required): Starting URL
  - `max_pages` (number, optional): Maximum pages to crawl (default: 50)
  - `max_depth` (number, optional): Maximum link depth (default: 3)
  - `include_patterns` (string[], optional): URL patterns to include
  - `exclude_patterns` (string[], optional): URL patterns to exclude
- **Use when:** You need content from multiple pages across a website

### summarize_content
- **Cost:** 4 credits
- **Description:** AI-powered content summarization with configurable length and style
- **Parameters:**
  - `url` (string, required): URL of content to summarize
  - `max_length` (number, optional): Maximum summary length in words
  - `style` (string, optional): Summary style - "brief", "detailed", "bullet_points"
- **Use when:** You need a concise summary of long-form content

### analyze_content
- **Cost:** 3 credits
- **Description:** Analyze content for sentiment, language, readability, and key topics
- **Parameters:**
  - `url` (string, required): URL of content to analyze
  - `analyses` (string[], optional): Types of analysis - "sentiment", "language", "readability", "topics"
- **Use when:** You need content intelligence (sentiment, topics, language detection)

### batch_scrape
- **Cost:** 5 credits
- **Description:** Process multiple URLs in a single request for efficient bulk scraping
- **Parameters:**
  - `urls` (string[], required): Array of URLs to scrape (max 100)
  - `extract` (string, optional): What to extract - "text", "metadata", "links", "content"
- **Use when:** You have multiple URLs to process and want to minimize API calls

### scrape_with_actions
- **Cost:** 5 credits
- **Description:** Scrape dynamic pages using browser automation (clicks, form fills, scrolling)
- **Parameters:**
  - `url` (string, required): Starting URL
  - `actions` (object[], required): Array of browser actions to perform
    - Action types: "click", "type", "scroll", "wait", "screenshot"
    - Each action has `selector` (CSS selector) and action-specific params
  - `extract_after` (boolean, optional): Extract content after actions complete
- **Use when:** Content requires interaction (login, infinite scroll, dynamic loading)

### deep_research
- **Cost:** 10 credits
- **Description:** Multi-stage AI research combining web search, content extraction, and synthesis
- **Parameters:**
  - `query` (string, required): Research query or topic
  - `depth` (string, optional): Research depth - "quick", "standard", "comprehensive"
  - `max_sources` (number, optional): Maximum sources to consult (default: 10)
  - `verify_sources` (boolean, optional): Cross-reference facts across sources
- **Use when:** You need thorough research on a topic with source verification

### track_changes
- **Cost:** 3 credits
- **Description:** Track content changes with baseline capture, scheduled monitoring, and alerts
- **Parameters:**
  - `url` (string, required): URL to track
  - `operation` (string, optional): "compare" | "create_baseline" | "monitor" | "get_history" | "get_stats"
  - `trackingOptions` (object, optional): Granularity, ignoreWhitespace, trackText, trackLinks, trackStructure
  - `monitoringOptions` (object, optional): Scheduled interval, webhook, notification thresholds
- **Use when:** You need to detect and track website changes over time

### extract_structured
- **Cost:** 3 credits
- **Description:** LLM-powered schema-driven extraction with CSS selector fallback
- **Parameters:**
  - `url` (string, required): URL to extract from
  - `schema` (object, required): JSON Schema describing the target data shape
  - `prompt` (string, optional): Natural-language extraction instructions
  - `llmConfig` (object, optional): Provider and API key for LLM extraction
  - `selectorHints` (object, optional): CSS selectors guiding the fallback
- **Use when:** You want typed structured output without writing bespoke selectors

### generate_llms_txt
- **Cost:** 5 credits
- **Description:** Analyze a site and generate standard-compliant llms.txt and llms-full.txt
- **Parameters:**
  - `url` (string, required): Website to analyze
  - `format` (string, optional): "both" | "llms-txt" | "llms-full-txt"
  - `complianceLevel` (string, optional): "basic" | "standard" | "strict"
  - `analysisOptions` (object, optional): maxDepth, maxPages, detectAPIs, analyzeContent
  - `outputOptions` (object, optional): organizationName, contactEmail, customGuidelines, customRestrictions
- **Use when:** Publishing AI interaction guidelines for your site

### stealth_mode
- **Cost:** 5 credits
- **Description:** Anti-detection browsing with fingerprint randomization and proxy rotation
- **Parameters:**
  - `url` (string, required): URL to access in stealth mode
  - `proxy_country` (string, optional): Country code for proxy (default: "US")
  - `fingerprint` (string, optional): Browser fingerprint profile
- **Use when:** Target sites have anti-bot protection or block automated requests

### localization
- **Cost:** 2 credits
- **Description:** Access geo-targeted content from 26 countries with automatic localization
- **Parameters:**
  - `url` (string, required): URL to access
  - `country` (string, required): Target country code (e.g., "US", "GB", "JP", "DE")
  - `language` (string, optional): Preferred language code
- **Supported countries:** US, GB, CA, AU, DE, FR, ES, IT, NL, SE, NO, DK, FI, PL, CZ, AT, CH, BE, PT, IE, JP, KR, CN, IN, BR, MX
- **Use when:** You need location-specific content or pricing from different regions

### extract_with_llm
- **Cost:** 3 credits
- **Description:** LLM-powered extraction that returns structured data described in natural language. Defaults to local Ollama (no API key required); OpenAI and Anthropic are optional alternatives.
- **Parameters:**
  - `url` (string, required): URL to extract from
  - `prompt` (string, required): Natural-language description of the data to extract
  - `schema` (object, optional): JSON Schema describing the target shape
  - `provider` (string, optional): "ollama" (default) | "openai" | "anthropic"
  - `model` (string, optional): Provider-specific model name
- **Use when:** You want LLM-driven extraction without committing to a paid provider, or you need to swap providers per request

### list_ollama_models
- **Cost:** 0 credits (free)
- **Description:** List the local Ollama models available to `extract_with_llm`. Useful for discovering installed models before running an extraction.
- **Parameters:** None
- **Use when:** You want to confirm which Ollama models are installed locally before calling `extract_with_llm`

## Integration Examples

### Claude Desktop (MCP)

Add to your Claude Desktop MCP configuration:

```json
{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "@crawlforge/mcp-server"],
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_YOUR_API_KEY"
      }
    }
  }
}
```

### Cursor IDE

Add to `.cursor/mcp.json`:

```json
{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "@crawlforge/mcp-server"],
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_YOUR_API_KEY"
      }
    }
  }
}
```

### TypeScript / Node.js

```typescript
const response = await fetch('https://www.crawlforge.dev/api/v1/tools/fetch_url', {
  method: 'POST',
  headers: {
    'X-API-Key': process.env.CRAWLFORGE_API_KEY!,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ url: 'https://example.com' }),
});

const data = await response.json();
// { success: true, data: {...}, credits_used: 1, credits_remaining: 999 }
```

### Python

```python
import requests

response = requests.post(
    'https://www.crawlforge.dev/api/v1/tools/fetch_url',
    headers={
        'X-API-Key': 'cf_live_YOUR_API_KEY',
        'Content-Type': 'application/json',
    },
    json={'url': 'https://example.com'},
)

data = response.json()
```

## Rate Limits

| Plan | Requests/sec | Requests/min |
|------|-------------|--------------|
| Free | 1 | 60 |
| Hobby | 2 | 120 |
| Professional | 4 | 240 |
| Business | 10 | 600 |

## Error Codes

| Code | Meaning |
|------|---------|
| 200 | Success |
| 400 | Bad Request - invalid parameters |
| 401 | Unauthorized - invalid API key |
| 402 | Payment Required - insufficient credits |
| 429 | Too Many Requests - rate limit exceeded |
| 500 | Internal Server Error |

## Support

- Documentation: https://www.crawlforge.dev/docs
- FAQ: https://www.crawlforge.dev/docs/support/faq
- Email: support@crawlforge.dev
- Discord: https://discord.gg/crawlforge

## Blog Posts

- **CrawlForge v4.2.2 - New CLI + 3 Tools for Local AI Scraping** (https://www.crawlforge.dev/blog/crawlforge-v4-2-2-cli-and-3-new-tools): v4.2.2 launch announcement covering the new @crawlforge/cli package, extract_with_llm (Ollama-default local extraction), scrape_template (10 pre-built site scrapers), and list_ollama_models (local model discovery). That v4.2.2 release brought the catalog to 23 tools (now 27).
- **Web Scraping From the CLI - The CrawlForge CLI Guide** (https://www.crawlforge.dev/blog/web-scraping-cli-complete-guide): Complete reference for @crawlforge/cli with all 15 commands, JSON-piping patterns (jq), cron scheduling, GitHub Actions integration, and a CLI-vs-MCP-vs-API decision matrix.
- **Extract Web Data With Local LLMs (Ollama + CrawlForge)** (https://www.crawlforge.dev/blog/extract-data-with-local-llms-ollama): Deep dive on extract_with_llm and list_ollama_models. Run LLM-powered structured extraction on your own machine -- no API key, no data leaving localhost. Includes setup, schema patterns, provider comparison, and cost analysis.
- **Scrape Amazon, LinkedIn & 8 More Sites With One Tool** (https://www.crawlforge.dev/blog/scrape-amazon-linkedin-github-templates): Walkthrough of scrape_template with all 10 supported sites (Amazon, LinkedIn, GitHub, YouTube, Reddit, Hacker News, Stack Overflow, npm, Product Hunt, Twitter/X). One call, structured JSON, 1 credit.
- **CrawlForge MCP Is Now Live** (https://www.crawlforge.dev/blog/crawlforge-mcp-is-now-live): Official launch announcement covering CrawlForge MCP's 27 tools, credit-based pricing, and native MCP integration with Claude and Cursor.
- **CrawlForge MCP Quickstart** (https://www.crawlforge.dev/blog/crawlforge-mcp-quickstart): Step-by-step guide to get your first CrawlForge scrape running in under 5 minutes with Claude Desktop or the REST API.
- **Install CrawlForge MCP in Claude Code - Beginner's Guide** (https://www.crawlforge.dev/blog/install-crawlforge-mcp-claude-code-beginners-guide): Complete walkthrough for installing and configuring CrawlForge as an MCP server in Claude Code for terminal-based AI workflows.
- **18 Web Scraping Tools in One MCP Server** (https://www.crawlforge.dev/blog/18-web-scraping-tools-one-mcp-server): Deep dive into the CrawlForge tool catalog, explaining when to use each tool and how they work together for complex scraping tasks. (Historical post — catalog has since grown to 27 tools.)
- **5 Ways to Use CrawlForge with LangChain** (https://www.crawlforge.dev/blog/5-ways-use-crawlforge-langchain): Practical examples of integrating CrawlForge tools into LangChain agents for RAG pipelines, research automation, and data enrichment.
- **CrawlForge vs Apify vs ScrapingBee Comparison** (https://www.crawlforge.dev/blog/crawlforge-vs-apify-vs-scrapingbee-comparison): Feature-by-feature comparison of CrawlForge against Apify and ScrapingBee, covering pricing, AI integration, and developer experience.
- **How to Add Web Scraping to Claude Desktop** (https://www.crawlforge.dev/blog/how-to-add-web-scraping-to-claude-desktop): Tutorial for enabling web scraping capabilities in Claude Desktop using CrawlForge MCP with zero coding required.
- **Building an AI Research Assistant with Claude MCP** (https://www.crawlforge.dev/blog/building-ai-research-assistant-claude-mcp): Build a multi-source research assistant that uses deep_research and summarize_content to synthesize information from across the web.
- **MCP vs REST - Why Native MCP Beats REST for Scraping** (https://www.crawlforge.dev/blog/mcp-vs-rest-native-mcp-scraping-server): Technical comparison showing why MCP-native scraping tools outperform REST API wrappers for AI agent workflows.
- **Introducing Deep Research - AI-Powered Analysis** (https://www.crawlforge.dev/blog/introducing-deep-research-ai-powered-analysis): Introduction to the deep_research tool that combines multi-stage web search, content extraction, and AI synthesis for comprehensive analysis.
- **Web Scraping for AI Training Data - 2025 Guide** (https://www.crawlforge.dev/blog/web-scraping-ai-training-data-2025-guide): Best practices for collecting, cleaning, and structuring web data for AI model training, including ethical and legal considerations.
- **Welcome to CrawlForge** (https://www.crawlforge.dev/blog/welcome-to-crawlforge): Introduction to the CrawlForge platform, our mission to make web data accessible to AI agents, and what to expect from the product.
- **CrawlForge vs Firecrawl - MCP Web Scraper Comparison** (https://www.crawlforge.dev/blog/crawlforge-vs-firecrawl-mcp-web-scraper-comparison): Head-to-head comparison of CrawlForge and Firecrawl covering tool breadth, pricing models, MCP support, and unique features like stealth mode.
- **Build a Competitive Intelligence Agent with Claude and CrawlForge** (https://www.crawlforge.dev/blog/build-competitive-intelligence-agent-claude-crawlforge): Tutorial for building an automated competitive intelligence agent that monitors competitor websites, tracks pricing changes, and generates reports.
- **Stealth Mode Scraping - Bypass Anti-Bot Detection** (https://www.crawlforge.dev/blog/stealth-mode-scraping-bypass-anti-bot-detection): How CrawlForge's stealth_mode tool uses fingerprint randomization and proxy rotation to access sites with aggressive anti-bot protection.
- **Deep Research Automation - 10 Hours to 10 Minutes** (https://www.crawlforge.dev/blog/deep-research-automation-10-hours-to-10-minutes): Case study showing how deep_research reduces manual research workflows from hours to minutes with automated source discovery and synthesis.
- **Complete Guide to MCP Web Scraping** (https://www.crawlforge.dev/blog/complete-guide-mcp-web-scraping): Comprehensive guide to web scraping via the Model Context Protocol, covering setup, tool selection, error handling, and production best practices.

## Frequently Asked Questions

### How do credits work?
Each tool costs a set number of credits per use. Simple tools like fetch_url cost 1 credit, while advanced tools like deep_research cost 10 credits per query. Your credit balance is deducted atomically with each API call, and you can monitor usage in your dashboard.

### Do unused credits roll over?
Yes! On paid plans, all unused credits roll over to the next month and never expire while your subscription is active. The Free tier grants 1,000 credits once at signup; they do not refill.

### Can I change plans anytime?
Yes, you can upgrade or downgrade your plan at any time from your dashboard. Upgrades take effect immediately with prorated billing. Downgrades take effect at the start of your next billing cycle.

### What payment methods do you accept?
We accept all major credit and debit cards (Visa, Mastercard, American Express) through Stripe. Annual plans and Enterprise contracts may also pay via invoice.

### Is there a free trial?
The Free tier gives you 1,000 credits at no cost with no credit card required. This lets you fully test all 27 tools before committing to a paid plan.

### What happens when I run out of credits?
When your credits reach zero, API requests will return a 402 Payment Required error. You can purchase credit add-ons starting at $3 for 1,000 credits, or upgrade your plan for a larger monthly allocation.

## Getting Started Quick Guide

1. **Sign up** at https://www.crawlforge.dev/signup to create your free account (no credit card required, 1,000 free credits).
2. **Generate an API key** from your dashboard at https://www.crawlforge.dev/dashboard. Your key will start with `cf_live_`.
3. **Install the MCP server** by adding CrawlForge to your Claude Desktop or Cursor IDE config: `npx -y @crawlforge/mcp-server` with your API key as an environment variable.
4. **Make your first request** by asking Claude to fetch a webpage or run a search. CrawlForge tools are automatically available as MCP tools.
5. **Monitor usage** in your dashboard to track credit consumption, view request history, and manage your subscription.

## Use Cases

### Lead Generation
Use extract_content and scrape_structured to pull contact information, company details, and prospect data from business directories, company websites, and professional networks. Combine with batch_scrape to process hundreds of prospect pages in a single request.

### Price Monitoring
Combine track_changes with scrape_structured to monitor competitor pricing across e-commerce sites. Set up automated change detection with significance scoring to get alerted only when prices change meaningfully, filtering out minor page updates.

### Data Extraction
Use scrape_structured with CSS selectors for known page layouts, or extract_content with AI-powered extraction for unstructured pages. Process PDFs and documents with process_document. Scale to thousands of pages using batch_scrape and crawl_deep.

### Competitive Intelligence
Build automated monitoring workflows with track_changes to watch competitor websites for product launches, feature updates, and messaging changes. Use deep_research to synthesize competitive landscapes from multiple sources into actionable reports.

### AI Research Workflows
The deep_research tool automates multi-stage research by searching the web, extracting content from top sources, cross-referencing facts, and synthesizing findings into structured reports. Combined with summarize_content and analyze_content, it replaces hours of manual research.

## Comparison: CrawlForge vs Firecrawl vs Apify

### CrawlForge
- MCP-native architecture purpose-built for AI agents
- 27 specialized tools covering scraping, research, and monitoring
- Credit-based pricing starting at $0 with 1,000 free credits
- Built-in stealth mode with anti-detection and proxy rotation
- Deep research tool for multi-stage AI-powered analysis
- Single npm package install for Claude, Cursor, and any MCP client

### Firecrawl
- REST API with MCP wrapper available
- Focused on web scraping and crawling
- Page-based pricing model
- Good markdown conversion capabilities
- Requires separate MCP adapter configuration

### Apify
- Actor-based platform with extensive marketplace
- Hundreds of pre-built scrapers for specific sites
- Usage-based pricing with compute units
- Steeper learning curve for custom actors
- REST API without native MCP support

## Additional Blog Posts

### Integrations and Framework Guides
- **How to Use CrawlForge with n8n** (https://www.crawlforge.dev/blog/crawlforge-n8n-integration-guide): Connect CrawlForge MCP to n8n for automated web scraping workflows with no-code pipelines that extract, transform, and load web data on a schedule.
- **How to Use CrawlForge with Vercel AI SDK** (https://www.crawlforge.dev/blog/crawlforge-vercel-ai-sdk-guide): Build AI apps with live web data using CrawlForge and the Vercel AI SDK. Register 27 web scraping tools with generateText and streamText in under 10 minutes.
- **How to Use CrawlForge with Windsurf IDE** (https://www.crawlforge.dev/blog/crawlforge-windsurf-integration-guide): Add 27 web scraping tools to Windsurf IDE with CrawlForge MCP. Fetch docs, scrape references, and research APIs without leaving your editor.
- **How to Use CrawlForge with Cline (VS Code)** (https://www.crawlforge.dev/blog/crawlforge-cline-integration-guide): Configure CrawlForge MCP in Cline for VS Code and let your AI coding assistant access live web data from inside your editor.
- **How to Use CrawlForge with OpenAI Agents SDK** (https://www.crawlforge.dev/blog/crawlforge-openai-agents-integration): Connect CrawlForge to the OpenAI Agents SDK for live data retrieval, research, and structured extraction inside GPT-powered agents.
- **How to Use CrawlForge with Anthropic Claude API** (https://www.crawlforge.dev/blog/crawlforge-anthropic-claude-api-guide): Connect CrawlForge web scraping tools to the Claude API via tool_use. TypeScript examples with Claude Sonnet for live web data.
- **How to Use CrawlForge with Smithery** (https://www.crawlforge.dev/blog/crawlforge-smithery-integration-guide): Discover, install, and configure CrawlForge via the Smithery MCP marketplace with CLI commands and tool usage examples.
- **How to Use CrawlForge with Cursor Rules** (https://www.crawlforge.dev/blog/crawlforge-cursor-rules-guide): Create .cursorrules files that teach Cursor AI to use CrawlForge tools effectively, with ready-to-use rules for web research and data extraction.
- **How to Use CrawlForge with Mastra AI Agents** (https://www.crawlforge.dev/blog/crawlforge-mastra-integration-guide): Build AI agents with web scraping capabilities using Mastra and CrawlForge. TypeScript setup, tool integration, workflows, and agent examples.
- **How to Use CrawlForge with Dify Workflows** (https://www.crawlforge.dev/blog/crawlforge-dify-integration-guide): Add CrawlForge as a custom tool in Dify for web scraping in LLM app workflows. No-code and API integration guides.
- **How to Use CrawlForge with Make and Zapier** (https://www.crawlforge.dev/blog/crawlforge-make-zapier-automation-guide): Connect CrawlForge to Make (Integromat) and Zapier for automated web scraping with HTTP modules, webhooks, and workflow examples.
- **How to Use CrawlForge with LangGraph Agents** (https://www.crawlforge.dev/blog/crawlforge-langgraph-integration-guide): Build stateful web scraping agents with LangGraph and CrawlForge. TypeScript guide covering graph nodes, state management, and conditional scraping flows.

### Tutorials and Use Cases
- **Build an AI-Powered Price Monitoring System** (https://www.crawlforge.dev/blog/ai-price-monitoring-system-guide): Track competitor prices automatically with CrawlForge and Claude. Extract, compare, and alert on pricing changes across thousands of product pages.
- **Automate SEO Audits with CrawlForge MCP** (https://www.crawlforge.dev/blog/automating-seo-audits-crawlforge): Run comprehensive technical SEO audits automatically. Crawl your site, check metadata, find broken links, and generate actionable reports.
- **Real-Time Competitive Intelligence with AI Agents** (https://www.crawlforge.dev/blog/competitive-intelligence-ai-agents): Build an AI-powered competitive intelligence system using CrawlForge and Claude. Monitor competitors, track changes, and generate strategic insights weekly.
- **Web Scraping for AI Training Data Pipelines** (https://www.crawlforge.dev/blog/web-scraping-ai-training-data-pipelines): Build production-ready AI training data pipelines with CrawlForge. Extract, clean, and structure web content for fine-tuning LLMs and training ML models.
- **Content Migration Automation with CrawlForge** (https://www.crawlforge.dev/blog/content-migration-automation-guide): Migrate website content between CMS platforms automatically. Extract pages, preserve structure, and rebuild content in your new system.
- **Build a Lead Enrichment Engine with CrawlForge** (https://www.crawlforge.dev/blog/building-lead-enrichment-engine): Enrich sales leads with company data, tech stacks, and contact details automatically. Scrape public business data to qualify leads.
- **Build a Research Agent with CrawlForge Deep Research** (https://www.crawlforge.dev/blog/real-time-research-agent-deep-research): Create an AI research agent that gathers, verifies, and synthesizes information from dozens of sources in minutes using CrawlForge deep_research.
- **E-commerce Product Data Extraction at Scale** (https://www.crawlforge.dev/blog/ecommerce-product-data-extraction-scale): Extract product data from thousands of e-commerce pages with CrawlForge. Build catalogs, monitor inventory, and power comparison engines.

### Deep Dives and Reference Guides
- **Best Web Scraping Tools in 2026** (https://www.crawlforge.dev/blog/best-web-scraping-tools-2026): Compare 12 web scraping tools for 2026 including CrawlForge, Firecrawl, Apify, and Scrapy. Features, pricing, and recommendations for every use case.
- **MCP Protocol Explained: A Developer Guide for 2026** (https://www.crawlforge.dev/blog/mcp-protocol-explained-developers): Learn how the Model Context Protocol works, why it matters for AI agents, and how to build MCP servers and clients with architecture diagrams and code.
- **Web Scraping: Python vs MCP in 2026** (https://www.crawlforge.dev/blog/web-scraping-python-vs-mcp): Compare Python scraping (requests, BeautifulSoup, Scrapy) with MCP-based scraping. Side-by-side code, performance benchmarks, and when to use each.
- **How to Build a RAG Pipeline with Web Data** (https://www.crawlforge.dev/blog/build-rag-pipeline-web-data): Build a production RAG pipeline that crawls websites, extracts content, chunks text, generates embeddings, and serves retrieval-augmented answers.
- **Web Scraping by Industry: 2026 Playbook** (https://www.crawlforge.dev/blog/industry-web-scraping-guide-2026): Industry-specific web scraping strategies for real estate, finance, e-commerce, healthcare, and travel. Data targets, CrawlForge tools, and compliance rules.

## Alternatives and Head-to-Head Comparisons

CrawlForge maintains detailed comparison pages that explain when to choose CrawlForge versus each competitor. Each comparison covers architecture, pricing, AI integration, stealth capabilities, and includes a verdict plus a "Pick CrawlForge when / Pick competitor when" decision guide.

- **CrawlForge vs Firecrawl** (https://www.crawlforge.dev/alternatives/crawlforge-vs-firecrawl): MCP-native web scraping with 27 specialized tools versus REST-first crawling. CrawlForge wins on tool breadth, AI discoverability, and free tier; Firecrawl has a larger community.
- **CrawlForge vs Apify** (https://www.crawlforge.dev/alternatives/crawlforge-vs-apify): Simple MCP-native tools with transparent per-tool credit pricing versus Apify's full actor marketplace with compute-unit billing. CrawlForge wins on setup simplicity and AI integration.
- **CrawlForge vs ScrapingBee** (https://www.crawlforge.dev/alternatives/crawlforge-vs-scrapingbee): MCP-native tools with built-in structured extraction versus a traditional proxy-based scraping API. CrawlForge adds research, monitoring, and 27 tools ScrapingBee does not offer.
- **CrawlForge vs Scrapy** (https://www.crawlforge.dev/alternatives/crawlforge-vs-scrapy): Managed MCP web scraping service versus a self-hosted Python framework. Zero infrastructure versus full control - pick based on whether you want to own or outsource ops.
- **CrawlForge vs Bright Data** (https://www.crawlforge.dev/alternatives/crawlforge-vs-brightdata): Developer-focused MCP tools with transparent pricing versus an enterprise-grade data platform. CrawlForge wins on simplicity; Bright Data wins on enterprise volume and proxy networks.
- **CrawlForge vs Playwright** (https://www.crawlforge.dev/alternatives/crawlforge-vs-playwright): Managed web extraction service versus a browser automation library. Skip the infrastructure and get structured data via MCP tool calls instead of writing browser scripts.
- **CrawlForge vs Puppeteer** (https://www.crawlforge.dev/alternatives/crawlforge-vs-puppeteer): Managed MCP web scraping versus a Node.js browser automation library. CrawlForge removes Chrome instance management, proxy rotation, and scaling concerns.
- **Best Firecrawl Alternative: CrawlForge** (https://www.crawlforge.dev/alternatives/firecrawl-alternative): CrawlForge offers MCP-native web scraping with 27 tools, deep research with conflict detection, and 1,000 free credits as a direct Firecrawl alternative.
- **Best Apify Alternative: CrawlForge** (https://www.crawlforge.dev/alternatives/apify-alternative): CrawlForge replaces complex actor configurations with 27 MCP-native tools and simple per-tool credit pricing. Get started without the Apify learning curve.
- **Web Scraping API Comparison 2026** (https://www.crawlforge.dev/alternatives/web-scraping-api-comparison): Side-by-side comparison of CrawlForge, Firecrawl, Apify, ScrapingBee, and Bright Data for web scraping and AI data extraction.

## Use Case Deep Dives

Each use case page includes a problem statement, the CrawlForge solution, specific tools used, a working TypeScript code example, and credit cost estimates.

- **AI Agent Data Pipelines** (https://www.crawlforge.dev/use-cases/ai-agent-data-pipelines): Feed AI agents live web data with deep_research and extract_content. Agents get structured, real-time information without custom scrapers. ~12 credits per research task.
- **E-commerce Price Monitoring** (https://www.crawlforge.dev/use-cases/ecommerce-price-monitoring): Track competitor prices across hundreds of product pages using batch_scrape and track_changes. Get alerts when competitors adjust pricing. ~8 credits per batch.
- **SEO Analysis and Auditing** (https://www.crawlforge.dev/use-cases/seo-analysis-auditing): Audit sites for metadata, broken links, and content gaps using map_site, extract_metadata, and analyze_content. ~8 credits per site audit.
- **Competitive Intelligence** (https://www.crawlforge.dev/use-cases/competitive-intelligence): Gather competitor data through anti-bot protections using stealth_mode and scrape_structured with schema-based extraction. ~8 credits per protected page.
- **Content Migration** (https://www.crawlforge.dev/use-cases/content-migration): Crawl legacy sites and extract all content for CMS migration using crawl_deep and extract_text. Move thousands of pages in a single run. ~6 credits per page.
- **Lead Enrichment** (https://www.crawlforge.dev/use-cases/lead-enrichment): Enrich sales leads with company data using search_web and extract_metadata. Build on-demand enrichment pipelines. ~5 credits per lead.
- **AI Training Data Collection** (https://www.crawlforge.dev/use-cases/ai-training-data): Collect large-scale web datasets for fine-tuning using batch_scrape and extract_content. Clean, structured text ready for training pipelines. ~7 credits per document.
- **Real-Time Research Agents** (https://www.crawlforge.dev/use-cases/real-time-research-agents): Build agents that search the web and synthesize findings using deep_research and search_web with conflict detection. ~13 credits per research session.

## Complete Tool Documentation Reference

Every CrawlForge tool has a dedicated documentation page with parameters, response formats, code examples, and credit costs.

- fetch_url: https://www.crawlforge.dev/docs/api-reference/tools/fetch-url
- extract_text: https://www.crawlforge.dev/docs/api-reference/tools/extract-text
- extract_links: https://www.crawlforge.dev/docs/api-reference/tools/extract-links
- extract_metadata: https://www.crawlforge.dev/docs/api-reference/tools/extract-metadata
- scrape_structured: https://www.crawlforge.dev/docs/api-reference/tools/scrape-structured
- extract_content: https://www.crawlforge.dev/docs/api-reference/tools/extract-content
- map_site: https://www.crawlforge.dev/docs/api-reference/tools/map-site
- process_document: https://www.crawlforge.dev/docs/api-reference/tools/process-document
- localization: https://www.crawlforge.dev/docs/api-reference/tools/localization
- track_changes: https://www.crawlforge.dev/docs/api-reference/tools/track-changes
- analyze_content: https://www.crawlforge.dev/docs/api-reference/tools/analyze-content
- summarize_content: https://www.crawlforge.dev/docs/api-reference/tools/summarize-content
- crawl_deep: https://www.crawlforge.dev/docs/api-reference/tools/crawl-deep
- stealth_mode: https://www.crawlforge.dev/docs/api-reference/tools/stealth-mode
- scrape_with_actions: https://www.crawlforge.dev/docs/api-reference/tools/scrape-with-actions
- batch_scrape: https://www.crawlforge.dev/docs/api-reference/tools/batch-scrape
- search_web: https://www.crawlforge.dev/docs/api-reference/tools/search-web
- serp_rank: https://www.crawlforge.dev/docs/api-reference/tools/serp-rank
- deep_research: https://www.crawlforge.dev/docs/api-reference/tools/deep-research
- scrape_template: https://www.crawlforge.dev/docs/api-reference/tools/scrape-template
- extract_with_llm: https://www.crawlforge.dev/docs/api-reference/tools/extract-with-llm
- list_ollama_models: https://www.crawlforge.dev/docs/api-reference/tools/list-ollama-models
- scrape: https://www.crawlforge.dev/docs/api-reference/tools/scrape
- agent: https://www.crawlforge.dev/docs/api-reference/tools/agent
- get_batch_results: https://www.crawlforge.dev/docs/api-reference/tools/get-batch-results
- Tools index: https://www.crawlforge.dev/docs/api-reference/tools