Skip to content
CrawlForge MCP
AI Tool3 credits

extract_structured

Give the extractor a JSON Schema and a natural-language prompt. The LLM reads the page and returns data matching your schema. When no LLM provider is configured it falls back to CSS selector extraction using your hints.

Use Cases

Schema-First Product Extraction

Define the fields you want once; the LLM maps any e-commerce site to your schema.

Resume & Document Parsing

Extract candidate names, skills, and work history directly into a typed object.

Knowledge Graph Seeding

Extract entities and relationships from articles into structured JSON for graph loaders.

Endpoint

POST/api/v1/tools/extract_structured
Auth Required
1 req/s on Free plan
3 credits

Parameters

NameTypeRequiredDefaultDescription
url
stringRequired-
URL to extract data from
Example: https://example.com/product/123
schema
objectRequired-
JSON Schema describing the data to extract
Example: {"type":"object","properties":{"title":{"type":"string"},"price":{"type":"number"}},"required":["title"]}
prompt
stringOptional-
Natural-language instructions guiding the LLM extraction
Example: Extract the product name, current price, and whether it is in stock
llmConfig
objectOptional-
Optional LLM provider configuration (provider, apiKey). Omit to use CSS selector fallback.
Example: {"provider": "openai", "apiKey": "sk-..."}
selectorHints
objectOptional-
CSS selector hints to guide extraction (also used by selector fallback)
Example: {"title": "h1.product-title", "price": ".price"}
fallbackToSelectors
booleanOptionaltrue
Fall back to CSS selector extraction when LLM is unavailable
Example: true
respect_robots
booleanOptionaltrue
Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key.
Example: true

Request Examples

cURL — LLM extraction

terminalBash
curl -X POST https://crawlforge.dev/api/v1/tools/extract_structured \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product/123",
    "schema": {
      "type": "object",
      "properties": {
        "title": { "type": "string" },
        "price": { "type": "number" },
        "in_stock": { "type": "boolean" }
      },
      "required": ["title", "price"]
    },
    "prompt": "Extract the product name, price in USD, and availability",
    "llmConfig": { "provider": "openai", "apiKey": "sk-..." }
  }'

TypeScript — selector fallback

extractStructured.tsTypescript
// npm install crawlforge-sdk
import { CrawlForge } from 'crawlforge-sdk';

const client = new CrawlForge({ apiKey: process.env.CRAWLFORGE_API_KEY });

const result = await client.extractStructured({
  url: 'https://example.com/product/123',
  schema: {
    type: 'object',
    properties: {
      title: { type: 'string' },
      price: { type: 'number' },
    },
    required: ['title'],
  },
  selectorHints: {
    title: 'h1.product-title',
    price: '.price-value',
  },
  fallbackToSelectors: true,
});

// result.data is untyped in crawlforge-sdk 0.1 — its shape is the Response Example below.
const { data: product } = result.data as { data: { title: string; price: number } };
console.log(product.title, product.price);

Python

extract_structured.pyPython
# pip install crawlforge
from crawlforge import CrawlForge

client = CrawlForge()  # reads CRAWLFORGE_API_KEY

result = client.extract_structured(
    url='https://example.com/article/42',
    schema={
        'type': 'object',
        'properties': {
            'headline': {'type': 'string'},
            'author': {'type': 'string'},
            'published_at': {'type': 'string'},
            'tags': {'type': 'array'},
        },
        'required': ['headline'],
    },
    prompt='Extract headline, author, publish date (ISO 8601), and tags',
)

# result.data is a plain dict — its shape is the Response Example below.
print(result.data['data'])

Response Example

200 OK1.2s
{
  "success": true,
  "data": {
    "url": "https://example.com/product/123",
    "data": {
      "title": "Premium Wireless Headphones",
      "price": 299.99,
      "in_stock": true
    },
    "extraction": {
      "method_by_field": {
        "title": "selector",
        "price": "json-ld",
        "in_stock": "meta"
      },
      "llm_used": false,
      "note": "Selector/structured-data extraction; LLM-guided extraction is available via the CrawlForge MCP server"
    },
    "extracted_at": "2026-08-26T14:30:00.000Z"
  },
  "credits_used": 3,
  "credits_remaining": 997,
  "processing_time": 1240
}
Field Descriptions
data.dataOne key per property in your schema. A `number` or `integer` property takes the first number in the text found (`"4.5 out of 5"` is 4.5, `"$1,299.00"` is 1299; an `integer` only when whole); text with no number is returned as it is. A field that could not be found is null.
data.extraction.method_by_fieldHow each field was obtained: `selector` from a selectorHints entry, `json-ld` from structured data, `meta` from a meta tag, or `none` when nothing matched.
data.extraction.llm_usedAlways false on the hosted REST API — extraction is selector and structured-data based, never generative.
data.extraction.noteRestates where LLM-guided extraction is available.
data.extracted_atISO 8601 timestamp of the extraction.

Error Handling

Blocked by robots.txt (403 Forbidden)

The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to.

Credit Cost

3 credits
3 credits per request
Flat 3 credits whether the call uses the LLM or the selector fallback.

Tip: Pair with scrape_structured (2 credits, CSS-only) when you already have stable selectors and don't need LLM flexibility.

Related Tools

scrape_structured
CSS selector extraction (2 credits)
analyze_content
Sentiment, entities, and topic analysis (3 credits)