Skip to content
Structured Tool2 credits

scrape_structured

Extract structured data from any webpage using custom CSS selectors. Perfect for e-commerce product scraping, news aggregation, and any custom data extraction needs.

Use Cases

E-Commerce Product Scraping

Extract product titles, prices, descriptions, and images from online stores

News Article Extraction

Extract headlines, authors, dates, and content from news sites

Custom Data Transformation

Map any HTML structure to your desired JSON schema

Real Estate Listings

Extract property details, prices, and images from listing sites

Endpoint

POST/api/v1/tools/scrape_structured
Auth Required
1 req/s on Free plan
2 credits

Parameters

NameTypeRequiredDefaultDescription
url
stringOptional-
Page to fetch and extract from. Either `url` or `html` is required.
Example: https://example.com/product
html
stringOptional-
Raw HTML to parse instead of fetching. Pair with `base_url` to resolve relative attribute values.
Example: <html>...</html>
selectors
objectRequired-
Maps your field names to CSS selectors. At least one entry is required — the field names are yours and come back unchanged in the result.
Example: {"title": "h1.product-title", "price": ".price", "description": ".product-desc"}
base_url
stringOptional-
Base for resolving relative URLs found in extracted attributes.
Example: https://example.com
multiple
booleanOptionalfalse
When true, extract a list of repeating items instead of a single record. Changes the response shape — see below.
Example: true
clean_text
booleanOptionaltrue
Collapse whitespace and trim the extracted text.
Example: true
include_attributes
arrayOptional-
Also capture these HTML attributes from each matched element, not just its text.
Example: ["href", "src", "alt"]
max_items
numberOptional100
Maximum items returned when `multiple` is true, 1-1000.
Example: 100
respect_robots
booleanOptionaltrue
Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key.
Example: true

CSS Selectors:

Use any valid CSS selector syntax. Common patterns:

  • .className - Select by class
  • #id - Select by ID
  • tag.class - Combine tag and class
  • .parent > .child - Direct child
  • [data-id="value"] - Attribute selector

Request Examples

cURL - E-Commerce Product

terminalBash
curl -X POST https://crawlforge.dev/api/v1/tools/scrape_structured \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product/123",
    "selectors": {
      "title": "h1.product-title",
      "price": ".price-value",
      "currency": ".price-currency",
      "description": ".product-description",
      "image": "img.main-image",
      "rating": ".rating-value",
      "availability": ".stock-status"
    }
  }'

TypeScript - News Article

scrapeStructured.tsTypescript
// npm install crawlforge-sdk
import { CrawlForge } from 'crawlforge-sdk';

const client = new CrawlForge({ apiKey: process.env.CRAWLFORGE_API_KEY });

const result = await client.scrapeStructured({
  url: 'https://example.com/news/article-123',
  selectors: {
    headline: 'h1.article-title',
    author: '.author-name',
    publishDate: 'time.publish-date',
    category: '.category-tag',
    content: '.article-body',
    image: '.article-image img'
  }
});

// result.data is untyped in crawlforge-sdk 0.1 — its shape is the Response Example below.
const article = result.data as { headline: string; author: string; publishDate: string };
console.log('Article:', article.headline);
console.log('By:', article.author);
console.log('Published:', article.publishDate);

Python - Real Estate Listing

scrape_structured.pyPython
# pip install crawlforge
from crawlforge import CrawlForge

client = CrawlForge()  # reads CRAWLFORGE_API_KEY

result = client.scrape_structured(
    url='https://example.com/property/456',
    selectors={
        'address': '.property-address',
        'price': '.listing-price',
        'bedrooms': '.bed-count',
        'bathrooms': '.bath-count',
        'sqft': '.square-feet',
        'description': '.property-description',
        'images': '.gallery img'
    },
)

# result.data is a plain dict — its shape is the Response Example below.
property_data = result.data
print(f"Property: {property_data['address']}")
print(f"Price: {property_data['price']}")
print(f"Beds: {property_data['bedrooms']}")
print(f"Baths: {property_data['bathrooms']}")

Response Example

200 OK320ms
{
"success": true,
"data": {
"title": "Premium Wireless Headphones",
"price": "299.99",
"currency": "USD",
"description": "High-quality wireless headphones with active noise cancellation and 30-hour battery life.",
"image": "https://example.com/images/headphones.jpg",
"rating": "4.7",
"availability": "In Stock"
},
"credits_used": 2,
"credits_remaining": 998,
"processing_time": 320
}
Field Descriptions
data.titleExtracted from h1.product-title selector
data.priceExtracted from .price-value selector
data.descriptionExtracted from .product-description selector
credits_usedCredits deducted for this request (2 per scrape)

Error Handling

Blocked by robots.txt (403 Forbidden)

The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to.

Credit Cost

2 credits
2 credits per request
Each structured scrape costs 2 credits, regardless of the number of selectors.

Tip: For scraping multiple pages with the same structure, use batch_scrape for better efficiency.

Related Tools