scrape_ structured
Extract structured data from any webpage using custom CSS selectors. Perfect for e-commerce product scraping, news aggregation, and any custom data extraction needs.
Use Cases
E-Commerce Product Scraping
Extract product titles, prices, descriptions, and images from online stores
News Article Extraction
Extract headlines, authors, dates, and content from news sites
Custom Data Transformation
Map any HTML structure to your desired JSON schema
Real Estate Listings
Extract property details, prices, and images from listing sites
Endpoint
/api/v1/tools/scrape_structuredParameters
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
url | string | Optional | - | Page to fetch and extract from. Either `url` or `html` is required. Example: https://example.com/product |
html | string | Optional | - | Raw HTML to parse instead of fetching. Pair with `base_url` to resolve relative attribute values. Example: <html>...</html> |
selectors | object | Required | - | Maps your field names to CSS selectors. At least one entry is required — the field names are yours and come back unchanged in the result. Example: {"title": "h1.product-title", "price": ".price", "description": ".product-desc"} |
base_url | string | Optional | - | Base for resolving relative URLs found in extracted attributes. Example: https://example.com |
multiple | boolean | Optional | false | When true, extract a list of repeating items instead of a single record. Changes the response shape — see below. Example: true |
clean_text | boolean | Optional | true | Collapse whitespace and trim the extracted text. Example: true |
include_attributes | array | Optional | - | Also capture these HTML attributes from each matched element, not just its text. Example: ["href", "src", "alt"] |
max_items | number | Optional | 100 | Maximum items returned when `multiple` is true, 1-1000. Example: 100 |
respect_robots | boolean | Optional | true | Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true |
CSS Selectors:
Use any valid CSS selector syntax. Common patterns:
.className- Select by class#id- Select by IDtag.class- Combine tag and class.parent > .child- Direct child[data-id="value"]- Attribute selector
Request Examples
cURL - E-Commerce Product
curl -X POST https://crawlforge.dev/api/v1/tools/scrape_structured \
-H "X-API-Key: cf_test_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/product/123",
"selectors": {
"title": "h1.product-title",
"price": ".price-value",
"currency": ".price-currency",
"description": ".product-description",
"image": "img.main-image",
"rating": ".rating-value",
"availability": ".stock-status"
}
}'TypeScript - News Article
// npm install crawlforge-sdk
import { CrawlForge } from 'crawlforge-sdk';
const client = new CrawlForge({ apiKey: process.env.CRAWLFORGE_API_KEY });
const result = await client.scrapeStructured({
url: 'https://example.com/news/article-123',
selectors: {
headline: 'h1.article-title',
author: '.author-name',
publishDate: 'time.publish-date',
category: '.category-tag',
content: '.article-body',
image: '.article-image img'
}
});
// result.data is untyped in crawlforge-sdk 0.1 — its shape is the Response Example below.
const article = result.data as { headline: string; author: string; publishDate: string };
console.log('Article:', article.headline);
console.log('By:', article.author);
console.log('Published:', article.publishDate);Python - Real Estate Listing
# pip install crawlforge
from crawlforge import CrawlForge
client = CrawlForge() # reads CRAWLFORGE_API_KEY
result = client.scrape_structured(
url='https://example.com/property/456',
selectors={
'address': '.property-address',
'price': '.listing-price',
'bedrooms': '.bed-count',
'bathrooms': '.bath-count',
'sqft': '.square-feet',
'description': '.property-description',
'images': '.gallery img'
},
)
# result.data is a plain dict — its shape is the Response Example below.
property_data = result.data
print(f"Property: {property_data['address']}")
print(f"Price: {property_data['price']}")
print(f"Beds: {property_data['bedrooms']}")
print(f"Baths: {property_data['bathrooms']}")Response Example
{ "success": true, "data": { "title": "Premium Wireless Headphones", "price": "299.99", "currency": "USD", "description": "High-quality wireless headphones with active noise cancellation and 30-hour battery life.", "image": "https://example.com/images/headphones.jpg", "rating": "4.7", "availability": "In Stock" }, "credits_used": 2, "credits_remaining": 998, "processing_time": 320}data.titleExtracted from h1.product-title selectordata.priceExtracted from .price-value selectordata.descriptionExtracted from .product-description selectorcredits_usedCredits deducted for this request (2 per scrape)Error Handling
Blocked by robots.txt (403 Forbidden)
The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to.
Credit Cost
Tip: For scraping multiple pages with the same structure, use batch_scrape for better efficiency.
Related Tools
Ready to extract structured data? Sign up for free and get 1,000 credits.