deep_research
Give it a question and it runs several web searches, fetches the most promising results, scores every passage against your query, and returns the strongest ones — each carrying the URL and title of the page it came from.
Use Cases
Answer a question with citations attached
Every finding carries its source_url, so each claim can be traced back to the page it was taken from.
Survey what several sources say on a topic
One call runs multiple search queries and pulls from up to 10 distinct sources, rather than reading one page at a time.
Ground an LLM prompt in fetched material
Findings are verbatim passages, not paraphrase, so they can be passed to your own model as context without a second hop through someone else's summary.
Restrict research to sources you trust
research_scope.domains limits the search to as many as 10 domains — useful for regulatory, vendor or internal documentation research.
Endpoint
/api/v1/tools/deep_researchParameters
research_query, not topic or query, and it must be at least 10 characters. Unknown keys are silently discarded, so sending topic produces a 400 for a missing research_query.| Name | Type | Required | Default | Description |
|---|---|---|---|---|
research_query | string | Required | - | The question to research. Minimum 10 characters. Phrase it as a question or a specific claim — the wording is used both to run searches and to score passages, so a precise query ranks better than a bare keyword. Example: What are the tradeoffs of edge caching for API responses? |
research_scope | object | Optional | - | Optional controls over how wide and how recent the research is. |
max_sources | number | Optional | - | Override the source count implied by `depth_level`, 1-10. Takes precedence when both are set. Example: 8 |
respect_robots | boolean | Optional | true | Respect each source site's robots.txt. Left at `true`, a search result whose robots.txt disallows `CrawlForge` is not fetched — it stays in the source set with `fetched: false` and its search snippet only, and the reason is named in `warnings` rather than returned as a 403. The flat credit cost is unchanged. Set it to `false` only for targets you have your own agreement with; the override is recorded against your API key and does not reach a host on CrawlForge's permanent opt-out list. Example: true |
methodology.llm_used: false. For LLM-written synthesis, use the CrawlForge MCP server.How a research run works
Four stages, all inside a single request.
methodology.queries_run.depth_level or max_sources allows. methodology.sources_considered reports how many were seen before the cut.sources with fetched: false.key_findings, each truncated to 600 characters and tagged with the URL it came from.relevance_score is a term-overlap score against your query, not a judgement of factual accuracy or source credibility. A high score means the passage matches what you asked about — nothing more. Read the source_url before relying on a finding.Request Examples
# The query parameter is research_query, not topic. Minimum 10 characters.
curl -X POST https://crawlforge.dev/api/v1/tools/deep_research \
-H "X-API-Key: cf_test_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"research_query": "What are the tradeoffs of edge caching for API responses?",
"research_scope": {
"depth_level": "deep",
"time_range": "year",
"language": "en"
},
"max_sources": 8
}'
# Restrict the search to sources you already trust
curl -X POST https://crawlforge.dev/api/v1/tools/deep_research \
-H "X-API-Key: cf_test_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"research_query": "What does the EU AI Act require for general-purpose models?",
"research_scope": {
"domains": ["europa.eu", "eur-lex.europa.eu"]
}
}'Response Example
{ "success": true, "data": { "research_query": "What are the tradeoffs of edge caching for API responses?", "methodology": { "queries_run": [ "tradeoffs of edge caching for API responses", "edge caching API responses disadvantages", "CDN edge cache API latency consistency" ], "search_backend": "google_cse", "sources_considered": 27, "sources_fetched": 5, "synthesis": "extractive", "llm_used": false }, "key_findings": [ { "text": "Edge caching cuts round-trip latency by serving from a point of presence near the client, but it introduces a consistency window: until the TTL expires or an explicit purge lands, different regions can serve different versions of the same resource.", "source_url": "https://example.com/engineering/edge-caching", "source_title": "Edge caching in practice", "relevance_score": 0.874 }, { "text": "Purge propagation is the operational cost most teams underestimate. A global invalidation is not instantaneous, and designs that assume it is will read stale data during the propagation window.", "source_url": "https://example.org/cdn-invalidation", "source_title": "CDN invalidation strategies", "relevance_score": 0.791 } ], "sources": [ { "url": "https://example.com/engineering/edge-caching", "title": "Edge caching in practice", "snippet": "How edge caching changes the latency and consistency profile of an API...", "fetched": true, "domain": "example.com" }, { "url": "https://example.net/blocked-article", "title": "Caching at the edge", "snippet": "An overview of edge caching patterns...", "fetched": false, "domain": "example.net" } ], "summary": "Edge caching trades consistency for latency. The dominant operational cost is purge propagation, and the dominant design question is which endpoints tolerate a staleness window.", "notes": "Synthesis is extractive (no LLM on the hosted API). For LLM-synthesized deep research, use the CrawlForge MCP server.", "researched_at": "2026-08-26T14:30:00.000Z" }, "credits_used": 10, "credits_remaining": 990, "processing_time": 14260}data.methodology.queries_runThe searches actually performed, expanded from your query. Useful for judging whether the run understood the question.data.methodology.sources_consideredSearch results seen before selection; sources_fetched is how many were then retrieved.data.methodology.llm_usedAlways false on the hosted REST API — synthesis is extractive.data.key_findingsUp to 10 passages, highest-scoring first, each truncated to 600 characters.data.key_findings.source_urlThe page the passage was taken from verbatim — this is the citation.data.key_findings.relevance_scoreTerm-overlap score against your query, rounded to 3 decimals. Not a credibility signal.data.sources.fetchedFalse when the page could not be retrieved. It still appears here, but contributed no findings.data.summaryAssembled from the top-ranked passages, not written by a model.processing_timeResearch runs are slow — searching and fetching several pages typically takes 10-20 seconds.Error Handling
Missing research_query (400 VALIDATION_ERROR)
Usually caused by sending topic or query instead. Unknown keys are discarded, so the request arrives with no query at all.
Query too short (400 VALIDATION_ERROR)
research_query must be at least 10 characters. A bare keyword is both rejected and, in general, a poor query — passages are scored against these terms.
Search backend unreachable (502 RESEARCH_SEARCH_UNAVAILABLE)
The upstream search provider could not be reached. No credits are charged.
Search failed (502 RESEARCH_SEARCH_FAILED)
The search provider responded with an error, most often a quota limit. No credits are charged.
Disallowed by robots.txt (no error — the source is not fetched)
A search result that robots.txt disallows for CrawlForge is not fetched, but it stays in the source set with fetched: false and its search snippet, and warnings names it — so the run returns no 403 and the source count is unchanged. Set respect_robots: false to override for targets you have your own agreement with — the override is recorded against your API key, and it does not reach a host on CrawlForge's permanent opt-out list.
research_query, so the query does double duty as both the search input and the ranking key. A specific question with distinctive terms outranks a broad one — and research_scope.domains is more effective than a longer query when you already know which sources you trust.Credit Cost
depth_level or how many sources are fetched — a comprehensive run over 10 sources costs the same as a surface run over 3. Failed calls, including both search-backend errors, are not charged.Cost Breakdown:
Any research run, 3 to 10 sources: 10 credits
Plan Recommendations:
Free Plan: 1,000 one-time trial credits = 100 research runs
Hobby Plan: 5,000 credits/mo = 500 research runs ($19/mo)
Professional Plan: 50,000 credits/mo = 5,000 research runs ($99/mo)
Because cost is flat, there is no saving in running surface — use deep or comprehensive unless you need the speed.