Skip to content

Last Updated: October 2026

What leaves your machine, what we keep, for how long, and who we share it with. Every period on this page is the one the code enforces.

How Your Data Flows

CrawlForge has two ways in. The MCP server runs on your machine and talks to websites directly. The REST API runs on our servers. They handle your data differently, so they are described separately.

Solid arrows carry page content. The dashed arrow carries usage metadata only.MCP PATHYour machineMCP clientcrawlforge-mcp-serverTarget websiteCrawlForge APIpage contentusage metadata onlyREST PATHYour REST clientAPI function (Vercel)Target websiteBrowser worker (Render)response, not storedfive tools
Solid arrows carry page content. The dashed arrow carries usage metadata only.

MCP path: results stay on your machine

  • You install crawlforge-mcp-server on your machine. Your MCP client (Claude Desktop, Claude Code, Cursor and others) talks to it over stdio.
  • The server fetches target sites from your network. Page content, extracted data and screenshots are returned to your client and never pass through CrawlForge.
  • The server calls our API for three things: to validate your API key when it starts, to read your credit balance, and to report each call after it runs.
  • A usage report contains the tool name, credits used, status, duration, the server version and the call parameters with secrets masked. It never contains the result.
  • Tools that use a language model run it where you configure it: a local Ollama instance by default, or a provider you set an API key for. That traffic goes from your machine to that provider, not to us.

REST path: results pass through our servers and are returned, not kept

  • Your request reaches a serverless function on Vercel. The function fetches the target site, identified and signed as CrawlForge, builds the response and returns it to you.
  • Five tools (stealth_mode, scrape_with_actions, agent, extract_with_llm and list_ollama_models) run on a browser worker we operate on Render. The page content passes through that worker while the call runs.
  • Responses are not stored, with three time-limited exceptions listed in the retention table: asynchronous batch results, change-tracking baselines and oversized results held for read_result.
  • One usage row is written per call, with the same masked parameters as the MCP path, plus the endpoint and method.
  • The REST API runs no language model of its own. extract_with_llm and agent send text to the model endpoint configured on the Render worker.

What We Keep, and for How Long

This table is the retention schedule for CrawlForge. The privacy policy shows the same table. Every period comes from the code path that enforces it; where a period is set by a provider rather than by us, the table says so.

Data
Tool results on the MCP path (page content, extracted data, screenshots)
Where it is held
Your machine only
Kept for
Never sent to CrawlForge
Notes
The MCP server fetches sites directly and reports only usage metadata to our API.
Data
Tool results on the REST path
Where it is held
Returned in the HTTP response
Kept for
Not stored
Notes
The three time-limited stores below are the only exceptions.
Data
Batch results (asynchronous batch_scrape jobs, read back with get_batch_results)
Where it is held
Redis (Upstash)
Kept for
24 hours
Notes
Expire on their own. The job id is the only way to read them.
Data
Change-tracking baselines (the page text captured by track_changes)
Where it is held
Redis (Upstash)
Kept for
90 days
Notes
Creating a new baseline for the same URL and selector replaces the old one and restarts the period.
Data
Hosted monitor baselines (the last captured text of each monitored page)
Where it is held
PostgreSQL (Neon)
Kept for
Until the monitor is deleted
Notes
Every check replaces the previous capture, so only the latest text of each target is held. Deleting the monitor deletes its baselines.
Data
Hosted monitor checks (per-check status, counts, credits and the added and removed lines of each target)
Where it is held
PostgreSQL (Neon)
Kept for
30 days by default, 1 to 365 days set per monitor
Notes
The scheduled run deletes older checks; deleting the monitor deletes its history.
Data
Oversized results held for read_result (a result too large to return inline)
Where it is held
Redis (Upstash)
Kept for
1 hour
Notes
Stored under your account. The result handle is the only way to read it.
Data
Usage logs: tool name, masked parameters (including the target URL or query), status, duration, credits, API key id and client version
Where it is held
PostgreSQL (Neon)
Kept for
12 months
Notes
A daily job deletes older rows. Secrets are masked before a row is written and again before it is shown in the dashboard.
Data
Account data: email, name, password hash, sign-in provider ids, settings, credit balance and feedback-survey answers
Where it is held
PostgreSQL (Neon)
Kept for
While your account exists
Notes
Deleted with the account. See Account Deletion below.
Data
API keys
Where it is held
PostgreSQL (Neon), as a salted SHA-256 hash plus the key prefix
Kept for
Until you revoke them
Notes
The full key is shown once, when it is created. It is not stored and cannot be shown again.
Data
Sign-in sessions
Where it is held
httpOnly cookies, with a verification cache in Redis (Upstash)
Kept for
Access token 15 minutes, refresh token 7 days
Notes
Signing out clears both. The verification cache expires after 5 minutes.
Data
Password-reset and email-verification tokens
Where it is held
PostgreSQL (Neon), hashed
Kept for
Reset links 1 hour, verification links 24 hours
Notes
Single use. The link in the email is the only copy of the token.
Data
Fraud-prevention data: device fingerprint, risk score and sign-up rate-limit counters
Where it is held
PostgreSQL (Neon) and Redis (Upstash)
Kept for
Fingerprints and risk scores until account deletion, after which only the fingerprint hash is kept, unlinked (see Deleted accounts); rate-limit counters 1 hour to 30 days
Notes
Used only to block abusive sign-ups and key creation. Never used for profiling or advertising.
Data
Payment records
Where it is held
Stripe
Kept for
As long as tax and accounting law requires
Notes
We store the Stripe customer and subscription ids, the plan and the billing period. Card numbers never reach our servers.
Data
Newsletter subscription (email address)
Where it is held
PostgreSQL (Neon)
Kept for
Until you unsubscribe
Notes
Every newsletter carries an unsubscribe link.
Data
Deleted accounts
Where it is held
PostgreSQL (Neon)
Kept for
Deleted at once from Settings (within 30 days of an emailed request); one-way hashes kept indefinitely
Notes
Account, keys, usage logs, subscriptions, monitors, fingerprints, risk scores and tokens are deleted together. One-way hashes of the email address and device fingerprints stay, linked to nothing else, so a new account that matches one gets no free credits. Redis entries expire on the periods above. Stripe keeps payment records as the law requires.

The two hosted-monitor rows apply only once you create a monitor. An account with no monitors holds no monitor data.

Subprocessors

These are the providers that receive your data when you use CrawlForge, and what each one receives. Target websites are not subprocessors: they receive our crawler’s request, identified as CrawlForge, and nothing about your account.

Provider
Vercel
Purpose
Hosting for the website, the dashboard and the REST API
What it receives
Every request: IP address, headers and request body.
Provider
Neon
Purpose
PostgreSQL database
What it receives
Account data, API key hashes, usage logs, subscriptions and fraud-prevention data.
Provider
Upstash
Purpose
Redis
What it receives
Session verification cache, rate limits, credit reservations, batch results, change-tracking baselines, oversized results, webhook idempotency and email deduplication flags.
Provider
Render
Purpose
Browser and language-model worker for five REST tools: stealth_mode, scrape_with_actions, agent, extract_with_llm and list_ollama_models
What it receives
The target URL, the request parameters and the page content while the call runs. Text that extract_with_llm and agent send to a language model goes to the model endpoint configured on this worker.
Provider
Stripe
Purpose
Payments and subscriptions
What it receives
Email, name, the card details you enter on Stripe’s checkout page, and purchase history.
Provider
Resend
Purpose
Transactional email
What it receives
Email address, name and the content of each email: verification links, password resets, billing and credit alerts, and the newsletter.
Provider
Sentry
Purpose
Error monitoring
What it receives
Stack traces and request metadata. Cookies, authorization headers, API keys, tokens and passwords are removed before an event is sent.
Provider
Google, GitHub
Purpose
Sign in with Google or GitHub, only if you choose it
What it receives
We receive your email, name and avatar from the provider. The provider learns that you signed in to CrawlForge.
Provider
Brave Search API, DataForSEO
Purpose
Web search results for search_web, deep_research, agent and reddit_search (Brave Search API), and Google rankings for serp_rank (DataForSEO)
What it receives
The search query, and for serp_rank the domain being ranked.
Provider
Arctic Shift, PullPush, reddit.com
Purpose
Reddit archives for reddit_search
What it receives
The search query.

Our primary infrastructure is in the United States. We add a provider to this list before it receives customer data.

Crawler Identity and Site-Owner Controls

Both surfaces identify themselves the same way and follow the same rules on every fetch. Site owners can verify us, rate-limit us or block us without guessing.

  • Every outbound request carries the User-Agent CrawlForge/<version> (+https://crawlforge.dev). The product token is CrawlForge on both the MCP server and the REST API.
  • Every outbound request is signed with Ed25519 under RFC 9421 (the Web Bot Auth profile). The public key directory is at /.well-known/http-message-signatures-directory, so a site can verify that a request claiming to be CrawlForge really is.
  • robots.txt is honoured by default on both surfaces. A caller may pass respect_robots: false; the override is explicit, returns a warning in the response and is recorded against the API key in the usage log.
  • We do not solve CAPTCHAs or forge challenge tokens. A challenge page is reported as blocked, not bypassed. stealth_mode makes a real browser render like a real browser; it does not defeat access controls.
  • A site owner who opts out is added to a permanent blocklist that applies on both surfaces and cannot be overridden by any customer. Write to simon@crawlforge.dev to opt out.
  • The retired token CrawlForge-Bot is still honoured as a disallow rule, so rules site owners wrote before the rename keep working.

Credentials and Encryption

What we do with the secrets you give us, and what we do not claim.

  • All traffic to crawlforge.dev uses TLS, and the site sends a Strict-Transport-Security header.
  • Passwords are hashed with bcrypt. We never see or store the password itself.
  • API keys are hashed with SHA-256 and a server-side salt before they are stored, together with the key prefix shown in the dashboard. The full key exists in plain text once, in the response that creates it. We cannot recover it, so a lost key is revoked and replaced, not retrieved.
  • Sign-in sessions use a 15-minute JWT and a 7-day refresh token, both in httpOnly, secure, same-site cookies. State-changing dashboard requests carry a CSRF token.
  • Password-reset and email-verification links are single-use tokens stored as hashes with an expiry of 1 hour and 24 hours respectively.
  • Secrets in your requests, such as a password typed by scrape_with_actions, proxy credentials or an authorization header, are masked before a usage row is written and again before it is shown in the dashboard.
  • Neon and Upstash encrypt stored data at rest, and every connection to them is TLS.
  • What we do not claim: CrawlForge holds no SOC 2 or ISO 27001 certification, does not offer multi-factor authentication on accounts yet, and does not commission third-party penetration tests. If your procurement needs one of these, ask us where it stands before you buy.

Account Deletion

Delete your account yourself from Settings in the dashboard. Deletion is immediate and can't be undone.

  • Open Settings, choose Delete my account, and confirm with your email address, plus your password if the account has one.
  • Cancel any active subscription on the Billing page first; deletion stays locked until you do. A plan already set to end with its current period is ended at once. Unused credits are not refunded.
  • Deleting removes the account row, and with it your API keys, usage logs, subscription records, monitors, device fingerprints, risk scores, tokens, feedback-survey answers and newsletter subscription. Entries in Redis expire on the periods in the retention table, 90 days at most.
  • We keep one-way hashes of your email address and device fingerprints, linked to nothing else. A new account that matches one gets no free credits, so deleting an account can't be used to claim them twice. The hashes are kept indefinitely.
  • Stripe keeps payment records for as long as tax and accounting law requires.
  • Can't sign in? Email simon@crawlforge.dev from the address on your account with the subject "Delete my account", and we delete it for you within 30 days.

Vulnerability Disclosure

If you find a security issue in the website, the API, the crawlforge-mcp-server package or the SDKs, tell us directly. Details are also published at /.well-known/security.txt.

  • Email simon@crawlforge.dev with the subject "Security". Include the affected URL or tool, steps to reproduce and the impact you observed.
  • We acknowledge reports within five business days and keep you informed until the issue is fixed. We do not run a paid bug bounty.
  • Test only against your own account and API keys. Do not access, modify or delete other customers’ data.
  • No denial of service, no volume scanning, no social engineering of people who work on CrawlForge, and no testing against the target websites our tools fetch.
  • Stop as soon as you can demonstrate the issue and report it. Do not keep the data you reached.
  • We ask for reasonable time to fix an issue before it is published, and we will credit you in the fix if you want that.

Research that follows these rules is authorised. We will not pursue legal action against it and we will not report it to law enforcement.

Machine-readable contact: /.well-known/security.txt

Questions About Security?

If you need something this page does not answer, such as a data-processing agreement or a description of a control before you buy, write to us.

simon@crawlforge.dev