Changelog

New use cases page showcasing real-world CrawlForge applications, integrations hub for connecting with your favorite tools, and this changelog to track every update.

• Use cases page with industry-specific examples
• Integrations hub with setup guides
• Changelog page with full version history

Comprehensive SEO improvements across the entire site to boost search visibility and ensure correct indexing.

• Canonical URLs on all pages
• Static sitemap generation
• Enhanced structured data with JSON-LD schemas

Made CrawlForge discoverable by AI agents and crawlers with standardized machine-readable files.

• Added ai.txt for AI agent instructions
• Added llms.txt, llms-ctx.txt, and llms-full.txt
• IndexNow integration for instant search engine notification

Resolved critical bugs affecting performance and developer experience.

• Fixed Shiki singleton memory leak in blog syntax highlighting
• Resolved API reference tools 404 errors

Launched the CrawlForge blog with a library of tutorials, use cases, and web scraping guides.

• 18 launch posts covering tutorials and use cases
• MDX-based blog with syntax highlighting
• Category and tag filtering

Redesigned the user dashboard for a cleaner layout and better insight into usage patterns.

• Redesigned dashboard UI
• Improved usage analytics and charts
• Better credit balance visibility

Introduced automatic credit recharging and a comprehensive fraud prevention system to keep accounts secure.

• Credit auto-recharge when balance drops below threshold
• Device fingerprinting for fraud detection
• Risk scoring system with admin whitelist bypass

Improved reliability of Stripe webhook processing and fixed edge cases in the credit reservation system.

• Stripe webhook idempotency improvements with distributed locking
• Credit reservation edge case fixes for concurrent requests

New stealth_mode tool for accessing sites with advanced anti-bot protection using proxy rotation and browser fingerprint randomization.

• stealth_mode tool with proxy rotation
• Browser fingerprint randomization
• Anti-detection techniques for protected sites

Improved API key management and tightened rate limiting to ensure fair usage across all accounts.

• API key rotation improvements
• Enhanced rate limiting with sliding window

Premium deep_research tool that performs comprehensive multi-source analysis, synthesizing information from across the web into structured reports.

• deep_research tool (10 credits per use)
• Multi-source analysis with AI synthesis
• Structured report output with citations

CrawlForge MCP launches with a comprehensive suite of specialized web scraping tools, a credit-based billing system, and a full-featured developer dashboard.

• MCP tools for web scraping and data extraction
• Credit system with Stripe billing
• Developer dashboard with API key management
• Usage tracking and analytics