Firecrawl Release Notes

The Firecrawl Developer Index is now available, a specialized index for coding agents. It covers 70M+ artifacts across READMEs, external documentation, issues, pull requests, and OpenAPI specs, with semantic retrieval and metadata filters. Your AI agents answer questions about code behavior, API contracts, error messages, and known bugs from primary sources instead of general web pages.

Highlights

• Highest recall of any major coding-specific index. Scores 0.63 recall@10 across the 1,179 real developer queries in our open DevDex benchmark, ahead of every other provider we tested.
• 70M+ artifacts. READMEs, docs, issues, PRs, and OpenAPI specs from the most popular public repos and documentation sites, refreshed continuously with most sources updated daily.
• Answers with passages. Every result carries a stable id, a url, and the matched passages in markdown, so your agents act on the answer without a second scrape.
• Rich filters in the API. Scope by result type, repository, documentation source, language, topic, license, minimum stars, and more. These are API-only; agents on the CLI and MCP perform best without them.
• Keyless to start. No API key needed to try it; add one for higher rate limits. A developer search costs 2 credits per 10 results.
• Available everywhere you build. Query it through the API at /search/developer, plus the CLI, MCP, and SDKs, or install the companion skill with npx -y firecrawl-cli@latest setup developer-index.

Read the full blog here.

Original source

Coding agents spend a large share of their tool calls searching for one of three things:

• the repository that implements an idea
• the documentation page that answers a question
• the issue or pull request where a bug was fixed

That context is scattered across GitHub, docs sites, and threads, and the existing options for retrieving it are lexical, incomplete, or both.

Today we're launching the Firecrawl Developer Index, a specialized index for coding agents. It indexes the artifacts agents actually need to write working, current code (READMEs, external documentation, issues, pull requests, and OpenAPI specs) with semantic retrieval and metadata filters, refreshed daily.

Alongside the index, we're releasing DevDex, an open benchmark of 1,179 developer-search queries scored on Recall@10 and MRR@10, so teams can measure how well any retrieval system supports real coding-agent workflows.

Why we built the Firecrawl Developer Index

Coding agents are one of the biggest categories of what customers search and scrape for on Firecrawl. When we dug into what customers were actually retrieving, three patterns kept surfacing:

1. Agentic products (think Lovable, Replit, Bolt) doing backend debugging on behalf of end users.
2. Knowledge-base builders stitching internal and external repos into a single retrieval layer.
3. Frontier labs that need open developer documentation, code, issues, and PRs as training and evaluation data.

They were all working around the same gap. Existing providers weren't designed for a world of agents: search is lexical, not semantic, and getting complete artifacts (a README plus its issues plus its recent PRs) meant stitching together 50+ API calls, or building the whole pipeline from scratch. Even Firecrawl's general search and scrape gave subpar results for this shape of query, because a coding agent doesn't want a web page, it wants an artifact.

The Developer Index is that artifact layer.

What's in the Firecrawl Developer Index

• 70M+ artifacts across READMEs, pull requests, issues, OpenAPI specs, skills, and external documentation, refreshed continuously, with most sources refreshed daily.
• Issues and pull requests from top repositories, with their linked artifacts.
• READMEs from a broad set of public repositories.
• External documentation sources (Stripe, and everything of that shape).
• OpenAPI specs and popular skill repos.
• Metadata on every artifact: stars, licenses, artifact type.

To be clear about what it is not: the Developer Index does not store code, and it is not a general web search endpoint. It is a purpose-built retrieval layer organized around the artifacts coding agents produce and consume.

How the Firecrawl Developer Index works

You send a natural-language question to the Firecrawl Developer Index and get back ranked developer results with the passages that matched, so an agent can act on the answer without a second scrape. There are two ways to reach it:

• Firecrawl /search/developer returns developer sources only, with result-type, repository, and documentation-source filters. This is the surface to reach for when you want ranked developer results with matched passages.
• Firecrawl /search with categories: ["developer"] returns Developer Index results through the standard /search response, in the same shape as ordinary web results.

Every Firecrawl Developer Index result carries a stable id (like issue:owner/repo#123) whose prefix tells you the artifact kind (doc:, issue:, pull_request:, or readme:), a url, and its matched passages in markdown, so tables and code blocks survive. Through the API, filters let you scope by types, repos, sources, language, topic, license, min_stars, and more. You can also set skills: "only" to search indexed agent-skill files. These filters are API-only: on the CLI and MCP, agents perform best without them, so they are intentionally not exposed there.

No Firecrawl API key is needed to get started; add one for higher rate limits. A developer search costs 2 credits per 10 results, rounded up.

The easiest way to give your agent access is Firecrawl's dedicated developer skill, which plugs into the Firecrawl CLI or MCP server:

npx -y firecrawl-cli@latest setup developer-index

For the full parameter reference and response schema, see the Developer Index docs.

What is Firecrawl DevDex?

DevDex is an open benchmark for developer-search retrieval, scoring how well any system returns the right docs, GitHub pages, and Stack Overflow answers for real coding-agent queries. We built it because the standard search benchmarks don't reflect what agents actually look up while writing code.

How does DevDex measure developer search?

To measure whether a specialized artifact index like the Firecrawl Developer Index actually improves agent outcomes, we built DevDex. It's a benchmark of 1,179 developer-search queries across three tracks that mirror how coding agents actually retrieve.

We are releasing part of the dataset and the evaluation harness as an open-source evaluation so that DevDex can serve as a public standard for developer search on the web.

The three tracks:

• Repository discovery. Find the repo that matches a described capability without knowing its name. Example: "a library for incremental PDF parsing."
• Documentation lookup. Find the exact page that answers a how-to. Example: "How do I add Pydantic to my project using uv instead of pip?"
• Issue and PR resolution. Find where a specific bug was discussed and fixed. Example: "scikit-learn LogisticRegression random_state not working."

Each track is scored deterministically on Recall@10 and MRR@10 against fixed gold references, with a memorization check to drop any query the driver model can answer from pretraining. Every provider runs under a matched setup: same driver model (Claude Opus 4.8), one search tool active per run, same harness.

Results

The Firecrawl Developer Index leads overall recall, ahead of general web search and every other developer-search provider.

Recall@10 measures whether the correct artifact appears anywhere in the top ten results. Higher is better. Each overall score is the mean across the three tracks. The Firecrawl Developer Index scores 0.63, ahead of Firecrawl Search (no category) at 0.58, Parallel at 0.57, and Mintlify and Exa at 0.54. Native web search sits at 0.45, and Context7 at 0.17.

Broken out by track, the Firecrawl Developer Index leads issue and PR resolution at 0.66, and is statistically tied with Context7 on documentation lookup (both at 0.47, a 0.006 gap well inside the 95% CI). Context7 is docs-focused, and scores near zero on the other two tracks (0.01 on repository discovery, 0.03 on issues and PRs). On repository discovery the Developer Index posts 0.76, behind Parallel at 0.82 and Firecrawl Search at 0.78.

The gap between the "no tools" control and every other row is the size of the retrieval problem: coding agents can't answer these queries from pretraining alone. The gap between native web search and the specialized indexes is what a purpose-built artifact layer buys you.

Release

We're open-sourcing half of the dataset plus the evaluation harness so any team can reproduce results on their own systems.

• Repo: github.com/firecrawl/benchmark-devdex
• To submit a provider: open a PR against the repo with your results on the public half, and email [email protected] with valid API keys so we can rerun on the held-out half.

What you can build with the Firecrawl Developer Index

• Ship an agentic product that debugs its own code. Wire the Firecrawl Developer Index into your agent's tool loop and let it search issues and PRs the moment it hits an error, instead of guessing.
• Build a developer knowledge base. Use the Firecrawl Developer Index to pull external documentation and repo artifacts alongside your internal sources, without maintaining a GitHub scraper.
• Train or evaluate coding models. Use the Firecrawl Developer Index as a retrieval layer for RAG-style training data pipelines, and use DevDex to score whatever retriever you build.

Try it today

Firecrawl Developer Index is available now in the API, CLI, MCP, and SDKs. It plugs into any harness you already run, including Codex, Claude Code, and Grok Build.

Read the Firecrawl Developer Index docs

Original source

Today we're launching the Life Sciences category in the Firecrawl Research Index. It covers 41M+ papers across the drug discovery, clinical trial, and biology literature. Your AI agents get citable papers back for a query and can pull the full text on demand.

The whole Research Index is also free to use now. That includes the AI and ML literature it already carried alongside the new Life Sciences papers.

How the Life Sciences category makes research easier

• High-recall retrieval. The index hits 90% recall@10 on our paper-retrieval eval, so your AI agents find more of what matters on the first page of results.
• Millions of life sciences papers. The corpus holds 41M+ papers from authoritative sources and refreshes daily, so your AI agents only cite domain-specific literature.
• Abstract to full text. Your AI agents search abstracts to find the right papers, then pull the full text to verify a claim against the source.
• No DIY stack. Research Index stands in for the source APIs plus the parsing and ranking you would otherwise build yourself.

What you can use it for

• Build and improve life sciences models. Your AI agents call the index to support internal work on chemical compounds and predictive biology.
• Power a research platform's answers. Use the index as the backend for your platform. A user asks a question and the index returns the papers and passages your platform needs to answer it.
• Run targeted academic and clinical research. Point the research harness you already use at the index and your AI agents retrieve life sciences literature scoped to your objective.

Try it today

The Life Sciences category is live now in Firecrawl Research Index through the API at /search/research, plus the CLI, MCP, and SDKs. Every category is free to query.

Original source

The Life Sciences category is now available in Firecrawl Research Index. It covers 41M+ papers across the drug discovery, clinical trial, and biology literature. Your AI agents get citable papers back for a query and can pull the full text on demand. The whole index is also free to use now.

Highlights

• High-recall retrieval. The index hits 90% recall@10 on our paper-retrieval eval, so your AI agents find more of what matters on the first page of results.
• Millions of life sciences papers. The corpus holds 41M+ papers from authoritative sources and refreshes daily, so your AI agents only cite domain-specific literature.
• Abstract to full text. Your AI agents search abstracts to find the right papers, then pull the full text to verify a claim against the source.
• No DIY stack. Research Index stands in for the source APIs plus the parsing and ranking you would otherwise build yourself.
• Free to use. Every category is free, the AI and ML literature it already carried alongside the new Life Sciences papers.
• Available everywhere you build. Query it through the API at /search/research, plus the CLI, MCP, and SDKs.

Read the full blog here.

Original source

Firecrawl pdf-inspector

What is Firecrawl pdf-inspector

pdf-inspector is a from-scratch Rust library for PDFs, with 1.9M views and 8.1k GitHub stars on the repo at time of writing. It reads a PDF's internal structure (font encodings, text operators, image coverage) in milliseconds, without rendering anything, and decides per page whether the content is text-based or needs OCR.

• Text-based pages get native extraction directly, with reading order preserved.
• Scanned or image-heavy pages are flagged with the reason, so a vision pipeline can pick them up.

For a fully text-based PDF, pdf-inspector alone is the entire pipeline. For mixed documents, it's the smart router that decides what actually needs a GPU.

Why it matters on its own

Most PDF pipelines make the same wrong bet: treat every page as if it might be scanned, so route everything through OCR. That's slow, expensive, and often less accurate than the native text that was sitting in the PDF the whole time.

pdf-inspector fixes the routing layer:

• Per-page classification. Analyzes internal structure only. No rendering, no GPU, milliseconds per page.
• Native text extraction. Pulls text directly from text-based pages with reading order intact.
• Clean handoff for the rest. For scanned pages, it hands back the page reference and the reason, so an OCR pipeline can do the heavy lifting only where it's needed.

That routing layer is what makes Fire-PDF, Firecrawl's hosted PDF parsing engine, 3.5x to 5x faster than the previous pipeline: for a 200-page report where 150 pages are pure text, 150 pages skip GPU entirely.

How to use it

Add pdf-inspector to your Rust project directly from the repo. The README covers the classifier and native-extraction APIs, plus the reproducible-results branch if you want to benchmark it against your own PDFs.

Or don't integrate it directly at all: send PDFs to Firecrawl's /parse or /scrape and they go through pdf-inspector (and Fire-PDF for the scanned pages) automatically.

Firecrawl AnyDoc

What is Firecrawl AnyDoc

AnyDoc is the other half of the stack, also a from-scratch Rust library, also markdown out. It converts documents with a single call:

anydoc::to_markdown("file.docx")

It supports 14 formats in one binary:

• docx
• doc
• docm
• xlsx
• xls
• xlsm
• pptx
• ppt
• rtf
• odt
• ods
• odp
• epub
• csv

No API key. No system dependencies. Nothing to install alongside it.

Why it matters

No single existing library reliably covers every common document format. Each one handles a subset, and the formats it doesn't cover become someone else's dependency, with different output, different failure modes, and often much slower conversions. AnyDoc handles all 14 in one dependency-free library.

Who feels this most: the engineer whose users upload "whatever they have": a .docx contract, an .xls export from 2009, a pitch deck, an .epub. Today, getting usable text out of all of it is a plumbing project. AnyDoc is the part that stops being plumbing.

It's also quickly become a dev favorite. Here's Garry Tan on it:

[Embedded Tweet]

Here's Nick, Firecrawl's Co-founder and CTO, on how AnyDoc is different from pdf-inspector:

[Embedded Tweet]

How it compares

We benchmarked AnyDoc against the common alternatives on 94 documents spanning all 14 formats.

• Coverage: 14 of 14. AnyDoc is the only library in the benchmark that parses every format. The nearest alternative, LibreOffice, covers 12 of 14. Every other option covers a subset, which is exactly why teams end up stitching multiple libraries together.
• Speed: 4.6ms median per document. Across all 14 formats, AnyDoc's median conversion time is 4.6ms. The alternatives run between 52ms and 1,130ms, roughly 20x to 245x slower depending on the tool. LibreOffice sits at the slow end of that range.
• Quality: highest overall, with honest caveats. Quality is LLM-judged on completeness, structure, formatting, and cleanliness. AnyDoc scored highest on every format we tested, overall score 80 vs 68 for the next-best option. Two notes worth being upfront about: the corpus is ours, and mammoth scores higher than AnyDoc on completeness alone (95 vs 87) on the single format it supports (docx). If completeness on docx is the only thing you care about, that gap matters. If you care about coverage across every format your users actually upload, AnyDoc wins the aggregate.

How to use it

Add AnyDoc to your Rust project, then call it on any supported document:

use anydoc;
let markdown = anydoc::to_markdown("contract.docx")?;

That's the whole integration. No API key, no external service, no separate binary. JavaScript bindings are in progress; we'll share those when they land.

Or don't integrate anything at all: Firecrawl's /parse and /scrape endpoints already use AnyDoc automatically when they encounter a non-PDF document.

Two libraries, one shape

Same principles across the pair: from-scratch Rust, local execution, no API key, no system dependencies, markdown out. That's not a coincidence. It's the shape developers kept asking for after pdf-inspector shipped, and it's the shape that makes both libraries safe to drop into any pipeline without dragging in a heavy runtime.

The two are deliberately separate repos and deliberately separate products. AnyDoc is not "pdf-inspector grown up." pdf-inspector handles PDFs. AnyDoc handles everything else. One lineage, two Firecrawl products, and together they cover the document formats a real AI pipeline actually sees.

Try the stack

• pdf-inspector (PDFs): github.com/firecrawl/pdf-inspector
• AnyDoc (14 non-PDF formats): github.com/firecrawl/anydoc
• Use both via API: send any document to /parse or /scrape. PDFs go through pdf-inspector (and Fire-PDF); everything else goes through AnyDoc. No configuration.

If you're already stitching four libraries together to cover the document formats your users upload, we'd love to hear how the two of these hold up as a replacement.

Original source

Firecrawl is now available as an official ChatGPT plugin. You can install it directly inside ChatGPT and give it access to live, clean web data in seconds.

Getting started

Install the plugin in three steps:

1. Open ChatGPT
2. Go to the Firecrawl plugin page and click Install
3. Connect your Firecrawl account when prompted

That's it. ChatGPT now has direct access to live web data across every conversation and inside Codex.

What you can do

Once connected, the plugin gives ChatGPT a full web data toolkit:

• Search the web and get back the excerpts that best answer your query, with full page content included in the results.
• Scrape any page or document (PDF, DOCX, and more) and turn it into clean, structured data.
• Crawl entire sites to build datasets and knowledge bases.
• Interact with dynamic pages to reach content that regular scraping misses (logins, forms, pagination, infinite scroll).
• Monitor pages and get alerts whenever something on the web changes. Monitors can post directly to Slack, so the alert lands where your team already works.

You don't need to remember any commands. Just ask in plain language and ChatGPT picks the right Firecrawl tool automatically. The plugin also ships with a set of ready-to-use skills, so you can kick off common tasks (searching the web, scraping a page, crawling a site, running an interaction flow) in one click without writing the prompt from scratch.

Example prompts

Search the web for this week's biggest AI announcements

Scrape https://news.ycombinator.com and list the top 10 stories with links and points

Interact with Campspot.com to find campsites near Yosemite available Aug 15 to 17

Monitor https://openai.com/blog for new posts and send updates to my #product-updates Slack channel

Why this matters

ChatGPT is only as fresh as the web it can reach. Its built-in browsing gives you snippets and summaries, but the moment you need real answers grounded in real sources, you hit walls: shallow search results with no page content, noisy and unstructured pages, JavaScript that renders inconsistently, and messy HTML that ChatGPT can't parse properly. Firecrawl fixes both sides of that problem.

Firecrawl search runs the query, picks the best results, and hands ChatGPT the full page content along with the links. You get the excerpts that answer the question instead of ten titles and a guess. For pages or sites you already have in mind, Firecrawl handles JavaScript, dynamic content, PDFs, and DOCX, and gives ChatGPT clean markdown or structured JSON to work with.

Together, ChatGPT can go from a vague question to a well-sourced answer without you leaving the conversation. Same inside Codex: search for how something works, then pull the exact docs it needs to write and reason about your code.

Example

Search the web for the latest guidance on receiving Stripe events in a webhook endpoint. Pull the relevant Stripe docs and implement it in my app.

No more copy-pasting URLs into separate tools. No more writing custom scrapers. No more prompting around stale search snippets. Just ask ChatGPT to find and pull the data you need, and it works.

Use cases

A few ways developers, marketers, growth engineers, and sales folks are already using the plugin:

• Live research with real sources: Search the web for what's happening right now and get back the actual page content, not just links or stale summaries.
• Answer engine over any topic: Ask a question in plain language, let Firecrawl search and pull the top results, and have ChatGPT synthesize a grounded answer with citations.
• Documentation lookups: Search a topic or crawl a specific docs site, then get answers, comparisons, or ready-to-use code snippets inside your conversation.
• Competitive intelligence: Search for competitor coverage or scrape pricing and feature pages directly as structured data.
• Lead enrichment: Search for a company, then scrape its site for contact info, tech stacks, or product details.
• Change monitoring: Track pricing pages, job boards, or release notes and get notified when something changes.
• Dynamic sites: Use the interact capability to reach content behind logins, forms, or JavaScript-heavy flows.

Try it out

The plugin is live now. Install it from ChatGPT and start pulling live web data into your ChatGPT and Codex workflows.

• Install the plugin
• Read the docs
• Try Firecrawl

We'd love to hear what you build with it.

Original source

Firecrawl /search now returns the excerpts that best answer your query from each result. Agents using it achieve state-of-the-art accuracy on SimpleQA while using 10x fewer tokens than processing full pages.

Highlights

• Custom relevance model — A custom model scores every paragraph, list, and table against your query, returning the most relevant excerpts structured and ready for your agent to use.
• State-of-the-art accuracy on SimpleQA — Agents using Firecrawl /search score 94.7% on SimpleQA, higher than any other provider.
• 10x fewer tokens than full pages — The relevant excerpts returned by /search take up far less of your agent's context window and are cheaper and faster to reason over.
• Nothing to change in your code — The API shape is unchanged, so existing /search calls get better results automatically.

Read the full announcement here.

Original source

Today we're shipping a major upgrade to Firecrawl /search: a new custom relevance model that returns the excerpts that best answer your query from each result. Agents using /search achieve state-of-the-art accuracy on SimpleQA while using 10x fewer tokens than processing full pages.

Agents rely on search tools to find answers on the web, but those answers are often buried in noise. Navigation, boilerplate, and tangents wrap around the excerpt that matters, so an agent processes far more content than it needs to, which is slower, more expensive, and more likely to land on the wrong answer.

Our /search endpoint now runs every result through the new custom relevance model, returning only the excerpts that answer your exact query.

How the improved Firecrawl /search works

Our custom model scores every paragraph, list, and table against your query. It surfaces the most relevant excerpts wherever they appear on the page, then returns them structured and ready for your agent to use.

State-of-the-art accuracy on SimpleQA

SimpleQA is OpenAI's factuality benchmark, measuring whether models correctly answer short, fact-seeking questions. In our eval, agents using Firecrawl /search score 94.7%, higher than with any other provider we tested.

Evaluation methodology

A GPT-5.4 agent with high reasoning effort runs with up to 20 tool calls: search_web, backed by the provider under test, plus web_fetch through that provider's own extract API (Firecrawl Scrape, Parallel Extract, or Exa Contents). Claude native search is Claude Sonnet 4.6 with Anthropic's server-side web search, evaluated as a complete system.

Answers are graded by an LLM judge using GPT-5.4 and the official SimpleQA grader prompt. The full SimpleQA set of 4,326 questions was run across two sessions per provider, with the best observed score selected. As a baseline, GPT-5.4 with no search tools scores 43.8 under the same judge.

10x fewer tokens than full pages

By returning key excerpts instead of full-page content, /search leaves more of an agent's context window available for reasoning. Fewer input tokens also make downstream model calls cheaper and faster.

What you can build with it

• Multi-step agent search: Keep an agent's context lean across long chains of searches by returning only the excerpts that answer each query.
• RAG and LLM grounding: Add relevant excerpts directly to a prompt without separate chunking or reranking.
• Lead enrichment: Find specific facts about a person, company, or domain, including titles, revenue, and hiring signals.
• Research and market intelligence: Pull a specific claim, figure, or quote without processing the rest of the page.

Nothing to change in your code

Existing /search calls return more relevant context automatically, with no changes required. The API shape is unchanged, and each result still includes its title, URL, and description alongside query-relevant Markdown excerpts by default. If you need complete page content, you can still retrieve full-page Markdown in the same call:

from firecrawl import Firecrawl
firecrawl = Firecrawl(api_key = "fc-YOUR-API-KEY")
results = firecrawl.search(
  "what changed in the EU AI Act enforcement timeline",
  limit = 5,
  scrape_options = {"formats": ["markdown"]},
)
for result in results.web:
  print(result.title)
  print(result.url)
  print(result.markdown)

Try it today

The upgraded /search is live for every Firecrawl user, across the API, SDKs, CLI, and MCP.

1. Read the /search documentation
2. Try it in the Playground
3. Point your agent at /search and give it state-of-the-art search accuracy

Try /search in the Playground · Read the docs

Original source

Today we're launching web-scale /monitor

an always-on search that watches the web and pings you or your agent the moment something comes online.

Before, /monitor only worked for single pages or websites. You named the URLs, Firecrawl diffed them on a schedule, and notified you when something changed. Now you can use that same power on the entire web.

Define a query and a goal, and Firecrawl runs the search on your schedule, dedupes results, and alerts only when new, relevant pages appear. Every run delivers what's new and meaningful to your AI agent or app, via webhook or email.

What is web-scale /monitor?

Web-scale /monitor finds new pages across the web as they appear. You define one or more search queries, a recency window, and a plain-language goal. On every check, Firecrawl:

1. Runs the queries against the web
2. Filters to results inside your searchWindow
3. Dedupes by canonical URL against prior checks
4. Judges new results against your goal (when judging is enabled)
5. Pings you or your agent when something worth acting on comes online

Pages and sites vs. the entire web

Same scheduling, goals, judging, and notification channels underneath. If you've used /monitor for page watching, the API shape will feel familiar: web monitors use a search target instead of URLs.

Create your first web monitor

A web monitor is a standard create_monitor call with a type: "search" target:

[Code examples in Python and Node.js omitted for brevity]

Tune recall and precision

Two levers do different jobs:

• queries control recall: what each search pulls in. Cast a wide enough net with relevant terms and variations, then let your goal decide what actually alerts.
• goal controls precision: which retrieved results actually alert.

Good queries read like search terms, not sentences. Use includeDomains and excludeDomains for domain scoping instead of site: operators in the query string.

Get pinged by webhook or email

Point a webhook at your agent and subscribe to monitor.page and monitor.check.completed, the same events page monitors use. When a new matching result comes online, the payload arrives with status: "new" and a judgment explaining why it matters.

Email summaries work the same way: sent only when a check surfaces new or errored results.

What you can build with web-scale /monitor

Use case | Example queries | Goal

Track regulatory and legal changes | FDA approval, clinical trial posting, SEC filing | Alert when a new approval, trial, or filing is published

Monitor competitor developments | "Acme Corp" funding OR acquisition, competitor careers site | Alert on funding rounds, new roles, or product launches

Track breaking news and events | Phase 3 trial results biotech, M&A filing | Alert when trial results or deal filings appear

Replace the DIY stack

Before web-scale /monitor, staying current on the open web meant wiring together a scheduler, a search API, deduplication logic, a relevance filter, and a notification layer, then maintaining all of it as queries and sources shift.

One endpoint replaces that stack. You define the query once; Firecrawl handles the always-on search, dedupe, classification, and delivery.

Pricing: web monitors cost 2 credits per 10 results per check, plus 1 credit per result the judge evaluates when judging is enabled. There is no separate per-monitor fee.

Try it today

Web-scale /monitor is live for all Firecrawl users.

1. Create a monitor in the dashboard
2. Read the Web Monitoring docs
3. Wire the monitor.page webhook into your agent or pipeline

If you're already watching known URLs with /monitor, add a search target alongside them: page monitors for what you know, web monitors for what hasn't been published yet.

Get started with web-scale /monitor · Read the docs

Original source

Web-scale /monitor watches the entire web for you. Define search queries and a goal, and Firecrawl pings you or your agent the moment something new comes online.

Highlights

• Always-on web search — Give /monitor search queries and a goal, and it searches the entire web on your schedule, alerting you when something new matches.
• Tune recall and precision — queries control what each search pulls in, while goal decides which of those results actually alert.
• Webhook or email delivery — Get pinged the moment something new matches, with a plain-English explanation of why it matters.
• One endpoint replaces the DIY stack — Define your query once. Firecrawl handles the search, deduplication, filtering, and alerts.

Read the full documentation here.

Original source

Firecrawl v2.11.0 ships the Firecrawl Research Index, keyless endpoint access, automatic PII redaction, a deterministicJson format, and video discovery on any page - plus much more.

Highlights

• Firecrawl Research Index — Search across 3M+ arXiv papers and the GitHub code behind them (issues, merged PRs, and READMEs, refreshed daily), fetch a paper's details or related work, and check claims against full text. It has state-of-the-art recall on arXivQA, outperforming the next best provider by 18% at comparable cost.
• Keyless access for core endpoints — Use /scrape, /search, /interact, and /parse without an API key from official MCP, CLI, and SDK clients.
• Automatic PII redaction — A new redactPII option strips personal and sensitive data like names, emails, phone numbers, addresses, and secrets out of scraped content before it's returned.
• deterministicJson format — Get structured JSON without running an LLM on every request. Firecrawl generates a reusable extractor for your schema and caches it per site, so repeat scrapes are cheaper and return consistent results.
• Video discovery on any page — The video format now finds videos on any page, not just supported providers like YouTube, returning each video's URL, title, thumbnail, duration, and more.

Read the full changelog here.

Original source

AI/ML research moves fast, and the work that matters is split between new papers and the code that implements them. Most search providers omit or misrank key papers, leaving you to review sources by hand without ever being sure you've caught everything.

Today we're launching Firecrawl Research Index, a specialized index for agents pushing the frontier of AI/ML research. It gives agents the entire AI/ML literature and the code behind it, plus the tools to turn them into answers — agents retrieve the right papers, verify claims against full text, and pull code for implementation autonomously.

On arXivQA, the index has state-of-the-art recall, 18% above the next best provider at similar cost. It also scores 0.750 MRR, meaning the correct paper lands in the top two results.

How the Firecrawl Research Index makes agentic deep research easier

Highest recall, at comparable cost

On arXivQA, the index hits 53.3% recall at $0.32 per task, against 45.4% for the next best provider. Your agent works from a fuller set of papers without paying a premium.

Aemon (YC W26), which builds autonomous AI research engineers, saw the same pattern in their own benchmark of scientific and technical retrieval systems:

"Aemon is building autonomous AI research engineer that solve hard scientific and technical problems. To do that, our systems must continuously learn from the frontier of research—papers, implementations, benchmarks, and technical discussions across the web.
We use Firecrawl Research as part of the retrieval stack behind Aemon. In our internal benchmark of scientific and technical retrieval systems, it delivered the strongest recall of any provider we tested, particularly at deeper search depths where comprehensive coverage is critical. Firecrawl consistently surfaced relevant scientific and technical sources that would otherwise have been missed."
— Ray Xu, Co-Founder, Aemon (YC W26)

Surface the right source first

On arXivQA, the index scores 0.750 MRR, meaning the correct paper lands in the top two results. Higher MRR means fewer wasted tokens before an agent finds what it actually needs.

Search millions of papers alongside their code

The index includes all 3M+ arXiv papers, plus GitHub artifacts from top research repos (issues, merged PRs, READMEs), refreshed daily so agents always stay current.

A complete toolset for research loops

The built-in toolset lets agents run research end-to-end — retrieving the right papers, verifying claims against full text, and pulling code. Agents can go from literature to implementation in one query, with no manual filtering, cross-referencing, or review required.

What you can use it for

• Power your research platform's search: Plug in the index, and your platform ships state-of-the-art search across millions of papers and the code behind them.
• Autonomous research agents: Your agent finds the most relevant papers and code for its problem, follows citations, and verifies against full text before it builds on them. An agent tuning a training run overnight could pull an optimizer from a recent paper and a stability fix from a related GitHub issue, then test both in its next experiment.
• Literature reviews and discovery: Find the relevant work on any topic, including papers published this week. Then start from your best hit and surface its references, citers, and related work, reaching papers a keyword search misses.

Methodology

We benchmarked on roughly 200 queries from alphaXiv's ArXivQA, each labeled with up to 10 ground-truth arXiv IDs. To measure recall, we let Opus 4.8 run each provider through its MCP and SKILL.md, then scored the papers it surfaced against those labels.

Availability

Firecrawl Research Index is available now in the API via /search/research, plus the CLI, MCP, and SDKs. It plugs into any harness you already run, including Codex, Claude Code, and Grok Build.

Get started with Firecrawl Research Index

Original source

Setting up a web data API usually starts with a signup form. Create an account, generate a key, paste it into an env file, write your first line of code. That's friction before you've built anything.

Today we're launching Firecrawl Keyless. Search, scrape, and interact with the web without an API key. Every developer gets 1,000 free credits a month, automatically. Sign up only when you need more.

With Firecrawl Keyless, you can:

• Search the web for live results with full-page content
• Scrape any URL for clean markdown, including JavaScript-heavy pages
• Interact with pages to click, fill forms, paginate, and navigate dynamic sites

Live today across our MCP, CLI, and API.

How Firecrawl Keyless makes building easier

Agents work without setup

For coding agents, this matters even more. Connect Claude Code, Cursor, OpenClaw, Hermes Agent, OpenCode, or any other MCP-compatible host, and the agent starts scraping and searching immediately. No human in the loop to generate a key and paste it into config.

Demos and prototypes that don't break

API keys fail at the worst moment: a hackathon demo, a workshop, a side project picked up months later. Keyless removes that class of failures for projects where 1,000 credits a month is plenty.

Try it today

Firecrawl Keyless is live across every surface:

• MCP - point any MCP-compatible client at https://mcp.firecrawl.dev/v2/mcp
• CLI - run npx firecrawl-cli@latest and start scraping
• API - call the Firecrawl REST endpoints directly, no Authorization header required

You get 1,000 free credits per month, every month. When you outgrow it, sign up and bring your own key.

Get started with Firecrawl Keyless · Read the docs

Original source

Firecrawl Research Index is a specialized index for agents pushing the frontier of AI/ML research. It has state-of-the-art recall on arXivQA, outperforming the next best provider by 18% at comparable cost.

Highlights

• Benchmark-leading performance — On arXivQA, the index hits 53.3% recall versus 45.4% for the next best provider. The index also scores 0.750 MRR, putting the right paper in the top one or two results.
• Search millions of papers alongside their code — The index includes 3M+ arXiv papers, plus GitHub artifacts from top research repos (issues, merged PRs, READMEs), refreshed daily.
• A complete toolset for research loops — Agents can retrieve papers, verify claims against full text, and pull code, going from literature to implementation with no manual review.
• Plugs into any agent harness — Works with Codex, Claude Code, Grok Build, and more.

Read the full documentation here.

Original source

Agents can easily scrape any web page with Firecrawl. But knowing the moment a page changes — and staying focused on the changes that actually matter — is trickier. Traditionally, this required writing a cron job, standing up snapshot storage, diffing old against new, building a webhook layer with retries, and filtering out the noise (ads, timestamps, session tokens).

Today we're launching Firecrawl /monitor — the whole monitoring stack in one endpoint, only pinging your agent when something meaningful changes.

What is Firecrawl Monitoring?

/monitor runs scheduled checks on pages or entire sites, and notifies your agent when something you care about changes. Simply describe what you want to monitor in plain English, and Firecrawl configures the URLs, logic, and schedule for you.

The output is a structured diff: what was added, what was removed, what changed. Your agent only ingests what changes on a page, skipping unchanged content and noisy diffs — using up to 90% fewer LLM tokens.

How does Firecrawl /monitor keep your agents current?

Know the moment a page changes

Set a cadence (every 5 minutes, hourly, daily, or a custom cron schedule) and Firecrawl checks the page for you. When a change is detected, your agent gets a notification. When nothing has changed, nothing is sent. No polling loops, no wasted checks.

Watch only what matters

You can monitor a single page or an entire site, and track the whole page or drill down to specific fields. You can also edit content filters and scrape behavior manually if you need more precise control.

See exactly what's different

Every notification includes a rich diff (what was added, what was removed, what changed), formatted for easy reading by a human or an agent. Each check also gets a permalink so you can share a specific change with a teammate or hand it directly to an agent for further processing.

Deliver changes straight to your agent

Changes fire as signed webhooks with custom headers and per-event subscriptions. Your agent only wakes for the events it cares about. Email delivery works the same way, with the diff already in the message body.

Know your costs before you commit

See the estimated monthly cost before you turn a monitor on. You know exactly what your chosen schedule will cost before you commit to it.

How the /monitor pipeline works

1. Create a monitor for one or more web pages, or an entire site
2. Firecrawl scrapes the pages on your schedule and stores a snapshot
3. On each check, the new version is diffed against the last stored snapshot
4. If something changed, Firecrawl sends a signed webhook or email with the structured diff and a permalink

What you can build with /monitor

Use case What it looks like Triggering agents on change A deep research agent monitoring dozens of sources can generate a fresh summary the moment one of them updates. Keeping RAG pipelines fresh When a watched page changes, Firecrawl pings your pipeline with the diff so you can refresh just that document instead of rescraping everything. Tracking competitors and markets Watch competitor pricing pages, product catalogs, or job listings and know within minutes when something shifts: a price drop, a new SKU, a new role. Monitoring sources you rely on Get updates for new research papers, developer docs, changelogs, financial filings, and government regulations.

Try it today

Firecrawl Monitoring is live for all Firecrawl users. Create your first monitor from the dashboard or via the API.

Get started with Firecrawl Monitoring · Read the docs

Original source