AI Release Notes

What's new

AI Crawl Control now supports per-crawler drilldowns with an extended actions menu and status code analytics. Drill down into Metrics, Cloudflare Radar, and Security Analytics, or export crawler data for use in WAF custom rules, Redirect Rules, and robots.txt files.

Status code distribution chart

The Metrics tab includes a status code distribution chart showing HTTP response codes (2xx, 3xx, 4xx, 5xx) over time. Filter by individual crawler, category, operator, or time range to analyze how specific crawlers interact with your site.

Extended actions menu

Each crawler row includes a three-dot menu with per-crawler actions:

• View Metrics
• Filter the AI Crawl Control Metrics page to the selected crawler.
• View on Cloudflare Radar
• Access verified crawler details on Cloudflare Radar.
• Copy User Agent
• Copy user agent strings for use in WAF custom rules, Redirect Rules, or robots.txt files.
• View in Security Analytics
• Filter Security Analytics by detection IDs (Bot Management customers).
• Copy Detection ID
• Copy detection IDs for use in WAF custom rules (Bot Management customers).

Get started

1. Log in to the Cloudflare dashboard, and select your account and domain.
2. Go to AI Crawl Control > Metrics to access the status code distribution chart.
3. Go to AI Crawl Control > Crawlers and select the three-dot menu for any crawler to access per-crawler actions.
4. Select multiple crawlers to use bulk copy buttons for user agents or detection IDs.

Learn more about AI Crawl Control.

AI Search now supports reranking for improved retrieval quality and allows you to set the system prompt directly in your API requests.

Rerank for more relevant results

You can now enable reranking to reorder retrieved documents based on their semantic relevance to the user’s query. Reranking helps improve accuracy, especially for large or noisy datasets where vector similarity alone may not produce the optimal ordering.
You can enable and configure reranking in the dashboard or directly in your API requests:

Set system prompts in API

Previously, system prompts could only be configured in the dashboard. You can now define them directly in your API requests, giving you per-query control over behavior. For example:

Learn more about Reranking and System Prompt in AI Search.

Robots.txt tab

AI Crawl Control now includes a Robots.txt tab that provides insights into how AI crawlers interact with your robots.txt files.

The Robots.txt tab allows you to:

• Monitor the health status of robots.txt files across all your hostnames, including HTTP status codes, and identify hostnames that need a robots.txt file.
• Track the total number of requests to each robots.txt file, with breakdowns of successful versus unsuccessful requests.
• Check whether your robots.txt files contain Content Signals directives for AI training, search, and AI input.
• Identify crawlers that request paths explicitly disallowed by your robots.txt directives, including the crawler name, operator, violated path, specific directive, and violation count.
• Filter robots.txt request data by crawler, operator, category, and custom time ranges.

Take action

When you identify non-compliant crawlers, you can:

• Block the crawler in the Crawlers tab
• Create custom WAF rules for path-specific security
• Use Redirect Rules to guide crawlers to appropriate areas of your site

To get started, go to AI Crawl Control > Robots.txt in the Cloudflare dashboard. Learn more in the Track robots.txt documentation.

What's new

Track crawler requests over time

• By Crawler — Track activity from individual AI crawlers (GPTBot, ClaudeBot, Bytespider)
• By Category — Analyze crawler purpose or type
• By Operator — Discover which companies (OpenAI, Anthropic, ByteDance) are crawling your site
• By Host — Break down activity across multiple subdomains
• By Status Code — Monitor HTTP response codes to crawlers (200s, 300s, 400s, 500s)

Analyze referrer data (Paid plans)

• View top referrers driving traffic to your site
• Understand discovery patterns and content popularity from AI operators

Export data

• Download your filtered view as a CSV:
  • Includes all applied filters and groupings
  • Useful for custom reporting and deeper analysis

Get started

1. Log in to the Cloudflare dashboard, and select your account and domain.
2. Go to AI Crawl Control > Metrics.
3. Use the grouping tabs to explore different views of your data.
4. Apply filters to focus on specific crawlers, time ranges, or response codes.
5. Select Download CSV to export your filtered data for further analysis.

Learn more about AI Crawl Control.

Deepgram's Flux model release on Workers AI

Deepgram's newest Flux model @cf/deepgram/flux is now available on Workers AI, hosted directly on Cloudflare's infrastructure. We're excited to be a launch partner with Deepgram and offer their new Speech Recognition model built specifically for enabling voice agents. Check out Deepgram's blog for more details on the release.

The Flux model can be used in conjunction with Deepgram's speech-to-text model @cf/deepgram/nova-3 and text-to-speech model @cf/deepgram/aura-1 to build end-to-end voice agents. Having Deepgram on Workers AI takes advantage of our edge GPU infrastructure, for ultra low latency voice AI applications.

Promotional Pricing

For the month of October 2025, Deepgram's Flux model will be free to use on Workers AI. Official pricing will be announced soon and charged after the promotional pricing period ends on October 31, 2025. Check out the model page for pricing details in the future.

Example Usage

The new Flux model is WebSocket only as it requires live bi-directional streaming in order to recognize speech activity.

• Create a worker that establishes a websocket connection with @cf/deepgram/flux
• Deploy your worker
• Write a client script to connect to your worker and start sending random audio bytes to it

Learn more about Workers AI.

Browser Rendering Updates

• Playwright support is now Generally Available and synced with Playwright v1.55, giving you a stable foundation for critical automation and AI-agent workflows.
• We’re also adding Stagehand support (Beta) so you can combine code with natural language instructions to build more resilient automations.
• Finally, we’ve tripled limits for paid plans across both the REST API and Workers Bindings to help you scale.

To get started with Stagehand, refer to the Stagehand example that uses Stagehand and Workers AI to search for a movie on this example movie directory, extract its details using natural language (title, year, rating, duration, and genre), and return the information along with a screenshot of the webpage.

AutoRAG is now AI Search

The new name marks a new and bigger mission: to make world-class search infrastructure available to every developer and business.

With AI Search you can now use models from different providers like OpenAI and Anthropic. By attaching your provider keys to the AI Gateway linked to your AI Search instance, you can use many more models for both embedding and inference.

To use AI Search with other model providers:

• 1. Add provider keys to AI Gateway
  • 1. Go to AI > AI Gateway in the dashboard.
  • 1. Select or create an AI gateway.
  • 1. In Provider Keys, choose your provider, click Add, and enter the key.
• 1. Connect a gateway to AI Search: When creating a new AI Search, select the AI Gateway with your provider keys. For an existing AI Search, go to Settings and switch to a gateway that has your keys under Resources.
• 1. Select models: Embedding models are only available to be changed when creating a new AI Search. Generation model can be selected when creating a new AI Search and can be changed at any time in Settings.

Once configured, your AI Search instance will be able to reference models available through your AI Gateway when making a /ai-search request:

In the coming weeks we will also roll out updates to align the APIs with the new name. The existing APIs will continue to be supported for the time being. Stay tuned to the AI Search Changelog and Discord for more updates!

AutoRAG Metrics

AutoRAG now includes a Metrics tab that shows how your data is indexed and searched. Get a clear view of the health of your indexing pipeline, compare usage between ai-search and search, and see which files are retrieved most often.

You can find these metrics within each AutoRAG instance:

• Indexing: Track how files are ingested and see status changes over time.
• Search breakdown: Compare usage between ai-search and search endpoints.
• Top file retrievals: Identify which files are most frequently retrieved in a given period.

Try it today in AutoRAG.