Datadog Release Notes

Agent

Prelude

Released on: 2026-11-09

• Please refer to the 7.85.2 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Configuration streaming between the core Agent and the other Agent processes (trace-agent, process-agent, and others) is now disabled by default. It was on by default in 7.85.0 and 7.85.1. On Kubernetes, that could make those processes ignore DD_* environment variables set only on their own container, such as APM settings on the trace-agent container.
• Fix macOS installer GUI-session detection so the Datadog GUI helper and desktop monitor start for the console user after installation or upgrade when a GUI session exists, including installations run outside a desktop terminal.
• Recover Windows process-manager Agent account lookups after a host rename or image clone when the installed domain is stale but the Agent service is configured to use the same local account. This restores startup of Agent Data Plane and other processes using the Agent account, including their DogStatsD listener.

Upgrade Notes

• The openmetrics.d/adp_compat.yaml.example configuration example is no longer shipped. When ADP is enabled, its DogStatsD, forwarder, and aggregator telemetry is now included in the Core Agent's go_expvar values, so a separate OpenMetrics check against ADP is not needed and would count ADP's values twice alongside go_expvar.

New Features

• Expose DogStatsD, forwarder, and aggregator telemetry from the Agent Data Plane through the Core Agent's go_expvar paths when ADP is enabled. Existing go_expvar users can continue using their current configuration while the compatibility layer combines Core Agent and ADP values on each /debug/vars read, including Core Agent expvars created after startup. The mapped ADP telemetry remains internal rather than being published as separate expvars. Expvars without an equivalent ADP metric remain Core Agent-only. Aggregator coverage includes DogStatsD contexts and samples, processed and flushed events/service checks, per-flush counts, and available flush timings. ADP's NumberOfFlush analogue counts non-empty aggregation flushes. ADP's broader transactions__dropped counter is not mapped to forwarder/Transactions/DroppedOnInput because its meaning is not equivalent to the legacy expvar. Legacy splitter expvars were removed in Agent PR #38834 and are not reintroduced.

Datadog Cluster Agent

Prelude

Released on: 2026-11-09 Pinned to datadog-agent v7.85.2: CHANGELOG.

For teams that own an application running on Kubernetes, updating its observability configuration should be as straightforward as updating any other Kubernetes resource. But configuring integration checks, log collection, and tracing often means working across multiple configuration paths. If those settings live in pod annotations, adding a metric check or changing log processing rules requires a Deployment rollout. If they live in Datadog Agent configuration files, an application team must coordinate the change with the platform team, which may need to restart the Agent.

The DatadogInstrumentation custom resource provides a Kubernetes-native way to manage observability configuration for individual workloads. DatadogInstrumentation brings integration checks, log collection, and Application Performance Monitoring (APM) settings into a single Kubernetes custom resource. Unlike pod annotations, which require an application rollout to change, DatadogInstrumentation applies check and log changes at runtime, without restarting the application or the Agent. Application teams can configure observability by using the same tools and workflows they use to manage their Deployments. Platform teams retain control of the Datadog Agent deployment.

In this post, we’ll show you how to use DatadogInstrumentation to:

• Configure observability for a specific Kubernetes workload
• Define metrics, logs, and traces in one resource
• Change observability configuration independently of application releases

Configure observability for a specific Kubernetes workload

Instead of spreading observability settings across pod annotations and Agent configuration, teams consolidate them in one DatadogInstrumentation manifest for each workload and manage that manifest through version control.

Consider an example team that operates a checkout application and manages it with a Deployment in the apps namespace. To configure the application’s observability, the team uses a DatadogInstrumentation resource alongside its Deployment manifest. When the team needs to change what telemetry data it collects, it edits the resource and reviews the change in a pull request. It then applies the change through its GitOps workflow, just as it does for its Deployment.

Define metrics, logs, and traces in one resource

Moving the checkout service to DatadogInstrumentation replaces two separate configuration sources with one resource. The following example walks through the equivalent DatadogInstrumentation resource, then shows how the team applies and verifies it.

Without DatadogInstrumentation: Settings split across the DatadogAgent resource and annotations

The checkout team’s observability settings live in two places. The platform team deploys the Agent with the Datadog Operator and maintains APM targeting rules in the Operator’s DatadogAgent resource, including a rule that selects the checkout pods and configures their Java SDK and trace settings. The checkout team configures its HTTP check and log collection as JSON in annotations on the Deployment’s pod template.

With this split, a tracing change means asking the platform team to edit its DatadogAgent resource. A change in checks or log configuration requires editing the pod template, which triggers a rollout of the application.

With DatadogInstrumentation: Checks, logs, and APM in one resource

After adopting the DatadogInstrumentation custom resource definition (CRD), the checkout team consolidates these settings into a single Kubernetes resource. In the following example, the spec.targetRef points to the checkout Deployment and the spec.config specifies how the Datadog Agent collects metrics, logs, and traces from that application.

apiVersion: datadoghq.com/v1alpha1
kind: DatadogInstrumentation
metadata:
  name: checkout-instrumentation
  namespace: apps
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout
  config:
    checks:
      - integration: http_check
        containerName: checkout
        instances:
          - name: checkout-health
            url: "http://%%host%%:8080/health"
            collect_response_time: true
    logs:
      - containerName: checkout
        source: java
        service: checkout
        tags:
          - env:production
          - team:checkout
    apm:
      enabled: true
      ddTraceVersions:
        java: "default"
      ddTraceConfigs:
        - name: DD_SERVICE
          value: checkout
        - name: DD_ENV
          value: production

Metrics: The spec.config.checks section defines the HTTP check as structured YAML instead of embedded JSON. The check polls the checkout container’s /health endpoint and records response times, and containerName selects the container to check. The Autodiscovery template variable %%host%% resolves the container’s IP address at runtime, so the configuration doesn’t depend on hardcoded addresses that change as pods are replaced.

Logs: The spec.config.logs section sets the logs’ source and service values for the checkout container and adds tags. See Kubernetes log collection with DatadogInstrumentation for more information about collecting logs.

Traces: The spec.config.apm section enables Single Step Instrumentation (SSI) for the workload. ddTraceVersions selects the Java SDK, and ddTraceConfigs sets the service name and environment—the same settings the platform team set in its DatadogAgent targeting rule, but the checkout team now owns them. The resource’s targetRef selects the workload. SSI adds the specified SDK to eligible application pods when they’re created.

Apply the resource and verify the configuration

DatadogInstrumentation provides configuration feedback through Kubernetes. The Cluster Agent validates the resource at apply time, so in the case of an invalid configuration, kubectl apply returns an error.

After successfully applying the resource, the team checks its ChecksReady, LogsReady, and APMReady status conditions with kubectl describe to diagnose any issues:

kubectl describe datadoginstrumentation checkout-instrumentation -n apps

The status conditions confirm that the Cluster Agent accepted each section of the configuration:

Status:
Conditions:
Message:  1 check(s) configured for Deployment/checkout
Reason:   Configured
Status:   True
Type:     ChecksReady
Message:  1 log config(s) configured for Deployment/checkout
Reason:   Configured
Status:   True
Type:     LogsReady
Message:  APM settings configured for Deployment/checkout; restart the workload to apply them
Reason:   Configured
Status:   True
Type:     APMReady

To verify the data collection, the team checks Datadog to see network.http.can_connect and network.http.response_time metrics, logs from the checkout service, and, once the pods have been replaced, traces from checkout in APM.

Change observability configuration independently of application releases

Suppose the checkout team is investigating intermittent slow requests and needs to change what it collects. It wants to see additional application metrics to understand the cause of any backlogs, mask customer email addresses in its logs, and enable profiling to identify where requests spend time. The team makes all three changes in its existing DatadogInstrumentation resource following its GitOps workflow, without changing the Deployment or the Agent configuration.

Add an OpenMetrics check

To see whether slow requests correlate with a growing backlog, the team adds an OpenMetrics check alongside its existing HTTP check. The new integration: openmetrics entry shown here collects checkout_active_requests and checkout_queue_depth from the application’s /metrics endpoint and reports them as checkout.active_requests and checkout.queue_depth.

config:
  checks:
    - integration: http_check
      containerName: checkout
      instances:
        - name: checkout-health
          url: "http://%%host%%:8080/health"
          collect_response_time: true
    - integration: openmetrics
      containerName: checkout
      instances:
        - openmetrics_endpoint: "http://%%host%%:8080/metrics"
          namespace: checkout
          metrics:
            - checkout_active_requests: active_requests
            - checkout_queue_depth: queue_depth

After the team applies the updated resource, the new check takes effect without requiring an application rollout or Agent restart.

Mask sensitive data in logs

Next, the team adds a processing rule that masks email addresses so logs can be viewed and shared without compromising privacy:

logs:
  - containerName: checkout
    source: java
    service: checkout
    tags:
      - env:production
      - team:checkout
    log_processing_rules:
      - type: mask_sequences
        name: mask_email_addresses
        pattern: '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}'
        replace_placeholder: '[REDACTED_EMAIL]'

The mask_sequences rule replaces any email address in a log message with [REDACTED_EMAIL] before the Agent sends the log to Datadog, so customer email addresses in logs aren’t stored. As with the new check, the updated log configuration takes effect without an application rollout or Agent restart. The team can inspect the resource status for configuration feedback, then check logs in Datadog to verify that data is being redacted.

Enable Continuous Profiler

Finally, the team enables profiling in its APM configuration so they can troubleshoot any backlogs by visualizing where requests spend their time:

apm:
  enabled: true
  ddTraceVersions:
    java: "default"
  ddTraceConfigs:
    - name: DD_SERVICE
      value: checkout
    - name: DD_ENV
      value: production
    - name: DD_PROFILING_ENABLED
      value: "auto"

Adding DD_PROFILING_ENABLED to spec.config.apm.ddTraceConfigs enables Continuous Profiler through the Java tracer, and the auto value profiles only eligible processes, as the SSI documentation recommends. Because the change applies only to new pods, the team triggers a rollout to replace the existing ones. The Continuous Profiler shows that most threads’ wall time is spent waiting for a payment worker to become available, represented by the acquirePaymentWorker span.

Get started with DatadogInstrumentation

With DatadogInstrumentation, application teams change checks and logs on their own schedule, without an application rollout or an Agent restart. Every change in this post’s example happened in the same resource. Changes to tracing and profiling take effect when pods are created, so the team applies them with its next rollout. Platform teams continue to manage the Datadog Agent deployment, while workload owners configure the checks, logs, and traces their services need.

The example in this post, which combines checks, logs, and APM, requires Datadog Agent and Datadog Cluster Agent (both v7.85 or later), with Datadog Operator v1.31 or later or Datadog Helm chart v3.255.0 or later. To get started, install the DatadogInstrumentation CRD with the Datadog Operator or the Datadog Helm chart.

Next, follow the DatadogInstrumentation Autodiscovery setup guide, which lists version requirements and supported workloads. For configuration options, see the documentation for Kubernetes log collection and workload-specific APM instrumentation.

If you don’t already have an account, you can sign up for a free 14-day trial.

Moving your logs to a new platform is only part of a migration. Over the years, your teams have built saved searches, alerts, and dashboards in Splunk’s Search Processing Language (SPL). These assets capture which errors matter, which conditions trigger paging alerts, and which views an on-call engineer opens first. Rebuilding that library by hand is slow and error-prone, typically requiring you to inventory every asset, learn a new query syntax, rewrite each search, and manually confirm that the results match. This work can be complex and time-consuming even for experienced users, and often delays or blocks a migration entirely.

Datadog’s in-app migration tools, Translate and Import, help you recreate your SPL queries, alerts, and dashboards in Datadog without rewriting them from scratch. In this post, we’ll show how you can:

• Translate and validate individual SPL queries
• Import alerts and dashboards in bulk
• Track your migration over time
• Simplify searches with native Datadog features

Translate and validate individual SPL queries

Datadog’s Translate tool converts individual SPL queries into Datadog log searches and lets you validate the results against your own logs. Suppose you run an ecommerce platform and want to confirm that your critical observability workflows and assets will carry over to Datadog. One of the assets is a storefront health scorecard that tracks request counts, error rates, and latency across different endpoints. The underlying query that powers the scorecard combines search filters, conditional evaluations, aggregations, and commands to filter and sort the results:

search service=web-store http.status_code=* NOT http.url_details.path="/_health"
| eval is_5xx=if('http.status_code'>=500,1,0),
 is_4xx=if('http.status_code'>=400 AND 'http.status_code'<500,1,0)
| stats count AS requests,
 sum(is_5xx) AS errors_5xx,
 sum(is_4xx) AS errors_4xx,
 avg(duration) AS avg_ms,
 perc95(duration) AS p95_ms,
 max(duration) AS max_ms,
 dc(host) AS hosts
 BY controller, http.url_details.path
| eval error_rate_pct=round((errors_5xx / requests)*100,2),
 avg_ms=round(avg_ms,1),
 p95_ms=round(p95_ms,1)
| where requests > 100
| sort - error_rate_pct
| head 10

In Datadog Log Management, open Translate and paste the query. Alongside the translation, you get:

• A step-by-step explanation that maps each part of the SPL pipeline to its Datadog equivalent, so you can learn the new syntax as you go
• Live results from your own logs, so you can check that the output matches what you saw in Splunk
• Quick actions, such as opening the search in the Log Explorer or creating a monitor from it

Because the results come from your live data, you can spot problems right away. For example, if a field is named differently in Datadog than it was in Splunk, the results show you before you’ve built anything on top of the query.

Import alerts and dashboards in bulk

While you can use Translate to convert and validate individual SPL queries, Import enables you to translate alerts and dashboards in bulk. Import saves you from having to translate hundreds of assets separately and reduces the risk of overlooking one that matters.

In Log Management, open Import and follow the in-app instructions to export your saved searches, alerts, and dashboards from Splunk. You don’t need to install anything in your Splunk environment. Then drag the export file into Datadog.

Datadog translates each asset into its Datadog equivalent:

• Splunk alerts become Datadog monitors, with the translated query and the original trigger conditions.
• Splunk dashboards become Datadog dashboards, translated widget by widget.

Datadog keeps translated monitors and dashboard definitions as drafts until you’re ready to publish the assets. You can open any translated monitor or dashboard, compare it with the SPL that it came from, and check its results against your data. When you have verified the translation, publish it to make it available to your team. Publishing creates standard Datadog monitors and dashboards, so you can manage them the same way as any others, including through the API and Terraform.

In early access, one design partner translated 62 of 62 queries from a batch in a single bulk run. Another team uploaded more than 1,500 alerts at once with help from Datadog Implementation Services on the first pass.

Track your migration over time

A large migration happens in waves: one team, one Splunk app, or one set of critical alerts at a time. Import keeps every upload in a single inventory, so you can see the migration status of each asset without keeping a separate spreadsheet.

For each import, you can see which assets were translated, which were skipped, and which you’ve already published. You can filter by asset type or status to focus on what still needs work.

When Datadog can’t translate an asset reliably, it marks the asset as skipped instead of publishing a monitor or widget that might return the wrong results. Assets are usually skipped for a few reasons, such as an SPL command that the tools don’t support yet or a dashboard that depends on Splunk-specific features like summary indexes. You can adjust the SPL and import those assets again, finish the query in Translate, or ask Bits AI to help rewrite it.

Simplify searches with native Datadog features

Translating your existing assets gets your teams working in Datadog quickly. A migration is also a good time to simplify searches that grew in size, scope, and complexity. Several SPL patterns have a more direct equivalent in Datadog:

• Search-time field extraction with rex: Use Tap to Parse to turn values buried in a log message into structured fields in one click, without writing a regular expression.
• Lookups and subsearches: Use subqueries and Reference Table joins to filter or enrich logs with related events or external metadata, including in log monitors.
• Scheduled searches that send results via email: Use scheduled CSV reports to deliver Log Explorer results to email or Slack on a recurring schedule.
• Long, multi-stage pipelines: Use Notebooks analysis cells to chain transformations or write SQL when a single log search isn’t enough.

Start migrating your Splunk assets to Datadog

Translate and Import are generally available to Log Management customers in all Datadog sites except US1-FED, at no additional cost. You can check queries against your own logs, translate your full library in bulk, and publish each asset individually or all at once when you’re ready. To get started, see the Query Translation migration documentation. If you’re planning a large migration, your Datadog account team can help.

If you’re new to Datadog, you can sign up for a 14-day free trial to start using Translate and Import.

Improving an AI agent often involves changing its prompts, models, retrieval methods, or tools. But determining whether those changes actually improve performance can be difficult. A revised prompt might produce better answers for a handful of test cases but perform poorly on unexpected user requests. A cheaper model might reduce token costs while introducing tool-calling errors or longer execution times. Before rolling out a change, teams need to understand how it affects agent quality, reliability, and cost under real usage.

Datadog Online Experiments, available in Agent Observability, enables teams to compare agent variants by using live production traffic and measure how a change impacts agent behavior and performance. By bringing agent evaluations together with Datadog Experiments, Online Experiments helps you assess response quality alongside operational metrics such as latency, cost, token usage, and errors. You can also investigate the underlying traces to understand why an agent change caused variations in behavior.

In this post, we’ll walk you through:

• The differences between offline and online experiments
• How Datadog Online Experiments helps you compare agent variants using real production traffic
• How we used Online Experiments internally to evaluate a model change

Offline versus online experiments for AI agents

Offline experiments are an integral part of agent development. They let teams run versions of their agents against predefined datasets, compare outputs, and catch known regressions before deployment, without waiting for them to appear in live traffic. In Datadog Agent Observability, offline experiments support testing for both individual components and complete agent workflows.

But building comprehensive offline evaluation datasets takes time and product knowledge. Teams need to choose representative requests, agree on what good behavior looks like, and decide which checks would reveal a regression. A model-based judge needs a rubric your team understands and a human to check its assessments. The work continues as your product changes: New workflows expose gaps in the dataset, tools return different information, and a judge that captures one failure mode may miss another. Early in development, you may still be discovering the requirements that would let you choose the right cases and criteria.

Online experiments test performance and behavior in agent variants based on real user interactions. They rely on online evaluations, which score agent behavior in production, rather than predefined test cases. This allows you to compare how different versions of your agent behave under production workloads, so you can catch patterns that you may not otherwise have included in your offline evaluation datasets.

For example, consider a support agent that retrieves documents to answer customer questions. Your team might discover that its search tool frequently returns long passages, causing the agent to perform repeated searches before generating an answer. With offline evaluations, you could modify the tool to return shorter excerpts and check whether the agent still cites relevant evidence. But these tests might not capture the full range of requests users make or reveal whether the change helps users resolve their issues.

Online evaluations help you see, for example, whether users bring requests your team did not anticipate or ask follow-up questions, so you can better determine if the change you’ve made helps users resolve their issues. You can then run online experiments that use those scores to compare different versions of your agent. You keep the existing version as a control and test a changed version over the same period, using random assignment to separate the effect of your change from existing differences in usage (e.g., a revised search tool looks better because it receives easier questions). This helps you establish which version of your agent performs best in real production situations and, in turn, enables you to build better offline evaluation datasets as well.

Compare agent variants using real production traffic with Datadog Online Experiments

Running online experiments requires teams to connect each agent variant they are testing to the appropriate measurements and analyze the differences in performance. Datadog Online Experiments brings Agent Observability data together with Datadog Experiments to help you manage this process. You can use evaluators you’ve created in Agent Observability and connect their scores to experiment metrics, alongside the operational measures you choose to collect. With assignment and measurement configured, Experiments calculates the relative changes and confidence intervals.

You also choose how much exposure a change should receive. Once the variants are wired into your application, Datadog Feature Flags can manage the traffic split and let you adjust exposure without deploying the application again. You can begin with a small share of users while retaining a control group for the comparison.

The measures should reflect the decision you’re making. A revised tool might improve evidence quality while adding extra searches. A cheaper model might require more attempts to complete a task. Comparing evaluation scores with duration, cost, and errors helps you see those trade-offs.

From your experiments, you can also view traces in Agent Observability to inspect the individual executions: which tool the agent called, what arguments it supplied, and how it responded to the result. This provides context that helps you dig deeper when an experiment leaves you with questions. For example, a promising cost result may justify pursuing a change, while uncertainty about its other effects warrants a closer look. Next, we’ll show you how we used Online Experiments internally at Datadog to assess this exact type of change in one of our own agents.

How we used Online Experiments to evaluate a model change

We used Online Experiments to compare OpenAI’s GPT-5.6 Luna and GPT-6 Luna in our Insights agent, which investigates agent behavior by using trace evidence. We wanted to test whether Luna 6’s lower model prices translated into cheaper investigations and understand what the change meant for investigation quality and tool behavior. We split traffic evenly between the models, keeping Luna 5.6 as the reference while evaluating Luna 6 under real usage.

Our primary measure was investigation quality. An LLM judge scored investigations from 1 to 5 using the request, tool inputs and outputs, and the agent’s conclusions. The rubric covered accuracy, completeness, and how well conclusions followed from the available evidence, including whether the agent acknowledged uncertainty or missing data. We collected duration, estimated cost, estimated token usage, and workflow errors from the agent’s traces as separate measures.

Online Experiments brought those scores and operational measures into the same comparison in Agent Observability. We could assess cost alongside the work the agent performed to complete an investigation.

Luna 6’s estimated cost was 44.3% lower. The confidence interval—a range of differences consistent with the data—ran from 37.5% to 51.2% lower. This result supported our hypothesis that the model change could reduce investigation cost. Quality scores were close at 3.84 for Luna 5.6 and 3.86 for Luna 6, although close averages alone do not establish equivalent quality.

Duration and workflow errors left more uncertainty. The best estimates were 12.8% higher duration and 13.6% more workflow errors. But the duration interval ranged from 4% lower to 30% higher, and the error interval from 16% lower to 43% higher—effectively, a range between little-to-no difference and a meaningful increase. We inspected the traces to understand the risk before deciding whether to proceed.

In the traces we reviewed, Luna 6 supplied invalid tool parameters more often than Luna 5.6. It usually corrected those errors after retries. Our quality judge considered tool use, but its overall score did not tell us how often calls failed or how much work recovery required. We were thus able to use traces to expose a specific behavior in the agent’s handling of tool calls that could be improved in the Luna 6 version of the agent. We addressed the tool parameter issue that our investigation uncovered before proceeding with the Luna 6 version that helped us lower costs.

The experiment also revealed next steps that could help us improve our agent in the future. The failed calls we found are useful candidates for offline regression tests. A follow-up online experiment could test a change to tool definitions or error feedback while continuing to measure quality, duration, and cost.

Improve AI agents faster with Online Experiments in Agent Observability

Datadog Online Experiments helps teams evaluate AI agent changes under real production conditions, compare quality and operational performance, and investigate unexpected behavior by using traces. Whether you’re testing a different model, refining retrieval, or modifying a tool, controlled experiments provide evidence to help you determine which changes improve your agent and which require further work.

The findings from each experiment can also help improve offline evaluation coverage. As your team discovers new failure modes and user workflows, you can turn those observations into regression tests and use subsequent experiments to measure whether your fixes work under real usage.

Online Experiments is free for Agent Observability customers submitting up to 40,000 LLM spans per month. To learn more about getting started with Agent Observability, see the product page and documentation.

New to Datadog? Start a 14-day free trial.

Startup? Eligible companies get a year of free Datadog credits. Apply here.

Agent

Warning

Kubernetes installations: This Agent version may drop configuration settings in the trace-agent, process-agent, security-agent, and system-probe containers. As a result, these containers may crash, stop reporting data, or ignore configured settings

Recommended action: Upgrade to Agent 7.85.2.

Prelude

Released on: 2026-11-04

• Please refer to the 7.85.1 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• CSPM: On hosts in FIPS mode, oscap-io aborted before its first rule with "out of core in secure memory" wherever it could not lock memory, as in system-probe, whose seccomp profile denies mlock. It now evaluates the XCCDF rules there.
• CSPM: The Agent removes the temporary files of each oscap-io process once it exits. An oscap-io process that crashed or was killed left an oscap.* directory in /tmp, which on FIPS hosts filled /tmp.
• Fix a crash on startup of the 32-bit ARM IoT Agent when logs collection is enabled (unaligned 64-bit atomic operation). See #59066.

Datadog Cluster Agent

Prelude

Released on: 2026-11-04 Pinned to datadog-agent v7.85.1: CHANGELOG.

Teams that build agents on open models hosted on Fireworks AI, such as GPT-OSS, Kimi, GLM, Qwen, and MiniMax, need a reliable way to measure whether their LLM applications and agents are giving accurate, safe, and helpful answers efficiently. Datadog Agent Observability now supports Fireworks AI as a model provider, so you can get end-to-end visibility into your agents built on Fireworks AI, from traces and token usage to quality evaluations. You can bring your own Fireworks AI API key and use models hosted on Fireworks AI to power LLM-as-a-judge evaluations and to experiment in the Agent Observability Playground.

In this post, we’ll show you how Datadog’s Fireworks AI integration lets you run LLM-as-a-judge evaluations with Fireworks AI models and experiment with those models in the Agent Observability Playground. You can test prompts against Fireworks AI models in the Playground, then review the results alongside the rest of your Agent Observability data.

Run LLM-as-a-judge evaluations with Fireworks AI models

After you connect your Fireworks AI account once, every model it can access is available anywhere Agent Observability lets you choose a model provider. LLM-as-a-judge evaluators use a model to score the inputs and outputs of your LLM application, so you can catch issues such as hallucinations, off-topic answers, or negative user sentiment at scale.

With Fireworks AI models as your judge, you can match the model to the job: a larger model such as gpt-oss-120b for nuanced quality checks, a low-latency router endpoint for high-volume scoring, or your own fine-tuned model for domain-specific criteria. Because evaluations run on your Fireworks AI account, model usage follows the billing and governance controls you’ve set up, and you choose the trade-off between cost and quality for each evaluator. You also don’t need to build a separate pipeline to export traces, call a model, and write scores back.

Datadog then runs the evaluator against your production traces using the selected Fireworks AI model, scoring your LLM traces for qualities like faithfulness, topic relevance, or sentiment alongside the rest of your observability data. Results appear next to each span, and you can track them over time, filter traces by score, and alert on regressions.

Experiment with Fireworks AI models in the Playground

Datadog’s Agent Observability Playground lets you iterate on prompts before they reach production. For example, if you run a support assistant for an online travel agency, you may want to know whether an open model can answer refund and cancellation questions accurately and whether it’s fast enough for a live chat. In the Playground, you can run your system prompt and a real customer question against gpt-oss-120b and immediately see the response alongside its token count and latency.

From there, you can swap in other Fireworks AI models, including low-latency router endpoints like Nexus Auto, to weigh answer quality against speed and cost. You can also open the full trace for any run to see exactly what the model received and returned. When a prompt looks promising, run it against a dataset of real customer questions to help ensure it holds up beyond a single example. Once you have a model and prompt you trust, select “Get Code” to copy the configuration into your application. The same Fireworks AI models can power the evaluators that monitor it in production.

Get started with Fireworks AI and Datadog

The Fireworks AI integration is available today for all Datadog Agent Observability customers. To connect your account and learn more about evaluations and Playground, see the Fireworks AI integration documentation.

If you’re new to Datadog, you can get started with a 14-day free trial. New to Fireworks AI? Sign up for an account to get an API key.

Prompts are an important part of LLM application behavior, but teams often manage them alongside application code or across separate notebooks and configuration files. This can make prompt changes dependent on application releases and make it difficult to understand which version is running in each environment. When a change affects application quality, latency, or cost, teams also need a way to connect that behavior back to the prompt version that produced it.

Prompt Management in Datadog Agent Observability provides a centralized registry where you can create, version, deploy, and retrieve prompts independently of your application release cycle. Prompt Management works with Prompt Tracking to connect managed prompt versions with the LLM calls that use them, helping you understand how prompt changes relate to application behavior.

In this post, we’ll explain how you can:

• Manage prompt versions in one place
• Deploy prompt changes independently of application releases
• Control how prompt versions reach your applications
• Connect prompt changes to application performance

Manage prompt versions in one place

As teams iterate on LLM applications, prompt development can become difficult to coordinate. Different versions may live in application code, configuration files, or development environments, making it harder to identify the prompt that is running in production or compare it with an earlier version.

Prompt Management gives teams a centralized registry for prompts used by their LLM applications. Each managed prompt has a stable Prompt ID and a version history. You can update the prompt while applications continue to reference the same identifier, and you can inspect earlier versions to understand how the prompt has changed.

The Prompts page brings managed prompts and prompts captured through Prompt Tracking into the same catalog. If Agent Observability already tracks a prompt in your application, you can register it for management instead of recreating it separately. You can also create prompts directly in Datadog and define variables for content that your application supplies at runtime.

As you edit a managed prompt, Datadog preserves previous versions instead of overwriting them. Version history and comparisons help teams review prompt changes and return to earlier versions when necessary.

Deploy prompt changes independently of application releases

Prompt Management separates prompt deployment from application deployment. Instead of embedding each prompt revision in application code, your application retrieves a managed prompt at runtime by using its stable Prompt ID. Datadog resolves that request to the appropriate deployed version.

This separation gives teams more flexibility to iterate on prompt behavior, as they can publish a new prompt version and deploy it to an environment without shipping a new application build. If a prompt change produces unwanted results, you can deploy an earlier version without rolling back unrelated application changes.

Applications can also retain a local prompt as a fallback for cases when managed prompt retrieval fails. This lets teams introduce managed prompts while maintaining application-defined fallback behavior.

Prompt Management supports runtime retrieval in Python. For prerequisites and implementation details, including supported ddtrace versions, authentication, and fallback behavior, see the Prompt Management documentation.

Control how prompt versions reach your applications

Different stages of prompt development often need different versions of the same prompt. A team might want one version in development, another in staging, and a stable version in production without maintaining separate prompt identifiers for each environment.

Prompt Management lets you deploy versions of the same Prompt ID to different environments. Applications can retrieve the version associated with their environment, while developers can also request an exact version when they need to reproduce or validate specific behavior.

For more granular control, Prompt Management builds on Datadog Feature Flags to support targeting rules. Within an environment, you can keep one prompt version as the default while serving another version to requests that match attributes you define. This gives teams a way to expose prompt changes selectively without creating separate prompts or changing application deployments.

These controls provide a consistent path for moving prompt changes through development and production. Teams can keep a stable Prompt ID while deciding which version different environments or targeted requests receive.

Connect prompt changes to application performance

Managing prompt versions addresses one part of the prompt development life cycle. Teams also need to understand what happens after a version reaches their application.

When Agent Observability is enabled, Prompt Management works with Prompt Tracking to associate managed prompt versions with LLM spans. Datadog can then connect an individual prompt with information about its usage and performance.

From the Prompts page, you can compare versions by call volume, latency, token usage, and cost. You can also inspect version history, review differences between prompts, and jump to traces that used a particular version. This context helps teams investigate whether changes in application behavior coincide with a prompt update.

For example, if latency or token usage changes after a prompt update, you can identify the version associated with affected LLM calls and inspect the corresponding traces. Prompt Tracking also lets you filter Trace Explorer by prompt name, ID, or version, providing another way to investigate requests associated with a change.

Prompt retrieval and Prompt Tracking remain separate capabilities. Applications can retrieve managed prompts without enabling Agent Observability, while Agent Observability provides the LLM spans and prompt metadata needed to analyze prompt usage and performance. See the Prompt Tracking documentation for instrumentation and correlation details.

Manage prompt changes with Datadog Agent Observability

Prompt Management gives LLM application teams a centralized, versioned registry for changing prompts independently of application releases. Teams can deploy different versions across environments, control how versions reach application traffic, and return to earlier versions without changing the Prompt ID used by the application.

Combined with Prompt Tracking, Prompt Management also connects those versions to LLM calls and performance data, helping teams investigate how prompt changes relate to application behavior.

Datadog Agent Observability is free for Datadog customers submitting up to 40,000 LLM spans per month. To learn more about getting started with Agent Observability, see the product page and documentation.

If you don’t have a Datadog account, sign up for a 14-day free trial to manage and monitor prompts for your LLM applications.

When a mobile application takes too long to launch or becomes unresponsive, it can be difficult to find the cause. Datadog Real User Monitoring (RUM) helps developers detect these issues and understand which users are affected. But knowing that an application is slow doesn’t tell you which part of its code is responsible. Engineers often need to reproduce the problem locally, piecing together performance metrics and code execution information from separate tools. This can be particularly difficult when performance varies across devices, operating system versions, and real-world conditions. On top of that, it can be hard to share findings from local profiles with teammates.

Datadog RUM Profiling is an add-on to RUM that connects code-level CPU profiling data with performance measurements from real user sessions. RUM Profiling, which supports browser applications through Browser Profiler, now supports iOS and Android applications as well, enabling developers to see which methods consume CPU time during performance-critical operations on real users’ devices.

In this post, we’ll show how RUM Profiling helps you:

• Identify code-level bottlenecks in iOS and Android applications
• Collect code-level performance data from production mobile applications
• Troubleshoot frontend performance across browser and mobile applications

Identify code-level bottlenecks in iOS and Android applications

RUM Profiling attaches CPU profiles to the RUM events where mobile performance problems surface, from slow application launches to moments when an app stops responding, as well as to the workflows that developers define as critical RUM operations. Developers can use RUM Profiling to investigate the code behind a single slow or unresponsive event, or find the bottlenecks that have the biggest impact on performance.

Slow application launches can create friction before users have a chance to interact with an app. With RUM, developers can track startup performance through industry-standard measurements such as Time to Initial Display (TTID) and Time to Full Display (TTFD). These measurements help teams determine the severity of startup delays, but they don’t independently explain what could be causing those delays.

RUM Profiling helps explain why by adding code-level visibility through the association of CPU profiles with application startup events in iOS and Android applications. When developers identify a slow TTID measurement, they can examine the corresponding CPU profiles to see which functions executed during the affected startup period, without reproducing the same conditions on a local device.

For example, suppose an engineering team has configured a Datadog monitor to track TTID for cold application launches. After deploying a new version, the team is alerted about a significant increase in startup times. RUM helps them identify the affected application version and sessions, and RUM Profiling provides the associated CPU profiles. Developers can inspect the method call stacks in flame graphs and thread timelines to investigate which functions consumed CPU time during startup and identify potential sources of the regression. They can do all of this in RUM, without switching to a separate debugging tool, and can download profiles for further analysis as needed.

Track critical workflows with RUM operations

Application startup is one important use case, but developers also need to investigate performance during business-critical interactions. A checkout flow, for example, might occasionally stall for certain users even when overall application performance appears healthy. Critical RUM operations let developers explicitly mark the key moments to be profiled in their app, such as authentication, adding an item to a cart, or completing checkout. Enabling profiling for an operation tells Datadog that this moment matters. Datadog then collects CPU profiling data during the activity and attaches it to the operation’s RUM event, even if no out-of-the-box RUM event is captured during this time. This gives developers code-level context for important workflows, not just the events Datadog profiles automatically.

Understand the scope of performance issues with aggregated profiling views

A single CPU profile can help developers investigate an individual performance issue, but it doesn’t necessarily reveal the scope of that issue. A function that consumes excessive CPU time during one application launch, for example, might also contribute to slow startup performance across many sessions. Identifying these recurring patterns helps engineers prioritize improvements that can benefit more users.

RUM Profiling provides aggregated profiling views that help developers identify methods that repeatedly contribute to CPU consumption across multiple real user sessions. This helps them distinguish isolated performance issues from recurring bottlenecks that may warrant broader optimization work.

For mobile applications, aggregated profiles cover supported profiling scenarios, including application startup, TTID, TTFD, and critical RUM operations. For example, if multiple users experience slow application launches, aggregated profiling data can help identify methods that repeatedly consume CPU time during startup. Developers can then examine individual profiles for additional context and determine which code paths to investigate.

Developers can also compare aggregated profiling data across groups of sessions to investigate performance regressions. For example, they can compare slow application launches (at or above p90) with typical launches (at or below p75) to identify methods that consume more CPU time in slower sessions, or compare profiling data across application versions to investigate changes in code execution following a deployment.

After optimizing a CPU-intensive method, developers can use out of the box metrics provided through RUM without Limits, including TTID, TTFD, and critical RUM operations, to evaluate whether application performance has improved for real users.

Collect code-level performance data from production mobile applications

Traditional profiling tools can introduce CPU and battery overhead, potentially affecting the same performance they are intended to measure. Because of this, developers often profile in controlled testing environments. But it can be difficult or impossible to reproduce problems that occur only on particular devices or under real-world conditions in these environments.

RUM Profiling circumvents this problem by profiling only a sample of application launches and sessions, in production, on real user devices. It lets developers control profiling collection on iOS and Android through sampling rates they can adjust using RUM Remote Configuration. If application startup times or Android ANR rates increase, for example, a team can raise the profiling sample rate to collect more profiles, then lower it once it has gathered enough data.

The underlying collection mechanisms differ by platform. On Android, Datadog uses the operating system’s ProfilingManager API, subject to application and device-level rate limits. RUM Profiling also captures CPU profiles for ANR errors. On Android 16 or higher, these can include system traces showing what the main thread was waiting for, such as a lock held by another thread, a slow disk read, or a call to another process. On iOS, RUM Profiling associates CPU profiles with application launches, app hangs, and long tasks, so developers can examine the code executing when an application freezes or stops responding.

Troubleshoot frontend performance across browser and mobile applications

Many engineering teams maintain both web and native mobile applications, but the performance issues affecting these environments can differ considerably. In browser applications, long-running JavaScript tasks can block the main thread and delay user interactions. Native iOS and Android applications execute work across multiple threads, so developers may need to investigate activity on different threads to identify the code contributing to a slowdown. Although each platform uses different profiling mechanisms and supports different collection scenarios, Datadog packages and presents profiling data the same way for every platform. All profiles are attached to RUM events, can be queried alongside the rest of your RUM data, and roll up into the same aggregated profiling views.

For browser applications, RUM Profiling collects JavaScript CPU profiling data and correlates it with RUM events, including views, actions, long tasks, and performance vitals. Developers can use the Profiling Explorer to investigate CPU-intensive functions, identify recurring bottlenecks, and compare profiling data across application versions. You can learn more in our post on Browser Profiler.

With support for iOS and Android, engineers responsible for an ecommerce application, for example, can use RUM Profiling to investigate JavaScript functions contributing to slow checkout interactions in the browser while also examining CPU profiles associated with slow launches, Android ANRs, iOS app hangs, or RUM operations in their mobile applications.

Find root causes faster with Bits AI and the Datadog MCP Server

Developers don’t need to be experts in reading profiling data to act on it. From any profiled RUM event or the aggregated profiling page, they can ask Bits AI for the root cause of a slowdown, such as why the app froze or what caused a regression in the latest release, without manually reviewing call stacks or flame graphs. Developers can also use the Datadog MCP Server’s profiling tools to query for problematic sessions, such as those with slow app launches, frequent ANRs, or app hangs, and analyze the associated CPU profiles.

Investigate frontend performance down to the code with RUM Profiling

RUM helps developers identify the performance problems users experience, and RUM Profiling provides the code-level context to explain why they occur. With support for browser, iOS, and Android applications, engineering teams can use RUM Profiling to investigate code-level bottlenecks across frontend experiences and use profiling data from real users’ devices to guide optimization efforts.

RUM Profiling is available as an add-on to Datadog Real User Monitoring. To get started, see the RUM Profiling documentation for Android, iOS, and the browser. To learn more about investigating JavaScript performance, read our post on Browser Profiler.

If you’re new to Datadog, sign up for a 14-day free trial to start investigating real user performance with Datadog RUM.

When we launched Bits Investigation in December 2025, our goal was to give teams an always-on agent that could investigate every alert and reduce mean time to resolve (MTTR). Since then, we’ve focused on making Bits faster, more cost effective, and even more accurate.

At launch, the average investigation cost $25 and took 11 minutes. As of October 2026, Bits Investigation has:

• An 84% lower average cost per investigation, down to about $4.
• An 82% shorter average investigation time, down to about 2 minutes.
• A 44% higher investigation quality score on our internal benchmark, compared to our March 2026 release.

This post explores how Bits Investigation was built, the latest improvements to its investigations, and how these changes help teams resolve production issues faster.

Where we started: Building Bits

The first generation of Bits Investigation used a coordinated system of agents to investigate production issues. When presented with an incident, Bits agents first gathered relevant context, then ran an initial investigation using targeted telemetry queries. Based on those findings, the agents iteratively generated and tested root-cause hypotheses. The agents then expanded promising hypotheses into more specific hypotheses until they arrived at a probable root cause.

This approach established the foundation for autonomous investigations grounded in evidence from customers’ environments. However, the orchestration layer for the initial Bits Investigation was complex, as it required dedicated phases and tools for context retrieval, hypothesis generation, and testing.

As models and agent architectures evolved, we later introduced a new generation of the Bits agent built around a simpler ReAct architecture. In this framework, each agent reasons about the evidence, chooses a tool, and uses the results to decide what to investigate next. Newer models and better tools helped the agents adapt as they uncovered information, while expanded access to Datadog data sources gave them more context.

Altogether, this flexible structure reduced the average investigation cost to about $6.50.

Where we are now: Faster, more efficient investigations

The latest updates to Bits Investigation combine newer models, improved model routing, and better tools. With code execution through the Datadog MCP Server, Bits can write and run code to query multiple data sources, correlate findings, and summarize the evidence in a single tool call. Processing data before returning it to the model reduces the context Bits needs to handle and the back-and-forth required to investigate an issue. A simpler agent architecture also makes it easier to use the frontier model best suited to each task.

Together, these changes help Bits gather evidence efficiently and test hypotheses for deeper root cause analysis. For on-call engineers, that means less time gathering evidence and more time acting on findings.

Bits Investigation now costs about $4 per investigation and completes its analyses in about 2 minutes on average, helping engineers reach a diagnosis sooner.

Alongside these efficiency gains, Bits achieves a 44% higher investigation quality score than our March release, according to our internal investigation benchmarks. To assess the quality of Bits’s findings, we continuously compare them with the known root causes of past incidents. Together, faster results, lower costs, and improved investigation quality make Bits a more effective first responder for production alerts.

Where we’re going: Specialization through post-training

Looking ahead, we’re building on this progress with Scout, Datadog’s model designed for site reliability engineering (SRE) investigations. We’ve trained Scout on a wide range of failures using an in-house reinforcement learning system. Customized training improves investigation quality and reduces inference costs, allowing Scout to match leading models on investigation tasks at a lower inference cost. We plan to extend this targeted approach to autonomous remediation, helping teams move from identifying root causes to restoring service faster.

Experience an enhanced Bits

Flexible agentic architecture, rigorous evaluation, specialized training, and more diverse knowledge sources have led to a more effective Bits Investigation. As we explore the future of Bits, our in-house Scout model has shown that customized training can yield better performance and lower costs.

To start using Bits Investigation in your environment, explore the documentation.

If you’re not yet a Datadog customer, sign up for a free trial today.

AI agents are creating a virtually unbounded attack surface. To be useful, they need access to internal systems, developer tools, and sensitive business data. They also consume a wide range of external content through various tools, APIs, and MCPs, where malicious instructions can be hidden. Incidents like EchoLeak show how indirect prompt injection attacks can result in large reputational damage and sensitive data leaks. Adding guardrails at runtime that are directly instrumented inside your decisioning flow is critical to both observing and securing agents.

Traditional security controls weren’t built for agents. They can decide whether an agent is permitted to take an action, but not whether it should. Since agents reason their way to future steps rather than following a fixed path, it’s difficult to proactively test all of its possible actions. For example, an agent asked to triage GitHub issues might read an issue with cleverly hidden instructions in a tool call that directs it to collect deployment credentials and send them to an external endpoint. Although each action was allowed by the agent’s permissions, the agent still behaves in an unintended way.

In order to effectively contain indirect injection vulnerabilities like this one, agentic security controls need to operate at runtime, evaluating each action the agent takes in context and judging intent rather than checking static permissions. Otherwise, you’re forced to mitigate security risks by restricting what agents can access and act on entirely, which makes agents a lot less capable of delivering the critical business outcomes your organization requires.

We built AI Guard to sit in the agent loop, giving security engineers and developers the ability to block malicious activity and investigate alongside agent traces, logs, and infrastructure telemetry already in Datadog. In this post, we’ll explore why runtime AI security is crucial for operating agents safely, as well as how we’re developing AI Guard to empower security and development teams to proactively detect and neutralize emergent threats.

To make the right decisions, your guardrails need the complete picture

AI Guard runs inline with your agents. This gives its detection and evaluation engine a view of the full runtime context at each decision point in the agent loop: which service called the agent, what it depends on, which stored data it touched, and what the agent has already read and done so far. That position is what produces the full picture needed to effectively evaluate threats.

AI Guard starts with discovery, before any action is even evaluated. It discovers all the agents running in your cloud environment and shows which model endpoints, datastores, and connected services each one reaches, and which of those are exposed, giving your team a map of where agent risk actually lives.

As your agents run, AI Guard collects multiple layers of context:

• Session context: The full conversation history, system instructions, agent tasks, every tool call and the step about to execute, are all captured (with enterprise-level data retention controls) to stop indirect and multi-step attacks, when no single message alone looks malicious.
• Your custom defined context: Policies, tool blocking rules, and business context you set are all used to tune the guardrail’s memory to your specific agent’s use cases.
• Application context: The service, its dependencies, datastores, owner, and what the agent actually touched, are carried on traces you already run, which only exist at the application layer.

Evaluating intent rather than checking permissions requires a full view of the agent loop, and that means sitting inside the application where it runs. AI Guard turns full-context evaluations into detections you can trust and an alert queue that doesn’t just add noise.

Block attacks inline, in the agent loop

AI Guard runs directly alongside agents in real-time, with hooks at multiple steps in the agent loop, so it can block or sanitize any prompt, model output, tool call, or tool result before it executes. With sensitive data scanning enabled, PII and secrets are masked in real time as they pass through. AI Guard gives your teams the ability to insert guardrail evaluations at all of the key decision points in the agent loop.

Benign intermediate steps are allowed to proceed, as long as they align with your organization’s intent for that agent. If intent diverges, AI Guard stops the chain of behavior immediately. For instance, let’s consider the GitHub triage agent attack described in the introduction. If an attacker is able to seed a hidden instruction into a tool result, they can perform an indirect prompt injection attack using a seemingly benign prompt. AI Guard however, can block the step following the malicious tool call inline, before the agent acts on it. In this way, AI Guard catches multi-step attacks without stalling the agent loop.

With guardrails sitting at key decision points, security teams can ensure unsafe behaviors are blocked even when permissions are followed.

Investigate and remediate risk in one place

Blocking threats is only half the job. The next step is closing the loop between investigating a threat and remediating it. When an agent attempts an unsafe action, your team needs to understand what led to it and whether earlier actions affected other systems. In a SecurityHQ and Datadog survey of over 200 cybersecurity leaders, 39% of respondents said their organization could only partially reconstruct every action an AI agent performed during the previous 30 days. Investigating that sequence requires a record of what the agent read, which tools it called, and what those calls returned.

AI Guard surfaces the highest-priority security findings as signals so your team knows exactly where to start. Signals reconstruct the entire session that led to a blocked action, surface a summary of what happened, how it happened, and correlate all the agent interactions together for deeper investigation and response.

Diagnosing the cause of a compromised agent is the first step, but determining the full scope of an attack is equally important. A successful prompt injection can lead to compromised hosts, containers running code they shouldn’t, and queries against a database the agent wasn’t intended to access. Since AI Guard’s signals land on the same traces as your application and agent performance data, the Datadog security platform allows you to observe and secure the full stack your agents run on.

Finally, your system needs to provide ways to act that secure your environment from further compromise. AI Guard enables you to block offending users or IPs, open security cases, or trigger remediation workflows directly from a triggered signal. You can also push a case to Jira or your SIEM and handle it in your security teams’ existing systems.

Implement guardrails where engineers already work

Agents are owned by the engineers who build them, so controls that live outside the development workflow tend to stay unadopted. Because AI Guard lives in application teams’ tracing and monitoring systems, it gives engineers access to security enforcement where they already work. An engineer can turn on inline evaluation for their own agent without filing a ticket or standing up a proxy, and a security team can set the policy those evaluations enforce without knowing each agent’s internals.

AI Guard provides trace context for every evaluation, so your engineers aren’t left piecing together compromised data in logs after the request is already completed. In that same survey, 40% of respondents said that an AI agent making a decision without anyone knowing was their greatest concern. Visibility into what an agent reads and does gives your team a way to examine decisions that might otherwise go unnoticed. In this way, AI Guard bridges the gap between security engineering and development.

Engineers are already working within Datadog, making AI Guard an extension of their existing workflow, not a net new tool. On services already running the Datadog tracer, this enforcement can be turned on with one click. You can also hook AI Guard into your own AI gateway for a single integration point that covers every agent that routes through it.

Scale AI agents without scaling risk

As agents rapidly iterate and expand in scope, models become more intelligent and more privileged, and attackers swiftly develop new strategies, agentic security systems need to keep pace, which is why we’re building AI Guard to self-harden and learn from your environment’s history. AI Guard’s detection is tuned to your agents and your policies, analyzes each action against behavioral norms, and redacts PII and secrets in each agent interaction.

The goal of this system is to create a closed loop where runtime security controls proactively detect and neutralize active threats across your agent loops, service dependencies, and infrastructure. Because AI Guard uses Datadog’s wealth of observability and security technology and context, AI Guard allows you to unify runtime security controls with application, infrastructure, and agent performance controls. This way, you can protect your agents, your infrastructure, and the customers that used them with a single platform.

AI Guard is now generally available. For more information about getting started, see the documentation. Or if you’re brand new to Datadog, sign up for a free trial.

Teams building on Claude are shipping AI agents and applications faster than many organizations can establish standard practices for monitoring them. As these agents move into production, teams need visibility into their costs, reliability, and output quality.

Datadog is available on Anthropic’s Claude Marketplace, giving joint customers a direct path to bring Datadog’s monitoring capabilities into their AI stack alongside Claude.

What’s available

At launch, Claude Marketplace offers Datadog Agent Observability. With Agent Observability, you can monitor, trace, and evaluate the agents and AI applications your teams build on Claude, so failures and quality regressions surface before they reach end users.

Built on an existing relationship

Joint customers already use Datadog to monitor their Claude usage by tracing Claude API requests, bringing Anthropic cost data into Cloud Cost Management, sending Claude Code telemetry, and more. The marketplace listing is the next step: extending that relationship from “Datadog monitors your Claude usage” to “you can buy Datadog directly through Claude.”

How it works

Claude and Datadog are built to work well together: Claude powers the agents your teams build, and Datadog gives you the visibility to run them confidently and reliably. The marketplace listing doesn’t change how either product works, and it doesn’t require any new technical integration. It just makes that combination easier to put to use.

Here’s how to purchase Datadog Agent Observability through Claude Marketplace:

• Fill out the sales form that is available on the Datadog tile within Claude Marketplace, or talk to your Anthropic account team about adding Agent Observability.
• Apply a portion of your committed Anthropic spend toward your purchase. No new budget line is required.

Your contract and product relationship stay directly with Datadog; the marketplace just simplifies discovery and invoicing. If you already buy Datadog directly, that relationship and process are unaffected by Datadog’s availability on the marketplace.

Who it’s for

Datadog’s listing on Claude Marketplace is available to customers with US-based entities today, with plans to expand availability over time.

Get started with Datadog

If you’re already a Claude customer, talk to your Anthropic account team about adding Agent Observability to your plan. If you’re a Datadog customer and have questions about Claude Marketplace, reach out to your Datadog account team.

If you’re new to Datadog, you can sign up for a 14-day free trial.

When a native C or C++ application crashes on a user’s device, you often have limited visibility into what caused the failure. A stack trace can identify the affected code, but it doesn’t reveal what the user was doing before the crash or whether other users encountered the same issue. Without that context, you may have to reconstruct the events leading up to the crash or reproduce the issue locally, in an environment that may not match the user’s device.

The Datadog C++ SDK now extends Datadog Real User Monitoring (RUM) to native C and C++ applications running on Windows, macOS, and Linux. The SDK collects crash diagnostics, tracks errors, and captures instrumented user activity, so you can troubleshoot failures in the context of the user sessions where they happened.

In this post, we’ll cover how to:

• Catch and troubleshoot crashes in native C and C++ applications
• Track user activity with RUM session context
• Integrate RUM into native C and C++ applications

Catch and troubleshoot crashes in native C and C++ applications

The SDK supports native crash reporting and Error Tracking to help you diagnose unhandled crashes. On Linux and macOS, it captures signals such as SIGSEGV (segmentation faults) and SIGABRT (aborts), while on Windows, it captures unhandled structured exceptions. The resulting crash reports include stack traces to help you identify where a failure occurred and what caused it.

Say a desktop application crashes when a user opens a saved document. With the SDK, you can examine the captured crash report and its stack trace. When you upload your debug symbols, symbolication translates stack trace addresses into human-readable function names and source locations, which can help you pinpoint the affected code.

Choose a crash handler for your application

The SDK offers two crash handlers:

• An in-process handler that captures crashes within the application
• An out-of-process handler that collects more comprehensive crash diagnostics

The in-process crash handler requires only C++17 and no additional executable. It captures the crashing thread’s stack trace and uploads the crash report the next time the application launches.

The out-of-process crash handler uses Crashpad, the open source crash reporting system that Chromium uses. Because Crashpad runs as a separate crashpad_handler process, it can capture failures such as stack overflows and heap corruption that the in-process handler may miss. It also captures stack traces from all threads and uploads crash reports immediately, without waiting for the application to relaunch.

Both crash handlers report RUM errors enriched with the latest RUM context at the time of the crash, including the session and last active view. You can use this context to investigate crashes alongside the rest of your session data in Error Tracking. The RUM session timeline also lets you review the user actions, view loads, and network requests that preceded the crash, helping you reconstruct what happened before the application failed.

Identify recurring crashes and track handled errors

Error Tracking groups related failures into issues so you can identify recurring crashes, distinguish isolated failures from widespread ones, and prioritize investigations by user and session impact.

The SDK also lets you report handled errors that don’t terminate the application. For example, a desktop application might fail to deserialize a saved file, or a native client might encounter an unexpected response while processing data. Linking these errors to RUM sessions lets you investigate failures that affect users even when the application keeps running.

Track user activity with RUM session context

Because the SDK isn’t tied to a particular UI framework, you define the events that reflect how users interact with your application, and the SDK associates them with RUM sessions. For example, you can instrument a desktop application to capture:

• Views: Application screens or states, such as a document editor, settings page, or file browser
• Actions: User interactions, such as selecting a file, clicking a button, or starting an export
• Resources: Network requests and their outcomes, including timeouts and other failures
• Operations: Multistep workflows, such as opening a document, authenticating a user, or uploading a file
• Long tasks: Stalls that block the main thread for an extended period, such as when a per-frame update or an event handler exceeds a budgeted duration

If a user encounters a failure while opening a document, you can see the view they were on, the action that started the workflow, and any related resource failures. RUM operations can also track whether the document opens successfully, and how long it takes to load its contents and prepare for editing.

Integrate RUM into native C and C++ applications

The SDK provides a C++ API for codebases that target C++17 or later, plus a C99-compliant C API. Both work with native build environments, including CMake-based projects. You can also wrap the C API in foreign function interface (FFI) bindings to instrument applications written in other languages and runtimes.

The SDK sends RUM data directly to Datadog without requiring the Datadog Agent on each user’s device. For example, you can instrument a PC game to capture views, user actions, errors, and operations such as loading a level or connecting to a game server. You can then investigate failures in RUM without a plugin specific to your game engine.

The SDK also includes tracking-consent controls that let you manage local storage and uploads of RUM data based on each user’s consent preferences.

Start monitoring native applications with Datadog RUM

To start monitoring your native C or C++ application, instrument the workflows most relevant to your users and configure crash reporting to collect diagnostic information when failures occur.

For setup details, see the C++ SDK setup documentation and the open source C++ SDK on GitHub.

If you’re not already a Datadog customer, sign up for a free 14-day trial to start monitoring native C and C++ applications with Datadog RUM.

Datadog Real User Monitoring (RUM) and Product Analytics help engineering and data teams understand application availability and performance, as well as user behavior from user session data. Teams can ingest the full scope of this data into Datadog for monitoring and analysis. However, they might also need the event data in infrastructure they control to support custom analytics and reporting, machine learning (ML) workflows, or long-term data access.

Export Pipelines provide a native way to continuously send RUM and Product Analytics data from Datadog to your cloud storage. You can then load the files into data warehouses like Snowflake, BigQuery, or Databricks, or process the output in your custom pipelines.

In this post, we’ll show how you can:

• Continuously send user session data from Datadog to your cloud storage
• Use exported data for analytics, ML models, and compliance-driven retention

Continuously send user session data from Datadog to your cloud storage

Export Pipelines move RUM and Product Analytics data into your own cloud infrastructure. You configure the pipeline from the Datadog UI, select the performance or behavioral data that you want to export, and choose a destination in Amazon S3, Azure Blob Storage, or Google Cloud Storage. After you configure the pipeline, Datadog continuously sends matching events to that destination without requiring your team to maintain custom export scripts or scheduled jobs.

Export Pipelines support Parquet and JSON file formats so that you can choose a format based on how downstream systems will process the data. Parquet loads directly into Snowflake, BigQuery, and Databricks without preprocessing.

JSON suits teams that have their own processing pipelines or need a more flexible representation of each event.

Datadog organizes files by date and hour in the storage location, so downstream processes can target files from the relevant time period instead of scanning every exported file. You can enable, disable, or update an export configuration as your requirements change. The Export Pipelines page displays each pipeline’s current state—Active, Error, Pending, or Paused—so you can see whether your pipelines are running as expected.

Use exported data for analytics, ML models, and compliance-driven retention

Sending RUM and Product Analytics events to your cloud storage makes that same technical and behavioral dataset available to workflows beyond Datadog. Teams can load the files into a data warehouse for analysis, feed them into ML pipelines, or retain them long-term to help meet compliance requirements.

For example, an analytics team might want to understand how application behavior relates to business outcomes that are already modeled in its warehouse. The team can export Product Analytics events to a Google Cloud Storage bucket and load the Parquet files into BigQuery. There, analysts can join application actions and user behavior with datasets such as subscriptions, account attributes, or financial data without introducing a second set of application instrumentation.

Data science teams can also send RUM and Product Analytics events to cloud storage for ML workflows. They can then use their own data transformation and modeling pipelines to train models for personalization, churn prediction, recommendations, or anomaly detection.

Export Pipelines can also help organizations meet compliance requirements that call for long-term access to event data in infrastructure they control. Datadog RUM provides standard 30-day retention and offers 6-month storage with RUM Managed Archive, and Product Analytics provides 15-month retention. Organizations that need access to event data beyond these periods can export the relevant events to cloud storage and apply their own storage, access, and life-cycle policies.

Get started with Export Pipelines

Export Pipelines give you a direct path to move RUM and Product Analytics data from Datadog into infrastructure you control without writing or maintaining custom export code. Whether you’re feeding a data warehouse, training ML models, or addressing long-term compliance retention, your data stays available in the format and destination that works best for your workflows. To learn more, read the Export Pipelines documentation.

If you’re new to Datadog, you can sign up for a 14-day free trial to start exporting RUM and Product Analytics data to your cloud storage.

Teams can use Datadog Real User Monitoring (RUM) to get direct visibility into what users experience in their frontend applications. With RUM without Limits™, you get metrics and service level objectives (SLOs) that reflect all of your traffic, while retention filters keep only the most relevant user sessions.

However, sessions that are not captured by retention filters and are dropped today may become important tomorrow for a customer support ticket, an audit, or an investigation. Teams operating in regulated environments or handling complex support issues are therefore left choosing between retaining all sessions up front, which is usually cost-prohibitive, and retaining selectively, which risks that a critical session won’t be there when they need it.

RUM Managed Archive adds a fallback to the RUM without Limits model. Instead of choosing between cost and coverage, you can store all ingested sessions and recover specific ones on demand.

In this post, we’ll explore how you can use RUM Managed Archive to:

• Store all sessions for later access
• Recover sessions on demand
• Investigate support issues by recovering archived sessions

Store all sessions for later access

RUM Managed Archive stores all ingested sessions for extended periods, including those not retained by your filters, as a less expensive alternative to indexing them. Instead of discarding unretained sessions, Datadog stores them in Datadog-managed storage so that you can access them later if needed.

You can configure a storage period of 1, 3, or 6 months at the application level, letting you choose different archive strategies for applications with different investigation, risk, or audit requirements.

The Managed Archive UI provides a dedicated view where you can explore stored sessions. These archived sessions are not fully indexed, but they include a lightweight set of metadata—including timestamp, session ID, user ID, and account ID—that allows you to search for and identify relevant sessions. This makes it possible to locate specific sessions without incurring the cost of indexing everything up front.

With Managed Archive, you can keep your retention filters focused on high-value data while still preserving access to the full set of sessions for future use.

Recover sessions on demand

RUM Managed Archive lets you selectively recover stored sessions when an investigation requires their full context. You can search Managed Archive by attributes and quickly identify an individual session or a set of sessions associated with a particular user, incident, or time period.

After you recover a stored session, Datadog fully indexes it and makes it available in RUM for a 30-day analysis window. A recovered session has the same investigation capabilities as a regularly indexed session retained by a retention filter, including access to its events and attributes.

This approach to storage and recovery lets you retrieve relevant sessions that weren’t captured by retention filters or that have fallen outside the retention period. For example, a support engineer can recover an individual session associated with a customer ticket, while an SRE can recover sessions from a relevant time window during a retrospective incident investigation.

Teams can also use this workflow when an audit or compliance investigation requires access to sessions beyond their standard retention period. RUM Managed Archive helps preserve access to that historical session data, while recovery lets teams fully investigate only the sessions that are relevant to the request.

Investigate a support issue that surfaces weeks later

Consider a financial services team that receives a support ticket from a high-value customer weeks after an issue occurred. Last month, the customer submitted a trade through the mobile app. The confirmation screen froze, and they weren’t sure whether the transaction went through.

Because this user’s session didn’t match any retention filters, it wasn’t retained. Without RUM Managed Archive, the team would have no way to access it. The team would then have limited visibility into what the user experienced, prolonging the investigation and delaying resolution.

With Managed Archive, the session is still available, and the team can locate it by searching the archive for attributes such as user ID or account ID. After locating the relevant session, the team can recover it for full investigation in RUM. In this case, the team discovers that:

• The user was on a mobile app version affected by a UI update.
• The UI update delayed the price refresh, and the displayed price briefly showed a stale value before updating.
• The backend executed the trade at the correct real-time price, despite the display error.

With this insight, the team compares behavior across versions, confirms that a later release fixed the issue, and provides the customer with a clear explanation of what occurred. This type of investigation is possible only because the session was preserved, even though it wasn’t originally retained.

Recover the sessions you need with RUM Managed Archive

RUM Managed Archive gives teams that use RUM without Limits a safety net for sessions they need later, whether those sessions weren’t retained or have aged past their standard retention period. With this feature, teams can keep their regular retention strategy focused on day-to-day monitoring while maintaining access to sessions needed for customer support, debugging, incident investigations, and audits.

To learn more, check out the documentation on RUM Managed Archive. If you’re new to Datadog, sign up for a 14-day free trial.

Improving an AI agent means knowing which behaviors to change and where those changes will have the greatest impact. Insights in Datadog Agent Observability improves AI agents by analyzing agent traces to identify recurring cost and reliability problems. It then explains the causes of those problems, assesses their impact, and recommends fixes backed by evidence.

At Datadog, we use Insights to improve the agent behind Insights. That is, Insights shows us why the agent spends more tokens than necessary or fails to complete a task. We then use that evidence to decide what to change. In this post, we’ll give examples of how we use Insights in Agent Observability to:

• Identify opportunities to reduce agent costs
• Investigate failures in the agent’s tool calls
• Prepare a code change

Identify opportunities to reduce agent costs

Recently, Agent Observability helped us identify two patterns in our agent’s traces that pointed to opportunities to reduce costs.

Tools can return more content than the agent needs

Our agent uses search_llmobs_spans to find evidence in recorded traces. When it searches too broadly, the tool can return more content than the model needs. If those results enter the conversation unchanged, the agent sends that extra content to the model again in later requests.

One insight showed that five search calls across two agent runs returned roughly 80,000 tokens of content that was then included in model requests. One result contributed 19,000 tokens to a subsequent request with 97,000 input tokens. (These counts describe the requests we inspected, not the savings we achieved.)

Here, Datadog recommends narrowing filters, requesting smaller result pages, and retrieving more content only when needed. In this example, when the behavior recurred after the original finding was resolved, a later insight recommended compact previews. The agent could then retrieve full span content when it needed more detail.

Previews must retain the identifiers needed to retrieve relevant evidence. To validate a fix, we replay the affected investigations and compare how much content the tools return, how many tokens the model uses, and how well the agent answers. Using fewer tokens helps only if the agent still investigates correctly and completely.

Changing context can prevent the agent from reusing cached input

Prompt caching lets a provider reuse work from processing instructions and tool definitions sent in earlier requests. This can reduce the cost of processing the same content again. To benefit, requests need to begin with matching content, called a shared prefix. (To learn more, see our blog post on monitoring prompt caching.)

One production insight revealed that information specific to each task appeared before the tool definitions. If that information changed, the provider couldn’t reuse work when processing the otherwise unchanged content that followed. Instead, similar requests repeatedly wrote nearly the entire prompt to the cache.

The chart shows how often requests reused cached input for less than 20% of their input tokens. It includes requests with more than 1,024 input tokens for the selected model and provider, helping us identify periods to investigate.

In this example, Datadog estimated that making the repeated instructions and tool definitions reusable could save about $2,734 per month. That projection applies $91.14 in estimated daily savings over 30 days to the requests analyzed. (Note that we haven’t measured those savings. The actual amount saved depends on how often requests reuse cached input, whether they qualify for caching, and how much the provider charges.)

Datadog recommended that we change how our agent builds requests by placing stable system instructions and tool definitions before task-specific details and conversation content. Keeping the beginning of requests consistent could let the provider reuse work it had already done.

Investigate failures in the agent’s tool calls

Insights in Agent Observability also identifies why our agent’s tool calls fail. In one case, the agent repeatedly sent a request with missing information. In another, it requested resources by using names that its tools couldn’t recognize.

The agent can repeat calls with missing arguments

One insight revealed that the agent made five identical invalid calls to update_insight over approximately 18 minutes. The agent supplied an identifier, application, and status, but left out the complete insight object required to replace the existing one.

The tool rejected each call because required information was missing. None reached the backend, and the run ended in error after four retries. Repeating the same request couldn’t resolve the problem. The agent needed to supply the missing information before trying again.

The agent can request resources that aren’t available

Our agent’s instructions require it to use the exact resource names returned by its skill loader. An insight revealed that the agent guessed or changed those names across 193 sessions in the sample. The tool couldn’t find the requested resources, so the calls failed and delayed the agent’s investigations.

Datadog recommended that we check each requested resource name against the skill loader’s list before attempting to retrieve it. If the name doesn’t match, the tool should return the valid choices so that the agent can correct its request. This change would make the tool enforce a rule already present in the agent’s instructions. To check whether it works, we would replay the affected turns and verify that the agent requests listed resources without repeating failed calls.

Prepare a code change

Once we understand a problem, we can use the evidence and recommendations in an insight to help a coding agent address it. The “Fix with Bits” option lets us pass that information to a coding agent, which can change the code and test the fix. For setup instructions, see the Insights coding-agent workflow.

We used the “Fix with Bits” option to address the recurring problem of searches returning too much content. It passed the finding, the recommended change, and the reference to the trace into a coding session. The coding agent then prepared a pull request (PR) to make searches return compact previews.

The proposed change preserved the identifiers needed to retrieve full span content and allowed the agent to request additional pages of results. The PR also included tests to check that previews worked for both newly retrieved and cached results.

You can follow the same workflow with your own coding agent. Connect it through the Datadog MCP Server or Pup CLI, then ask it to review Insights for your application. Using the evidence and recommendations, the agent can investigate your code, implement and test a fix, and prepare a PR for your review.

Improve your agents faster with Datadog Agent Observability

Insights in Datadog Agent Observability helps our team identify what to change in the agent’s code and instructions. It shows how strategically revising the agent requests helps reuse cached input and prepare a PR to generate smaller search previews. After measuring evidence and estimating impact, we determine how to prioritize these potential changes and how to test them.

With Datadog Agent Observability, you can investigate why your agent costs more than expected or fails to complete tasks. It lets you review the evidence and recommendations in an insight, then implement and test a fix. You can evaluate subsequent runs to check whether the fix worked and see if the problem returns.

Insights in Datadog Agent Observability is now generally available and offers a free tier with up to 40,000 LLM spans per month. Read the Insights documentation to apply this workflow to your own agents.

If you’re new to Datadog, sign up for a 14-day free trial. And for startups, eligible teams can get a year of free Datadog credits. Apply to Datadog for Startups here.