Datadog Release Notes

Collecting high-quality user feedback on agents, like from thumbs-up or thumbs-down buttons, is an important part of agent development. User feedback is needed for everything from basic gut checks on whether your agents are behaving well to planning and creating robust eval sets. It’s a critical part of Datadog’s Agent Observability, which provides explicit end-user feedback features for collecting and analyzing it. But while these features can be wired up to UI elements like thumbs-up buttons, you can’t force your users to actually click on them. In practice they rarely do, as we noticed while working on Bits Chat.

From a data science point of view, thumbs up, thumbs down, and similar user feedback are just another type of label. This led us to wonder whether we could derive good-enough user feedback from existing Agent Observability traces and other Datadog telemetry by using a technique called weak labeling. We used this technique to create a public session classification skill, which reads traces and other telemetry data to approximate the feedback you’d get from a manual button.

In this post, we’ll explore the weak labeling technique and show you how we tested and validated a classification skill that approximates user feedback from Datadog telemetry.

How weak labeling works

Weak labeling is commonly used in traditional ML projects to generate labels when collecting real ground truth is too expensive or too difficult, as it often is when trying to get comprehensive user feedback. It’s best for generating large amounts of good-enough training data.

The basic idea behind weak labeling is to start with the data you have and use it to derive the label you want via heuristics, trained models, or any automatable means. For many problems, it’s possible to combine several data sources to get a proxy for what you want. For example, a user explicitly clicking thumbs up on a social media post is the ultimate signal of whether they liked it. But combining data on how long they looked at the post along with data on whether they posted a positive comment can get you fairly close.

These derived labels are rarely as accurate as ground truth, so they need to be compared to a smaller golden dataset to understand how close they are and what they can be used for. In our case, the signal we’re after is whether a user had a good interaction with an agent. Did the agent answer their questions, or did the user walk away unhappy?

Choosing our three telemetry types

Well-instrumented applications already collect quite a bit of telemetry data that can help answer this question: Agent Observability collects detailed agent traces; Real User Monitoring (RUM) gives you visibility into what your users are actually doing, where they click, and how long they hover; and Audit Trail surfaces changes made across the platform. Our plan was to generate weak labels approximating a thumbs-up or thumbs-down response from these three telemetry types, then check them against a golden dataset of hand-labeled sessions.

Agent Observability traces contain details about an entire chat session, including transcripts from which we can derive user sentiment. RUM captures how the user interacted with the chat session: Did they bounce right away, or did they accept the agent’s suggestions? Finally, Audit Trail provides details around whether a given Datadog artifact, such as a dashboard or metric, changed. Many Bits Chat conversations involve exactly these changes, so we suspected this would be a useful signal. Together, these three sources give us the arc of a session: what the agent said, what the user did about it, and whether anything in Datadog changed as a result.

What our session classification skill does

You can find our session classification skill in our Datadog Labs repo if you’d like to try it yourself. It’s designed to produce useful results from as little data as possible, and quality improves as you add more. You can run it just on Agent Observability traces, but it improves with RUM and Audit Trail data.

The skill accepts three kinds of modes: an entire application (in which case it samples traces), or an individual trace or session, which it labels each directly. That makes it easy to label a sampled set of traces, where each label stands in for a thumbs up or down from an end user. That’s a useful signal when you’re troubleshooting an agent’s behavior over the last day in Agent Observability. The skill can also be chained into longer pipelines to label individual samples.

How we validated the skill

We wanted to answer two questions: whether we could extract a useful signal about user satisfaction from Datadog telemetry, and whether that signal would improve as we added more telemetry types. So, we built a simple ablation stack in Agent Observability Experiments that starts with traces alone, then traces and RUM, and finally traces, RUM, and Audit Trail. We then ran each version against a golden dataset of hundreds of Bits Chat sessions carrying hand-applied thumbs-up and thumbs-down labels. We cared most about concurrence with the binary thumbs-up and thumbs-down labels from Bits Chat, and because we had a reasonably balanced dataset, we chose accuracy as the primary optimization metric. The notebook below outlines our basic experiment setup and implementation:

A true positive here means that our label matched the hand label, while a false positive means it didn’t. We then measured accuracy across the three different versions of the classifier.

Experiment results

We expected that Agent Observability traces would provide the strongest single signal since they contain the agent conversation itself. We got decent results with just Agent Observability traces, which reached 78% accuracy compared to our ground truth dataset. By adding RUM and then Audit Trail on top of that, we reached 80% and 82% accuracy, respectively. The notebook below shows these results:

While 82% isn’t perfect, it’s a useful first pass to identify sets of traces that are worth inspecting manually. This cuts down the search space to 18% of your overall trace population, and it’s especially useful as a backstop if you lack a true customer-generated thumbs up or thumbs down.

We also validated the approach. The more data sources we added, the more our accuracy improved on the internal validation dataset. Achieving 100% accuracy was not expected, as that generally means you’re overfitting the dataset rather than succeeding. Still, our results suggest we can continue to improve accuracy with additional telemetry types, such as APM and log data.

Collect better feedback from your own agents

This approach works across a wide range of agents, so we’ve published the skill as part of our Datadog Labs agent skills repo. Agent Observability customers and general users can install the skill today from our repo. You can also use the techniques in it as a starting point for building your own skill.

To learn more about monitoring your agent’s performance, costs, traces, token usage, and errors, check out our Agent Observability documentation.

If you’re new to Datadog, get started with a free trial.

Keep multisignal investigation logic in code

Observability investigations rarely follow a straight line. A latency question might cause an AI agent to start with a metric, pivot into traces, compare a deployment window, and finish by reducing thousands of logs to a few patterns. Each individual query is easy, but propagating context throughout an entire investigation can be tricky and expensive.
With conventional MCP tools, each step becomes another exchange with the model: choose a tool, inspect its response, decide what to call next, and pull the new result into the conversation. That process works well for a focused lookup, but it can be inefficient in a multisignal investigation. The model ends up spending context on tool schemas, raw responses, and the intermediate steps between calls rather than focusing on outcomes.
Datadog Code Execution, generally available, gives AI agents a programmable way to investigate observability data through the Datadog MCP Server. From a sandboxed JavaScript environment, an agent can query several Datadog APIs, run independent work in parallel, branch on results, join data, and return only the evidence needed for the answer. By returning a more focused set of evidence to the model, Code Execution can improve answer accuracy while reducing the cost of running AI agents.
In this post, we’ll show how you can:

• Keep multisignal investigation logic in code
• Get more accurate answers while spending less on tokens and tool calls
• Run generated code without handing it your credentials
• Set up Code Execution

Keep multisignal investigation logic in code

The Datadog MCP Server gives AI agents access to tools for querying logs, metrics, traces, monitors, dashboards, and other Datadog data. Traditional MCP tools provide the agent’s underlying model with clear, bounded actions and remain the shortest path for a focused question. For an investigation that crosses several data sources, however, an agent might need to call multiple tools and pass each result back through the model before deciding what to do next. Code Execution moves that intermediate work into code.
Code Execution combines a small MCP interface with a programmable execution environment. The agent interacts with Code Execution through two MCP tools, execute_code and search_datadog_sdk, to explore and query Datadog across its entire API surface. Within the execution environment, both control flow and the intermediate data remain in code instead of passing through the conversation one tool call at a time.
Inside the sandbox, the agent can run independent queries together, use one result to shape the next query, normalize responses from different APIs, and join them on a shared field. It can also filter or aggregate large responses before returning any results to the conversation. The available API operations are based on the Datadog TypeScript client SDK, so generated code uses the same clients and request shapes as other Datadog integrations. Each API operation stays individually typed and subject to its required permissions. The agent composes them by using ordinary control flow.
For example, an agent can generate and run the following script to query logs and spans for errors over the same 1-hour window. The code runs the two queries in parallel, groups the results by service, joins them, and returns only the services that appear in both result sets:

[Code example omitted for brevity]

Each API in this example can return its top 25 services, but only the five services that appear in both result sets cross back into the model’s context. Without Code Execution, the model would have to receive both result sets and perform that join in the conversation.

Get more accurate answers while spending less

To measure how Code Execution affects investigation quality and cost, we compared it with Datadog’s Core toolset. The comparison covered 25 observability tasks across metrics, logs, traces, Datadog Error Tracking, and investigations that crossed more than one data source. We ran each task three times with GPT-5.6 Terra, GPT-5.6 Sol, Claude Sonnet 5, and Claude Opus 4.8, and then we scored the final answers for correctness.
Code Execution improved answer correctness with every model we tested, with gains ranging from 7.7 percentage points (pp) to 21.8 pp:

MODEL | CORE TOOLSET | CODE EXECUTION TOOLSET | CHANGE
GPT-5.6 Terra | 77.6% | 85.3% | +7.7 pp
GPT-5.6 Sol | 74.4% | 94.0% | +19.6 pp
Claude Sonnet 5 | 66.7% | 88.5% | +21.8 pp
Claude Opus 4.8 | 77.5% | 90.6% | +13.1 pp

Because model costs depend on token usage, reducing the amount of context sent to a model can lower the cost of running an investigation. The averaged results across the four models showed that Code Execution used 73.2% fewer input tokens and 39.6% fewer tool calls:

CATEGORY | CORE TOOLSET | CODE EXECUTION TOOLSET | CHANGE
Answer correctness | 74.1% | 89.6% | +15.6 pp
Input tokens | 159.4k | 42.8k | -73.2%
Tool calls | 4.08 | 2.47 | -39.6%

Note: Values in the Core toolset and Code Execution toolset columns are rounded. Values in the Change column are calculated from the unrounded values.

Run generated code without handing it your credentials

Letting a model generate code against production observability data requires a clear security boundary. Code Execution keeps execution and authentication on opposite sides of that boundary.
Generated JavaScript code runs in an isolated sandbox without access to the caller’s credentials. When the code calls a dd.* method, the trusted MCP service makes the request on the caller’s behalf, enforces their existing Datadog permissions and Code Execution policies, sanitizes the response, and returns it to the sandbox. The code can work with the resulting data, but it never handles the credentials that are used to retrieve it.

Set up Code Execution

To get started with Code Execution, connect the Datadog MCP Server to your AI client and enable the code-exec toolset.
When Code Execution is enabled, ask your agent a question that requires it to correlate multiple kinds of observability data. For example: “Find the services whose error rate changed after last night’s deployments, then show me the trace patterns that changed with them.” The agent can use Code Execution to gather the relevant data, correlate it, and return the evidence behind its answer.

Start investigating across Datadog with less model context

Code Execution helps AI agents use the Datadog MCP Server to carry out multistep observability investigations while keeping intermediate logic and data inside a sandbox. By reducing the amount of intermediate context and the number of tool calls that pass through the model, Code Execution can lower the cost of running AI agents while helping them produce more accurate answers. To learn more, see the Code Execution documentation, the toolset configuration guide, and the MCP Server documentation.
If you’re new to Datadog, you can sign up for a 14-day free trial to try Code Execution for your investigations.

A funnel can tell you that 40% of users dropped off between checkout and payment. What it can’t tell you is what those users did instead, such as return to an earlier form field, leave the flow for a support page, encounter an error, or take another route entirely. Because actions and views between funnel steps don’t affect the conversion calculation, two very different experiences can produce the same funnel result.

Journey Paths in Datadog Product Analytics helps you understand the behavior behind those numbers. Journey Paths visualizes the sequences of pages and actions that users take through a defined flow, ranks them by frequency, and lets you compare paths for users who converted with paths for users who dropped off. Patterns that once required reviewing individual sessions one at a time become quantified behavioral data.

In this post, we’ll show you how to:

• Understand how users convert, not just how many
• Find the friction behind drop-offs
• Investigate a path with Conversion Analysis, Session Replay, and RUM

Understand how users convert, not just how many

A successful conversion doesn’t mean that it was efficient. In a multistep checkout, one user might move directly from shipping to billing to payment, while another repeatedly moves between form steps before eventually reaching the same payment event. Both users count as conversions, but their experiences are substantially different.

Converted paths rank the sequences that led users to convert, surfacing differences the conversion rate hides. A repeated back-and-forth pattern can point to confusing form requirements, unclear navigation, or another part of the experience that deserves investigation even though users eventually complete the workflow.

A recurring detour or repeated action might not reduce conversion yet, but it can still increase the time and effort that users spend completing a task. Quantifying that behavior gives product and engineering teams a measurable starting point for deciding where to investigate.

Find the friction behind drop-offs

Dropped-off paths show what users did when they didn’t complete the flow. Instead of stopping at a drop-off percentage, you see the sequences of events associated with that outcome, ranked by frequency.

Two patterns are worth naming: dead ends, where a path stops before users reach the next expected step, and detours, where users leave the intended flow and don’t come back. Because paths are ordered by frequency, you can determine whether one of these behaviors represents a recurring pattern rather than an isolated session.

For example, suppose a checkout flow includes a support link to help users who have questions. Funnel analysis shows a large drop-off before payment, while Journey Paths reveals that many of those users clicked the support link and never returned. The link was intended to help users, but the quantified path gives your team a concrete hypothesis to investigate: The support experience may be pulling users away from the task they were trying to complete. Rather than starting with a few replays and trying to infer whether they represent a broader pattern, you can start with the ranked paths and identify behavior that occurs repeatedly across users.

Investigate a path with Conversion Analysis, Session Replay, and RUM

Once you know which sequences correlate with conversion or drop-off, you can investigate which users are associated with those outcomes. The Conversion Analysis panel in Product Analytics funnels uses statistical analysis to rank user attributes and behavioral segments that correlate with conversion or drop-off at a funnel step.

Where a path tells you which sequences users took, Conversion Analysis helps you determine which users are associated with an outcome. For example, if that support-link detour is concentrated among mobile users in one region, that’s a different problem than one affecting everyone. A path that appears primarily for a particular population can point toward a more specific issue than a pattern that occurs broadly across users. You can use that context to narrow the population that you examine before moving into individual sessions.

From there, you can pivot to a Session Replay of a real user who took a path. Because you already know the pattern and how common it is, Session Replay becomes the last step rather than the first: You’re confirming what a known behavior looked like on screen instead of hunting for one at random.

Product Analytics also works alongside Real User Monitoring (RUM) so that you can investigate technical conditions around user behavior. For example, if users repeatedly leave a flow at the same point, RUM data can help you examine whether errors, latency, rage clicks, dead clicks, or other experience signals coincide with the behavior. Product Analytics and RUM share the same Browser and Mobile SDKs and configuration, so teams already using RUM don’t need to instrument anything new.

Turn funnel conversions and drop-offs into measurable user behavior

Journey Paths turns a funnel conversion or drop-off from a percentage into a measurable pattern of user behavior. By comparing converted and dropped-off paths, you can find inefficient conversion routes, identify recurring dead ends and detours, and use Conversion Analysis and Session Replay to investigate the users and experiences behind those patterns. And because Journey Paths is its own chart type, you can save it, add it to a dashboard, and keep important journey patterns in front of your team.

Read our Journey Paths documentation to learn how to get started. For more on the surrounding analysis workflow, see the Product Analytics funnel documentation and Product Analytics charts documentation.

If you don’t have a Datadog account, sign up for a free 14-day trial to investigate user journeys with Product Analytics.

Infrastructure-as-code (IaC) security scanning can catch common misconfigurations before deployment, but every organization also has internal requirements that a default rule catalog cannot cover. For example, teams may need to enforce required tags, approved instance types, or naming conventions.

With custom rules for Datadog IaC Security, security and platform teams can define these requirements as Rego policies and run them alongside Datadog’s default rules during IaC scans.

In this post, we’ll show how you can use custom rules for Datadog IaC Security to:

• Enforce your organization’s IaC standards
• Write custom policies with natural language
• Validate rules before publishing
• Find custom policy violations in existing workflows

Enforce your organization’s IaC standards

IaC Security detects misconfigurations (such as missing encryption or overly permissive access) before infrastructure is deployed. Datadog continuously scans configured repositories and then links any findings about misconfigurations to the relevant repository, branch, and file path. IaC Security’s default rule catalog provides checks for common security risks, but those checks cannot account for every policy that an organization develops for its own infrastructure.

Custom rules extend default coverage with requirements that are specific to your organization. For example, you might require teams to apply a standard set of tags to Terraform resources, restrict workloads to approved instance types, or enforce internal network boundaries. You can also encode checks that support company-specific compliance requirements, rather than relying on engineers to verify these policies manually during code review.

Custom rules use Rego, the policy language from Open Policy Agent (OPA), and run alongside Datadog’s default rules during IaC scans. Custom rules support Ansible, AWS CloudFormation, Dockerfile, Kubernetes, Terraform, and GitHub Actions. After publication, a custom rule runs in subsequent scans where its specified platform applies. You can use IaC Security configuration to further control which rules run and where they apply.

To get started, navigate to the IaC Rules page and select “Create Rule.” Creating, editing, or publishing a custom rule requires the appsec_vm_write permission. As you build a custom rule, you provide a name and select its platform, category, and severity. You can optionally specify a provider and add a Common Weakness Enumeration (CWE) identifier.

Write custom policies with natural language

Rego gives teams a flexible way to express infrastructure policies, but writing a policy from scratch normally requires knowledge of Rego. With custom rules, you can just describe the requirement in natural language, such as an internal policy for how a particular infrastructure resource should be configured. The AI rule creator can use that description to generate the Rego policy, along with a sample IaC configuration that triggers the rule. You can then review, edit, and test both directly in the editor. You can use Bits Chat to help create a policy.

When you’re creating a rule from scratch, the editor provides a starter policy and sample file. You can also clone a default or custom rule, which is useful when your requirement applies to the same platform and resource type as an existing check. Cloning copies the rule’s metadata, policy, sample file, and description so that you can then modify the new rule to reflect your organization’s requirement.

Validate rules before publishing

A custom policy needs to properly identify the configuration you intend to flag. To help ensure that a rule works correctly before you save it, Datadog lets you evaluate a policy in the rule editor before the rule runs against your repositories.

For example, suppose you’re creating a Terraform policy that flags an aws_s3_bucket_versioning resource when its status is explicitly set to Suspended. Start by adding a sample Terraform file containing that configuration and run the policy. The editor should return a finding for the affected status attribute. Then change the value to Enabled and run the policy again to verify that it produces no findings. If the rule needs more work, select “Save as draft” to prevent it from running during scans. When the rule is ready, select “Save and publish” to make it available for subsequent IaC scans.

Datadog also maintains a version history as custom policies change. Editing a rule creates a new version. You can review the rule’s version history, compare any two version, or restore an earlier version. Version history gives teams a record of how an organization’s infrastructure policies have changed over time and provides a path to roll back an unwanted change.

Find custom policy violations in existing workflows

Once you publish a custom rule, its findings are available to the same workflows that incorporate Datadog’s default IaC findings. Developers can review violations directly in pull request comments, the IDE extension, and the IaC Security findings explorer. Teams can also use PR Gates to block pull requests that violate custom policies. Findings Automation Pipelines in Datadog Security can trigger automated actions based on those findings. These options let organizations act on their internal IaC standards without introducing a separate workflow.

Custom rules also use the existing IaC Security configuration model. You configure repository-wide rule settings either in Datadog or in a code-security.datadog.yaml file, including run or ignore rules, severity filters, path filters, and per-rule configuration. Inline comments support local exclusions when an exception applies to a particular line, block, or file. See the IaC Security configuration documentation for supported configuration options.

Enforce your infrastructure policies before deployment

Custom IaC Security rules help teams detect organization-specific infrastructure policy violations in the same scanning workflow they use for Datadog’s default rules. By turning internal requirements into testable Rego policies, security and platform teams can reduce reliance on manual review while giving developers feedback before infrastructure changes reach production.

To create your first rule, read the IaC Custom Rules documentation. For details about supported Rego syntax, parsed IaC inputs, and platform-specific patterns, see the IaC Custom Rule Reference. You can also review the broader IaC Security documentation and our guide to improving the security of IaC deployments.

If you don’t already have a Datadog account, sign up for a 14-day free trial to start scanning your IaC configurations with Datadog.

Agent

Prelude

Released on: 2026-09-16

• Please refer to the 7.83.2 tag on integrations-core for the list of changes on the Core Checks

Enhancement Notes

• gpu: Constant metrics (gpu.device.total , gpu.memory.limit , gpu.core.limit , and gpu.memory.bar1.total ) are reported on the cadence set by the new gpu.static_metrics_reporting_interval (15 seconds by default) to ensure accuracy when using weighted sums.
• Single Step Instrumentation's tracer config mechanisms (the ddTraceConfigs , remote-config policies, and the admission.datadoghq.com/apm-inject.tracer-configs pod annotation) now also accept OTEL_ prefixed environment variable names, in addition to the existing DD_ prefix. This allows configuring a tracer's native OpenTelemetry mode (e.g. OTEL_TRACES_EXPORTER , OTEL_EXPORTER_OTLP_ENDPOINT ) through SSI.

Bug Fixes

• Fixed the Agent Data Plane pre-flight sending its metrics and API key to datadoghq.com instead of the configured site. The generated pre-flight configuration was built from the fully resolved Agent configuration, so dd_url 's default value appeared in it as though it had been set explicitly, and an explicit dd_url takes precedence over site . The pre-flight configuration is now built from the settings the operator actually supplied, matching what a normally-supervised Agent Data Plane reads from datadog.yaml .
• Fix DD_NETWORK_PATH_COLLECTOR_FILTERS so JSON-encoded Network Path collector filters are parsed and applied correctly.

Datadog Cluster Agent

Prelude

Released on: 2026-09-16 Pinned to datadog-agent v7.83.2: CHANGELOG.

Map and optimize your data before it reaches Microsoft Sentinel

Microsoft Sentinel is Microsoft’s cloud-native SIEM for detecting, investigating, and responding to threats across your environment. To query security data and run analytics rules, Sentinel expects telemetry data in specific table schemas. But firewall, VPN, and network detection logs arrive in vendor-specific formats, making source-specific mappings difficult to maintain as environments grow. Teams that collect logs in Microsoft Azure Event Hubs may also want to filter what they send to Sentinel while retaining full-fidelity raw logs in Azure Storage.

Datadog Observability Pipelines addresses these challenges with Microsoft Sentinel Packs. Each Pack contains preconfigured mappings that convert logs from a particular source to a Sentinel table schema in your pipeline, before they reach Sentinel. With Microsoft Sentinel Packs, logs can arrive with the expected structure already in place for analytics rules, workbooks, and investigations.

In this post, we’ll explore how Microsoft Sentinel Packs help you:

• Map and optimize your data before it reaches Microsoft Sentinel
• Investigate security activity across every source
• Control Microsoft Sentinel ingest costs without losing visibility

Microsoft Sentinel table schemas give security teams a consistent set of fields to use in queries and analytics rules. Observability Pipelines can populate those fields within your pipeline through Microsoft Sentinel Packs. The initial release includes Packs for:

• Palo Alto Networks (network and perimeter): Maps 10PAN-OS log types (traffic, threat, system, config, GlobalProtect, decryption, correlation, HIP Match, User-ID, and IPTag) to CommonSecurityLog, deriving LogSeverity per log type.
• Fortinet (network and perimeter): Maps FortiGate traffic, UTM, IPS, VPN, and authentication events to CommonSecurityLog, and detects FortiGate logs arriving directly from Sentinel.
• Cisco ASA (network and VPN): Maps access control, connection, VPN, and authentication events to CommonSecurityLog, deriving LogSeverity and DeviceAction from ASA message codes.
• Cisco Meraki (network): Maps flow, VPN firewall, IP flow, URL, and event logs to the Syslog table.
• ExtraHop (network detection and response): Tags ExtraHop Reveal(x) detections with risk severity, extracts offender and victim IPs, and drops low-risk noise.

Teams can browse and add Packs directly from Observability Pipelines. Each pack comes preconfigured with mapping logic, so teams do not have to build mappings for each supported source from scratch.

To understand how these mappings work, suppose you’re a security engineer sending Cisco ASA logs to Microsoft Sentinel. Before you add a Pack, a raw ASA syslog message may require an understanding of ASA message codes, while Sentinel’s analytics rules expect fields in the CommonSecurityLog schema. An analyst writing a rule for denied connections needs to determine which ASA message code indicates that the connection was denied and identify the source IP, destination IP, port, and protocol in the raw message.

After you add the Cisco ASA pack, Observability Pipelines maps that event to CommonSecurityLog. The Pack derives DeviceAction (such as deny or permit) from the ASA message code and maps IP, port, protocol, and interface information to their corresponding fields. Analytics rules can now target CommonSecurityLog fields instead of parsing raw ASA syslog messages.

Once the Pack is added, you can use Datadog Live Capture to validate the mapping against production log samples as they pass through the pipeline. The following screenshot shows a raw ASA event entering on the left and exiting on the right as a CommonSecurityLog event, with DeviceAction, LogSeverity, and the relevant network fields.

Investigate security activity across every source

When the same activity appears in different formats across systems, teams can end up maintaining source-specific detection and investigation logic. Microsoft Sentinel Packs apply one mapping model across supported sources before the data reaches Sentinel. Firewall traffic, VPN logins, and network detections can arrive in consistent Sentinel tables, enabling teams to build analytics rules and workbooks around the same schema fields. Inside each Pack, processors match specific events and apply the corresponding schema mapping automatically, giving analysts events that are already structured for investigation.

This standardization also reduces the amount of source-specific parsing required during an investigation. Suppose you’re investigating suspicious outbound activity that spans your firewalls and VPN. Without a shared schema, you may need to query each source in its own format and stitch the timeline together by hand. With Microsoft Sentinel Packs applied in the pipeline, you can query CommonSecurityLog fields like SourceIP, DestinationIP, and DeviceAction to see related activity across Cisco ASA, Fortinet, and Palo Alto side by side.

Control Microsoft Sentinel ingest costs without losing visibility

Because Microsoft Sentinel Packs apply mappings before logs leave your environment, you can decide which data you send to Sentinel and focus its per-GB ingest on high-value data. For example, a team might send denied connections, failed authentications, and high-severity threats to Sentinel while routing full-fidelity raw logs to low-cost storage and dropping repetitive allow events. When the relevant events have already been mapped, these decisions can use normalized fields such as DeviceAction and LogSeverity instead of relying on source-specific values.

The ExtraHop Pack can also reduce low-risk noise before data reaches Sentinel by dropping detections below its configured risk threshold. Together, mapping and filtering in the pipeline can help teams reduce low-value ingest volume while retaining the events they need for detection and investigation.

Map your security logs for Microsoft Sentinel investigations

Microsoft Sentinel Packs map supported security logs to Sentinel table schemas before ingestion, helping teams investigate across sources with consistent fields while reducing manual mapping work and unnecessary ingest volume.

Microsoft Sentinel Packs are available for Observability Pipelines for Palo Alto Networks, Fortinet, Cisco ASA, Cisco Meraki, and ExtraHop. To get started, open the Packs gallery in Observability Pipelines and add the Pack that matches your log source. For more information about configuring pipelines, see the Observability Pipelines documentation. You can also read about how Observability Pipelines can enrich logs with Reference Tables before routing them downstream.

If you don’t yet have a Datadog account, sign up for a 14-day free trial to map and route security logs before they reach Microsoft Sentinel.

Start an investigation

When an issue in production triggers an alert, the people responding to it are often working in Slack while the evidence they need is elsewhere. Responders need to move between conversations, telemetry data, source code, and incident tooling as they form hypotheses, coordinate actions, and keep stakeholders informed. That context switching can slow down a time-sensitive investigation and make updates harder to follow.

Bits Chat brings Datadog’s natural-language interface into Slack, giving responders access to Bits AI from the channel where they’re already collaborating. During an incident, teams can ask Bits to investigate the alert, pull in relevant telemetry data, generate a pull request for the fix, and keep track of follow-ups without leaving Slack.

In this post, we’ll follow an incident from alert to resolution and show how you can manage your incidents directly in Slack to:

• Start an investigation
• Troubleshoot with Bits and your responders
• Act on the investigation
• Resolve the incident and capture what happened

Start an investigation

Let’s say you’re an on-call engineer for an ecommerce website and you receive a monitor notification that the website’s recommendation service is experiencing errors and failing because of timeouts. When you declare an incident, Datadog creates a dedicated Slack channel so that you and your team can collaborate on fixing the issue. Instead of leaving the conversation to begin gathering evidence, you can type @Datadog investigate in the channel to ask Bits to start an investigation.

Bits Investigation analyzes the issue by forming hypotheses from relevant telemetry data, runbooks, and past incidents. You can also include additional context or a hypothesis in the initial @Datadog investigate message to help Bits focus its investigation from the start. As the investigation runs, Bits posts updates in Slack so that responders can follow its work alongside their own discussion.

Troubleshoot with Bits and your responders

When Bits Investigation finishes, it returns its root cause findings and recommended next steps to the Slack conversation. In our example of the ecommerce website, Bits identifies a recent code change as the likely cause of the issue and shares the supporting telemetry data.

Team members can then mention @Datadog to ask questions about the findings, go deeper into analyzing the telemetry data, compare the findings with what the team has observed, and decide how to resolve the issue. For example, they can ask which endpoints and customer regions are affected and whether the errors are affecting any downstream services. All of these activities can happen in the same Slack channel, and the investigation stays connected to the related discussion.

Act on the investigation

After responders identify the likely cause, the next challenge is turning that finding into action. Bits Remediation suggests next steps based on the investigation and enables teams to take actions directly from the incident conversation in Slack. Those actions can include adding more responders, running incident workflows, or posting Status Pages updates as the incident progresses.

In our example incident, the responders ask Bits to start a fix. Bits Remediation passes the relevant context to Bits Code, which creates a dedicated code channel in Slack and uses the investigation findings to generate the fix. Bits Code then creates a pull request for the responders to review.

Other incidents might call for different actions. Bits Remediation can also trigger triage actions from chat, including sending messages to teammates, paging engineers via Datadog On-Call, and creating incident or follow-up records.

Resolve the incident and capture what happened

Addressing the problem does not end the incident workflow. Responders still need to communicate the outcome, resolve the incident, preserve the investigation context, and create follow-up work. These final stages of incident response can also happen in Slack through @Datadog.

Once the issue in our example has been addressed, Bits confirms that error rates have returned to their normal level. The responders then ask Bits to resolve the incident and create a postmortem notebook from the investigation. Because Bits already has the context from the response, the notebook documents the incident summary, relevant findings, resolution, and next steps.

The notebook gives responders a starting point for postmortem and follow-up work, removing the need to reconstruct the incident from separate conversations and tools. Creating postmortems helps teams and Bits respond to future incidents more quickly.

Start managing incidents in Slack with Bits Chat

Bits Chat in Slack brings investigation, collaboration, and action into the conversation where incident responders are already working. From the initial alert through remediation and follow-up, teams can handle incidents in Slack without having to move between tools to coordinate the response. To get started, follow the Bits Chat setup instructions for Slack. You can also learn more about using Bits Investigation to investigate issues and using Bits Code to generate code fixes.

If you don’t have a Datadog account, you can sign up for a 14-day free trial to start using Bits Chat in Slack.

Kafka is a crucial and widely used technology: 80% of the Fortune 100 rely on the event streaming platform as part of their stack, according to Apache. But Kafka issues can be complex to manage and even more difficult to troubleshoot, as the same symptom can point to very different problems. Suppose consumer lag on your checkout-events topic suddenly exceeds its SLA. The cause might be an offline partition, a consumer that has stopped processing messages, a configuration change, or a payload that the consumer cannot deserialize. Determining which problem you have, and whether unprocessed messages are approaching their retention limit, can require correlating many different layers of your stack to piece together Kafka health, application behavior, configuration and schema changes, and message content.

Datadog Kafka Console collects health and performance data directly from your Kafka infrastructure, providing a mission-control center from which you can identify and remediate issues across your clusters, brokers, topics, partitions, consumer groups, configurations, schemas, and messages. With context from these sources, Kafka operators, platform engineers, and application developers can determine what is failing, understand the potential impact, and investigate the underlying cause without piecing together information from separate workflows.

In this post, we’ll show how Kafka Console helps you:

• Identify issues across brokers, partitions, and schemas
• Troubleshoot Kafka at the message level
• Optimize Kafka performance
• Connect Kafka infrastructure and application context

Identify issues across brokers, partitions, and schemas

Kafka Console helps you determine whether Kafka is available and keeping up with the applications that depend on it. After you set up Kafka Console, a Datadog Agent check connects to your Kafka cluster and collects health and performance data. The Clusters, Topics, and Brokers views provide health information for resources including brokers, topics, partitions, and consumer groups.

Kafka Console also provides recommended monitor templates for these Kafka resources, including offline partitions and consumer lag approaching time- or byte-based retention limits. This lets you alert teams when a Kafka condition puts a pipeline at risk rather than waiting for someone to notice the problem on a dashboard.

For example, if lag spikes on the checkout-events topic, you can inspect its partition state, throughput, and consumer lag from the topic view. Kafka Console surfaces conditions such as offline and under-replicated partitions and shows the maximum consumer lag across partitions. It can also indicate whether lag is approaching the topic’s retention limit, which helps you understand when a processing delay is becoming a data loss risk rather than relying on a manual estimate.

Kafka Console also correlates configuration and schema changes with Kafka health. Change events appear on throughput and lag graphs, and you can open a detected configuration change to inspect what changed. If lag starts increasing after a configuration or schema update, that temporal context gives you a concrete place to begin investigating instead of assuming that the broker or consumer is responsible.

When application context is available, the topic view also lists producer and consumer services and links them to ownership information from the Software Catalog, such as the responsible team, repository, on-call engineer, and Slack channel. This context helps platform teams determine which services are affected and identify the appropriate owner when a producer or consumer is misbehaving.

Troubleshoot Kafka at the message level

Metrics can tell you that a consumer has fallen behind or started returning errors, but they cannot always explain what was in the message that triggered the problem. Kafka Console lets authorized users retrieve messages from topics on demand so they can inspect the payload associated with an issue.

Consider a consumer that begins reporting deserialization errors on checkout-events. Instead of relying only on an application stack trace to infer the problematic input, you can use the Messages view to retrieve messages by partition and offset and inspect their payloads. Combined with schema version history, this can help you determine whether an unexpected message structure or schema change contributed to the failure.

Message inspection requires additional access because payloads can contain sensitive application data. In addition to the Kafka permissions required for cluster monitoring, the Datadog Agent user needs read access to the relevant topics. Remote Configuration must be enabled at the organization, Agent, and API key levels, and Datadog users who retrieve messages need the Data Streams Monitoring Capture Messages permission. See the Kafka Console setup documentation for the complete prerequisites.

Kafka Console can also collect schemas from a configured schema registry. From a topic, you can inspect its current key or value schema and compare versions. Bringing schemas and message payloads into the same investigation as lag, throughput, and configuration changes helps you distinguish an infrastructure problem from a data problem that metrics alone cannot explain.

Using all of this rich context, Bits AI can investigate Kafka issues for you. Within any monitor, click “Investigate with Bits” to get a clear investigation report, a hypothesis tree, and a recommended fix to resolve issues in minutes.

Optimize Kafka performance

After you identify a Kafka performance problem, determining which configuration to change can require another round of investigation. Kafka Console provides recommendations for Kafka configurations to help teams improve throughput, reduce lag, and use cluster resources more efficiently.

For example, persistent consumer lag can indicate that a consumer’s configuration does not match the workload it needs to process. Kafka Console can surface configuration recommendations that give you a starting point for tuning rather than requiring you to identify candidate settings through trial and error.

Kafka Console brings these recommendations into the same Datadog workflows where teams investigate Kafka health. Kafka context is available alongside Data Streams Monitoring (DSM) data when both are in use. In addition, Kafka Console works with Datadog’s AI capabilities, including Bits AI and the Datadog MCP Server, as part of troubleshooting and optimization workflows. All Kafka data surfaced in the Datadog UI is available through the MCP Server, including bundled skills that bring expert-level Kafka knowledge to common investigation workflows. Kafka users can simply ask, “Why is my topic lagging?” without needing to understand the minutiae of brokers, replication, serialization, and configuration options.

Connect Kafka infrastructure and application context

Kafka Console and Data Streams Monitoring provide complementary views of Kafka-based applications. Kafka Console collects information directly from Kafka clusters, so you can monitor cluster health without instrumenting producer and consumer application code. Meanwhile, DSM application instrumentation maps data flow and latency across producer and consumer services.

This distinction matters when troubleshooting. If checkout-service falls behind on checkout-events, Kafka Console can help you inspect broker and partition health, lag, configurations, schemas, and messages. DSM can add application-level context about the producer and consumer services that participate in the pipeline. Teams can use Kafka Console independently, use DSM independently, or combine them for broker-to-application context.

Because Kafka Console reads Kafka health data from the cluster, application teams can investigate infrastructure-level causes even when their applications are not instrumented for DSM. When DSM instrumentation is available, platform teams can connect Kafka problems to affected services and application owners, reducing the ambiguity that often surrounds responsibility for streaming pipeline failures.

Investigate Kafka issues with Kafka Console

Kafka Console helps Kafka operators and application teams identify what is failing, understand the potential impact, and investigate the cause with Kafka health, configuration, schema, and message context in Datadog, using Bits AI or with your preferred agent via the Datadog MCP. This control center for Kafka is particularly useful when symptoms such as consumer lag could originate from infrastructure, application behavior, or the data itself.

To configure Kafka Console and review its requirements, see the Kafka Console documentation and Kafka Console setup guide. You can also review the Kafka monitors and automation documentation to start alerting on conditions such as offline partitions and consumer lag approaching retention limits.

If you don’t have a Datadog account, sign up for 14-day free trial to monitor, inspect, and tune Kafka from broker to message.

How we extended CUPED to segments and percentiles

When Datadog acquires a company, we usually rebuild the product rather than plugging it in as is. That’s exactly what we did with Eppo, an experimentation and feature-management platform. Eppo’s feature-management capabilities became Datadog Feature Flags, while experimentation became Datadog Experiments. This post focuses on the experimentation platform and four changes we made to help you get to a decision faster.

Datadog Experiments lets you analyze randomized experiments, or A/B tests, alongside observability data to understand how releases affect business outcomes, user behavior, and application performance. While warehouse-native metrics often act as the primary basis for decision making, you can also leverage Real User Monitoring (RUM) or Product Analytics as part of your experiment metrics.

We focused the Experiments rebuild on helping you move from an experiment result to a confident decision more quickly, with statistical efficiency, interpretable impact, verifiability, and faster feedback. In this post, we’ll cover key features of Experiments, including how we:

• Extended CUPED to segments and percentiles
• Rebuilt global lift around coverage
• Made warehouse metrics easier to verify
• Added behavioral and performance metrics to experiments

How we extended CUPED to segments and percentiles

CUPED (Controlled-experiment Using Pre-Experiment Data) uses information about each subject from before an experiment to reduce metric variance, which tightens confidence intervals. The result is faster experiment conclusions with smaller sample sizes.

At Eppo, we implemented CUPED as a regression adjustment with a separate subject-level model for each variant. The resulting lift estimate was unbiased only for the population used to fit those models. This made it impractical to extend CUPED to segments, since each segment would need its own refitted model.

Apply CUPED consistently across segments

For Experiments, we built a more flexible CUPED model inspired by CUPED’s original authors, who describe the method as a metric augmentation rather than a regression adjustment. You start from the unadjusted lift and add adjustment terms chosen to average out to zero over many randomizations. Because those terms are zero in expectation, the adjustments add no bias, but can reduce variance.

A single set of coefficients gives zero-expectation adjustments for any segment. So a single model run, reusing the same global coefficients, now enables unbiased, CUPED-adjusted results for any segment. In practice, this generally reduces variance, and it also keeps segment results consistent with the overall result.

The waterfall view above starts from the raw lift and applies each adjustment term to arrive at the CUPED-adjusted lift. In the example experiment, that variance reduction shortened the estimated run time by about 52%.

Extend variance reduction to percentile metrics

The metrics that best capture application performance are often percentiles. Variance reduction on percentiles has long been out of reach for commercial experimentation platforms. The standard, regression-based implementation of CUPED does not naturally extend to percentile metrics, because there is not an obvious subject-level outcome to predict. But augmentation removes that limit.

We take the ordinary p90 and add adjustment terms like we would for any other metric. A p90 page load time can now help inform an experiment or guard a rollout on less traffic than an unadjusted percentile would need. Because CUPED now extends to percentiles, a change that causes a latency spike can be caught sooner and rolled back faster.

How we rebuilt global lift around coverage

Global lift estimates how an experiment’s result would change your company-wide metric total if you rolled the treatment out to every eligible user. One approach would be to apply the experiment’s relative lift to your company-wide total, but that assumes every user in that total is affected by the change. If only a fraction are, the full relative lift overstates the effect. Global lift accounts for that gap. It depends on local lift, the relative lift among enrolled subjects, and coverage, the share of your company-wide metric total that the experiment’s eligible users account for.

At Eppo, we calculated global lift and coverage separately and reported both. The two weren’t designed to combine, and there was no clear relationship between them. Multiplying coverage by local lift wouldn’t reproduce the reported global lift. Global lift also didn’t extend to windowed metrics. Getting a global lift number for a windowed metric meant building a second, unwindowed version, which might have a different local lift. Global lift then came from that second metric’s lift, not the metric you were evaluating.

For Datadog Experiments, we rebuilt global lift as the product of two factors (local lift and coverage) you can read on their own:

Global lift = Coverage × Local lift.

Say an experiment’s eligible users make up half your revenue. Coverage is 50%. Multiply that 50% coverage by a 10% local lift on revenue, and you get a 5% global lift. Every global lift number comes from multiplying coverage by local lift, so you can always check it yourself.

The metrics table above shows relative lift, coverage, and global lift as separate columns, with a 13.0% relative lift and 40.3% coverage producing the 5.23% global lift.

Windowed metrics are often advantageous for decision making and impact reporting. Now you get both of these benefits in the Experiments platform. Global lift works for windowed metrics on Experiments because coverage is computed separately from global lift, rather than inside it. You no longer have to maintain a second metric to get global lift, or worry about two metrics that might have two different lifts.

How we made warehouse metrics easier to verify

At Eppo, about 20 pipeline steps sat between your raw tables and a reported result. Two things were hard to do with that pipeline: confirm a result was correct and diagnose one that looked wrong. To understand how the pipeline produced a particular number, an analyst had to handcraft queries to reproduce what it did. One assumption that differed from the pipeline’s was enough to produce discrepancies that were difficult to trace. So, an analyst would often send Eppo the query, and the Eppo team would help track down where its assumptions diverged from the pipeline’s.

For Datadog Experiments, we built Copy SQL to reduce that back-and-forth so you can verify your own results. For your warehouse-native metrics, you can copy the query behind a number and run it in your own warehouse to trace an unexpected result. A copied query lands very close to the dashboard number, though because the pipeline is incremental, late-arriving data can cause small differences.

The example above shows the full query behind the revenue metric, with the pipeline’s steps as named stages. It is also important to note that Copy SQL reproduces results for metrics without CUPED adjustment. Separately, Experiments runs diagnostic checks before you interpret results, flagging issues like a traffic imbalance between variants or a metric with no data.

How we added behavioral and performance metrics to experiments

At Eppo, experiments read data from the warehouse and reported results on a schedule. The business metrics that decide whether you ship a change, like revenue or retention, can land hours later. Early in a test, you want to know the change hasn’t broken something before those warehouse metrics arrive.

In Experiments, you can now build metrics from Real User Monitoring (RUM) and Product Analytics data. These capture how users behave and how your app performs, including pageviews, clicks, and errors, across web and mobile.

For example, the metric builder above lets you pick a RUM or Product Analytics event, like a checkout click or a product pageview, as the basis for an experiment metric. You can also build metrics from APM Spans, now in Preview. RUM and Product Analytics metrics update in near real time (typically about every 15 minutes), without any separate streaming infrastructure. Now you can see early behavioral and performance signals while your warehouse metrics are still landing. Your warehouse metrics stay as they are. You get an additional, early signal on whether a change may be degrading behavior or performance, so you can act sooner.

Within minutes of your experiment launching, you can also watch full Session Replays from the first users exposed to it. Confidence intervals will still be wide at this stage, but qualitative signals from Session Replay show how your new variant is behaving and how users are interacting with it.

Get started with Datadog Experiments

With Datadog Experiments, experiments conclude faster and their results are easier to trust. CUPED now extends to percentiles, so you can catch a latency spike sooner and roll it back faster. Global lift breaks into coverage and local lift, so you can check the math yourself. Copy SQL hands you the query to rerun in your own warehouse. Metrics from RUM and Product Analytics give you an early read on behavior and performance before your warehouse results are in. Together, these platform improvements can shorten the time from result to decision, and make it easier to verify results.

To learn more about Datadog Experiments, check out the documentation for how to plan and launch your first experiment.

If you’re not already a Datadog customer, sign up for a free 14-day trial.

Agent

Prelude

Released on: 2026-09-09

• Please refer to the 7.83.1 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Fix bug which made fast network path test billed to customer
• Release the containerd view snapshot and lease taken for a container image SBOM scan even when the scan is cancelled or times out. The release ran on the scan's own context, so a scan that hit its deadline left the snapshot behind, and on a lazy snapshotter that snapshot holds the layer it materialised.

Other Notes

• The fleet installer daemon now reports the DDOT (OpenTelemetry Collector) process state as part of the agent state sent to Datadog, so DDOT version and configuration updates can be monitored. The state is read from the process manager when it supervises DDOT, and from systemd or the Windows service manager otherwise. It is also visible in the output of
  datadog-installer status.

Datadog Cluster Agent

Prelude

Released on: 2026-09-09 Pinned to datadog-agent v7.83.1:
CHANGELOG.

Bug Fixes

• Fix Cluster Agent graceful shutdown to release the Kubernetes leader-election lock before exiting, allowing another replica to take over without waiting for the lease to expire.

Plan and instrument a launch before it ships

Launches are high-stakes moments for product managers (PMs), but understanding how a new product or feature is performing can be difficult. Teams may lack the instrumentation they need to track performance, or they may miss defects that affect specific user segments during QA. When a launch underperforms, PMs can spend days determining whether the problem is the feature itself or how it’s being measured, leaving analysts, engineers, and designers needing to rebuild what should have existed on day one.

Launches in Datadog Product Analytics connects launch planning, instrumentation, experimentation, user experience analysis, and reporting together in a single, connected workflow. Starting with a product brief and the Datadog Feature Flag, teams can define what they need to measure before release and monitor the most important user signals throughout the rollout.

In this post, we’ll show you how you can use Launches to:

• Plan and instrument a launch before it ships
• Catch broken experiences across real user segments
• Validate experiments as you roll out
• Bring launch signals into one place
• Keep measuring after the rollout

Most PMs already know what questions their launch has to answer, but they may not be sure those questions will be answerable on launch day. This happens when the necessary instrumentation to track how users are interacting with the new feature is not in place at rollout.

When you create a launch in Product Analytics, the planning workflow moves through four steps: context, questions, tracking plan, and experiment. Bits AI drafts each step for you to edit, rather than handing you a blank form.

For example, say you’re launching an exclusive cart offer that adds an upsell to an ecommerce cart. In the context step, you paste the brief and connect the feature flag the offer will ship behind. In the questions step, you can draft measurement questions, such as how many eligible users see the offer, how many interact with it, and whether it changes the average order value, or use Bits AI to automatically generate questions based on your launch’s brief. You can then edit, add, or remove these questions to match the decisions your team expects to make.

The tracking plan step derives the events and properties needed to answer each of those questions. This keeps instrumentation tied to the launch’s measurement goals, instead of defining events only around available UI interactions.

Datadog compares that plan against the events you already send and shows you which events and properties are missing to answer your questions. Where a relevant event already exists, Datadog reuses it instead of instrumenting the same thing twice. Once you confirm the tracking plan, Datadog can open a pull request with the required instrumentation. Your team can review the measurement logic and implementation before the rollout starts, rather than discovering gaps after data starts to arrive.

This workflow can also apply to product changes that don’t modify the web UI, such as ranking or recommendation algorithms. Because the tracking plan starts with outcomes rather than clicks, teams can define measurement around the behavior that matters for the launch.

Catch broken experiences across real user segments

Pre-release QA cannot cover every device, screen size, geography, network condition, and user behavior that a feature will encounter in production. This gap makes it difficult to catch defects that leave a feature technically available but difficult or impossible to use.

In our exclusive cart offer, for example, a quantity selector might render correctly and accept clicks without triggering the expected action. On a mobile screen, the component might also extend past the viewport, putting the Add to Cart button outside the usable page. A test on a developer’s laptop may not expose either problem.

As traffic increases, Launches shows you Session Replay and RUM data from users exposed to the feature flag. Comparing the treatment and control sessions helps you focus an investigation on friction associated with the product change.

Datadog also enables you to break down sessions by dimensions such as device type, screen size, or country. If an issue only affects one segment, you can move from that segment to the relevant issues and example Session Replays to understand what those users experienced.

This context complements experiment metrics, helping you better interpret product data. A conversion decrease can tell you that a treatment is underperforming, while a Session Replay that captures repeated clicks on a nonfunctional quantity selector helps explain why.

Validate experiments as you roll out

With instrumentation in place before launch day, the Datadog Feature Flag you connected to your launch becomes the link between the change and the users who saw it. Launches in Product Analytics runs your rollout through Datadog Experiments, so you can compare product, performance, and business outcomes between treatment and control using statistics your data science team has already approved.

In the experiment step of the launch workflow, experiment metrics will be prefilled based on the launch definition and remain editable before the experiment begins. This lets data science teams establish reusable experiment designs instead of redefining the methodology for every launch.

From here, Datadog Feature Flags controls targeting and progressive exposure. A rollout can begin with employees or a small percentage of traffic, giving teams an opportunity to find experience and configuration problems before increasing exposure. After validating the initial cohort, the team can increase traffic for the experiment.

Datadog Experiments also evaluates diagnostics while an experiment runs. These checks can surface issues such as sample ratio mismatch, over-assignment, dilution, and missing metric coverage while they are happening, rather than as a final outcome. Finding an uneven treatment split early, for example, can help the team correct the configuration instead of discovering the problem after the full experiment has run.

Bring launch signals into one place

Launch investigations become harder when product, engineering, and support teams each work from different signals. Metrics might indicate that adoption declined, while RUM data captures frontend errors, and support tickets describe the same problem from the user’s perspective.

The launch command center in Datadog Product Analytics organizes these signals around the feature flag that identifies who was exposed. As rollout data becomes available, the command center brings together the key signals about the launch, including rollout stage and exposure count, KPIs, experiment diagnostics and results, technical health, the segment coverage table with its friction and visual findings, the launch funnel, and support tickets.

Connecting these signals makes individual findings easier to interpret. If conversion drops, for example, you can investigate whether affected sessions also contain user frustration signals and frontend errors. Likewise, you can evaluate an experiment result alongside the performance and behavior of the treatment population.

Datadog can connect all of this context because the underlying signals already exist elsewhere across the platform. Product Analytics captures behavioral data, Session Replay provides session context, RUM captures frontend performance and errors, Feature Flags identifies exposure, and Experiments measures treatment effects. Launches in Product Analytics organizes those existing signals around the specific product change your team is evaluating.

Keep measuring after the rollout

The questions that define a successful launch remain useful after a feature reaches full exposure. Instead of treating the measurement plan as a temporary launch artifact, teams can continue using its KPIs to evaluate the success of a product flow.

After you set up your launch, Bits AI automatically generates an editable KPI dashboard based on the tracking plan you defined. This gives teams a persistent view of the metrics that will help them answer key questions about the launch. They can also use this dashboard as a starting point for ad hoc analysis when new questions arise.

For changes to important flows, such as checkout or login, that measurement context can also inform ongoing journey monitoring. Datadog Journey Monitoring brings together Product Analytics, RUM, Synthetic Monitoring and Testing, and Session Replay data to track traffic, conversion, time to completion, errors, and uptime across critical user journeys.

Launches in Product Analytics can suggest ongoing coverage for a flow that may have been affected by your release, while leaving the decision to create or modify the journey with the user. This lets teams carry useful measurement context beyond the rollout, so the questions that defined success before release can continue to inform how the team monitors the experience.

Measure launch success from planning through rollout

Launches in Datadog Product Analytics helps product teams define measurement before release and keep product, experiment, and user experience signals connected throughout a rollout. When results change, teams can investigate based on the context they established before launch, instead of reconstructing instrumentation and analysis afterward.

To understand the capabilities that support this workflow, read the documentation for Datadog Experiments, Feature Flags, Agentic Onboarding for Product Analytics, and Journey Monitoring.

If you don’t have a Datadog account, sign up for Product Analytics Launch Agent to get started, or start a 14-day free trial.

Cut time to remediate (TTR) with response actions

Modern threats increasingly unfold at runtime, where attackers exploit live workloads, spawn malicious processes, and move laterally across your environment. Detecting that activity is essential, but a signal only matters if you can stop it. When a threat appears, every step before a response gives an attacker more time to act.

Datadog Workload Protection can now directly remediate threats with both automated and manual response. When the Datadog Agent finds malicious activity, it can automatically kill or isolate the offending process based on your Agent rules. Alternatively, when a threat needs a closer look, you can step in and respond yourself. This post covers how Workload Protection helps you:

• Cut time to remediate (TTR) with response actions
• Contain threats with precision and control

Time to remediate (TTR) spans detection, investigation, decision, and intervention. Each step adds latency that widens an attacker’s window. Attackers now exploit vulnerabilities almost as soon as they’re disclosed, and that gap keeps shrinking. Workload Protection gives you two ways to stop a threat. You can automate the response with an Agent rule, or respond yourself when a signal is raised.

Automated response to close the attacker’s window

Turn on automated response for an Agent rule, and the Agent acts the moment it finds a match, reducing the manual steps between detection and intervention. Use Datadog’s default out-of-the-box (OOTB) rules or write your own custom rules. Automated response terminates matching processes, so it’s best suited to activity that is unambiguous and never legitimate.

For example, take a crypto mining threat. When automated response is active for an Agent rule, the Agent can terminate matching processes in the services where it’s safe to do so while continuing to monitor everywhere else. The attacker’s window closes when the Agent finds a match rather than when a person is able to act. The Agent also records each automated kill in a signal you can review afterward.

Manual response to investigate and act in one place

You don’t have to automate every rule. When a rule you haven’t automated raises a signal, you can investigate and then choose a manual response. Responding manually has traditionally meant extra friction. Security teams often have to leave their investigation workflow to contain a threat. This slows response, invites errors, and compounds into operational fatigue, a known risk factor for security breaches. Now you can respond to a threat in the same place you detect it.

When the Agent identifies suspicious activity at runtime, it raises a Workload Protection signal. From there, you can decide whether to kill the process or container, isolate the workload, or both. For example, the signal below shows the response actions a user took against an affected container, and the investigation graph maps how the attack unfolded.

Contain threats with precision and control

Acting on the wrong resource can take down healthy services, and without proper authorization controls, the feature itself could be turned against you. Workload Protection helps ensure every action, whether automatic or manual, is authorized by the right person and targets the right resource. It works at the kernel level, creating a real-time view of every process and container. That view lets it confirm who is allowed to act, what the action targets, and how it acts on that resource without disrupting anything healthy.

Authorizing the actor

First, an action has to prove it’s allowed to run. Killing a process or isolating a workload requires elevated permissions granted specifically for response actions. Without these permissions, you can’t configure an Agent rule for automated response or trigger a manual response. Every action is also recorded and timestamped, giving you a history you can audit later.

Identifying the resource

Next, the action has to reach the right resource without touching a healthy one. The Agent continuously maintains a process tree, tracking each one by namespace-independent process ID (PID) along with its parent and ancestor processes. To respond on a container, the Agent uses control group (cgroup) tracking and the one-to-one mapping between a container ID and its cgroup to associate each process with the container it belongs to. This gives the Agent a precise, real-time map of your execution environment. Once the right process or container is identified, the Agent acts on it directly.

Executing the response

Finally, the Agent can respond safely. It can kill the offending process or container to stop the threat at its source, isolate a compromised workload without disrupting the healthy traffic around it, or do both. To terminate a process, the Agent issues a kill both from user space and inside the kernel, targeting the process’s PID or cgroup.

To isolate traffic precisely, the Agent uses eBPF-based filters injected directly into the kernel, inspecting traffic on both ingress and egress. For each packet, a traffic control (TC) hook identifies the process and cgroup by first checking kernel structures and then falling back to the Agent’s record of network flows. The packet is then evaluated against the filters stored in an eBPF map:

• If the packet matches the isolation rule, it’s dropped.
• If it doesn’t match, it passes through untouched.

The filters operate at the kernel level and target PIDs and cgroups. Berkeley packet filters can also match on specific traffic, ports, and protocols, giving you full control over what you isolate. Isolation works at the finest granularity, distinguishing a single process or container from everything else sharing the same IP or port. Together, these controls help ensure that each action targets only the resources associated with the signal, and runs only when triggered by an authorized user or automatically by an Agent rule configured for automated response.

Get started with Workload Protection response actions

Workload Protection now lets you stop threats at runtime, automatically through Agent rules or manually from a raised signal. Every action is tracked, timestamped, and documented, so you can audit exactly what happened and when. This creates a tighter loop between detection and response without requiring a handoff or switching tools. Workload Protection no longer just detects threats. It acts on them.

To enable automated or manual response actions on your own workloads, review the response requirements.

If you’re not already a Datadog customer, sign up for a free 14-day trial.

Agent

Prelude

Released on: 2026-09-03

• Please refer to the 7.83.0 tag on integrations-core for the list of changes on the Core Checks

New Features

• Add a Data Security provider that schedules one-off database scan checks triggered through Remote Configuration. It is enabled when both data_security.enabled and shared_library_check.enabled are set, and currently targets PostgreSQL databases already monitored by the Agent (via the postgres check).
• Add k8s cluster receiver , k8s leader elector extension , and count connector to the DDOT (Datadog Distribution of OpenTelemetry Collector) default manifest, enabling collection of Kubernetes cluster-level metrics, leader election coordination for Kubernetes receivers, and count-based metric generation via the OpenTelemetry Collector pipeline.
• Add Helm rollback action
• Adds an Agent Data Plane (ADP) preflight mode, controlled by the new data_plane.preflight_mode setting (enabled by default).

When data_plane.enabled has not been set at all, the Agent starts ADP once at startup for 90 seconds in an isolated configuration, sends a single throwaway metric through it, then stops it and reports any startup errors to Datadog as agent telemetry. This surfaces environment-specific ADP problems before ADP is enabled for real.

The preflight process handles no customer data: it runs in standalone mode, listens only on a temporary DogStatsD endpoint under the Agent's run directory, does not register with the Agent's remote agent registry, and never takes over the Agent's own DogStatsD port. The temporary configuration it is given contains the Agent's resolved configuration, so it is written user-only and removed once the run finishes. To keep values that were only ever held in memory from being written out this way, the pre-flight does not run at all when secrets are in use: when secret_backend_command , secret_backend_type , or multi_secret_backends is set, or when any setting has already been resolved from a secret. Setting data_plane.enabled explicitly to either true or false , or setting data_plane.preflight_mode to false , also disables the pre-flight, as does running an Agent package that does not ship ADP.

• APM : Add support for span-derived primary tags on span metrics produced by the Datadog Distribution of OpenTelemetry Collector (DDOT). Set span_derived_primary_tags on the datadog connector's traces section to a list of attribute keys, and the value of each key found on a span (or, failing that, on its resource) is attached to the APM stats the connector emits, letting you break down span metrics by those tags:

connectors:
  datadog/connector:
    traces:
      span_derived_primary_tags: [team, region]

Keys absent from both the span and its resource attributes are omitted. Each key must also be configured as a primary tag in your Datadog organization; keys that are not registered as primary tags are dropped by the intake and do not appear on the resulting span metrics.

• CWS set actions accept a new capture field, a regular expression with a single capture group that is applied to the value of field to extract part of it. This makes it possible to lift an identifier embedded in an event field, such as a command id inside a file path or an IAM role inside an IMDS url, and store it as a scoped variable rather than storing the whole field value. capture can only be used together with field , and only on fields holding a single string. A value that does not match the expression leaves the variable untouched.
• When data_security.enabled is set, the Agent now forwards sensitive-data-scanner findings to Datadog as structured sds-result payloads on the event platform.
• Scaffold PostgreSQL support to the data security check.
• DDOT (Datadog Distribution of OpenTelemetry Collector): the embedded datadog exporter now honors the orchestrator_explorer setting when the OpenTelemetry Agent runs in standalone mode (DD_OTEL_STANDALONE=true). When enabled, Kubernetes resource manifests collected by a k8sobjects receiver in the exporter's logs pipeline are forwarded to the Orchestrator Explorer (Kubernetes Resources) intake. In connected mode this setting is ignored, as the Datadog Cluster Agent already collects and ships orchestrator data.
• Add support for additional log collection options for Kubernetes workloads through DatadogInstrumentation resources.
• Adds support for configuring Network Path Dynamic Test filters through Remote Config.
• Adds support for scheduling Network Path tests through Remote Config.
• Added ConfigMap collection to the Kubernetes Orchestrator. ConfigMap manifests are sent with their data and binaryData fields stripped. The collector is disabled by default (IsStable: false) and must be activated explicitly by listing configmaps in the collectors field of the orchestrator check instance configuration.
• The DDOT (OpenTelemetry) config converter now automatically injects the cumulativetodelta processor into metrics pipelines that export to the datadog exporter, converting all cumulative metric types (sum, histogram and exponential histogram) to delta. The processor is added only to metrics pipelines, and is skipped for any pipeline where a cumulativetodelta processor is already defined. This behavior is controlled by the new cumulativetodelta entry in otelcollector.converter.features , which is enabled by default; remove it from that list to disable the auto-injection.
• OTLP ingestion: Adds a new otlp_config.logs.infra_attributes.tags_as_ddtags option. When enabled, custom tagger-derived tags (for example, tags configured via kubernetesResourcesLabelsAsTags / kubernetesResourcesAnnotationsAsTags ) are written as real Datadog log tags instead of log attributes for OTLP logs ingested directly by the Agent. Default behavior is unchanged. (commit 522ee5f )
• DDOT: The infraattributes processor now supports a new logs_tags_as_ddtags option. When enabled, custom tagger-derived tags (for example, tags configured via kubernetesResourcesLabelsAsTags / kubernetesResourcesAnnotationsAsTags ) are written as real Datadog log tags instead of log attributes. Default behavior is unchanged. (commit 522ee5f )
• Added a com.datadoghq.remoteaction.agent Private Action Runner bundle exposing read-only datadog-agent operations (status, diagnose, and configuration) as remote actions executed against the local Agent's authenticated IPC API. (Preview)
• Added a generateFlare action to the com.datadoghq.remoteaction.agent Private Action Runner bundle that builds a flare archive on the Agent host. (Preview)
• The Private Action Runner now supports a split deployment model in which a dedicated on-demand executor runs actions in a separate process, reachable over a local gRPC socket secured with mutual TLS.
• Added the data_security.enabled configuration flag (disabled by default) which enables the sds-result event platform forwarder used to send sensitive-data-scanner results to Datadog.
• Add the Data Security feature to scan monitored PostgreSQL databases for sensitive data, driven remotely through Remote Configuration (DATA_SECURITY_DB_SCAN_TASKS). Enabled with data_security.enabled and shared_library_check.enabled .
• Agent Cloud Auth (delegated authentication / Workload Identity Federation) on AWS now resolves credentials from EKS IRSA, ECS task roles, EKS Pod Identity and EC2 IMDS in the trace-agent, standalone DogStatsD, private action runner, IoT Agent and Heroku Agent. Previously only flavors built with the ec2 build tag (main Agent, Cluster Agent, process-agent, security-agent, system-probe, installer) supported those credential sources; the others silently disabled the feature. Most notably the trace-agent is now covered, so APM no longer requires a statically configured api_key when Cloud Auth is in use. OpenTelemetry Collector (DDOT / otel-agent ) is not covered: it does not load the delegated authentication component, and still requires a statically configured api_key .
• Setting delegated_auth.aws.region without delegated_auth.provider no longer skips cloud provider auto-detection. The configured region is now applied to the auto-detected provider, as intended, instead of being treated as an explicit provider configuration.

Teams use AI coding agents and Bits Chat to troubleshoot systems and handle complex tasks, often uncovering repetitive work worth automating. But turning those routines into workflows can still require switching tools and recreating context manually.

Through the Datadog MCP Server, Workflow Automation now lets you build workflows from Bits Chat or AI coding agents like Claude Code, Cursor, and Codex. These agents can use context from Datadog, your development environment, and the Datadog Action Catalog to build, run, and debug workflows where you already work.

In this post, we’ll show how you can:

• Create workflows from Bits Chat across Datadog
• Build, run, and debug workflows from AI coding agents
• Create and update workflows from Slack

Create workflows from Bits Chat across Datadog

Repeated operational processes often become apparent during an investigation. For example, an API gateway error spike may send your team through the same sequence each time. Your team might check recent deployments, inspect traces and downstream services, determine whether remediation is safe, and escalate when it is not.

With Workflow Automation and Bits Chat, you can turn those repeated steps into an automated workflow from the Datadog context where you troubleshoot the error. From the dashboard where you are investigating the spike, you can ask Bits Chat to build a workflow that runs whenever the relevant monitor alerts and describe your response steps.

The workflow can first check whether remediation was recently attempted for the same failure. If so, it can escalate instead of retrying. Otherwise, Bits Investigation can examine recent deployments, error signatures, traces, and downstream service health to determine the likely root cause of the issue.

For a known failure, the workflow can run the appropriate remediation. If the investigation points to a code change, a custom agent can start a dedicated code session and open a pull request with a fix for review. If the cause is downstream or unclear, it can escalate to the owning team with the investigation findings attached.

Turn an investigation into a repeatable response

Bits Chat enables you to turn a repeated operational task into a workflow at the exact moment you recognize the need for automation. This same experience is available across Datadog, including Datadog Monitors, Incident Management, and Service Level Objectives. And because Bits Chat is context-aware across your Datadog usage, it can suggest multi-step tasks to automate as workflows.

Bits Chat can also help you execute and maintain workflows after you create them. If a run fails, the Fix with AI option can inspect the execution, diagnose the issue, and help you update the workflow.

Build, run, and debug workflows from AI coding agents

Your AI coding agent can also build a Datadog workflow without forcing you to leave that coding agent environment. Suppose you are using Claude Code to debug a frequent type of error spike and suspect a particular deployment caused the spike. You could ask Claude Code to build a workflow that automates a rollback in this scenario.

Claude Code can build a workflow that first uses a custom Bits Agent Builder triage agent to check whether the deployment is correlated with the error spike. If the check confirms the relationship, the workflow can run a rollback deployment, wait for the associated service to stabilize, and check error rate and latency again to verify recovery. If it doesn’t confirm the relationship, the workflow can trigger an investigation using a custom agent across traces and downstream service health. After the custom agent finds the likely root cause, it can then escalate to the owning team with the findings attached.

Build on existing Datadog context

Like Bits Chat, Claude Code or another coding agent can reference your existing monitors, Datadog workflows, integrations, and service metadata while generating the automation. For example, it can use a service’s existing monitors or tags to determine which signals should be checked after a rollback instead of requiring you to reconstruct that context.

You can then refine the workflow conversationally before publishing it. For example, you could ask Claude Code to change the recovery window, add a notification on either outcome, or adjust the conditions under which rollback is allowed. When the workflow is ready, the coding agent will validate its specification and return a link for review in Datadog.

Execute and debug the workflow from Claude Code

Once the rollback workflow is published, Claude Code can execute it and inspect the resulting run. If the recovery check fails, for example, the agent can inspect the execution data to determine which step needs adjustment. It might extend the wait period if metrics have not stabilized yet, or narrow the investigation window if unrelated deployments are being considered. Claude Code can then update the workflow and validate the revised specification before you run it again.

Use workflows as tools for custom agents

Custom AI agents built with Bits Agent Builder can execute this same rollback workflow or create new workflows. For example, when a monitor flags checkout failures on the web-store service, an org-wide incident response agent can find and confirm the right workflow to run and invoke it. The workflow performs its own correlation check, runs the configured rollback when appropriate, and verifies recovery afterward.

The result gives the agent concrete information to act on. If the service recovers, it can report the resolution. If the service remains degraded, it can escalate the incident instead. This lets the same workflow support both human-initiated and agent-initiated remediation while keeping the recovery check in just one place.

Create and update workflows from Slack

Automation opportunities can also emerge during team conversations. During an incident, responders may agree that a particular alert should always trigger an investigation, notify the owning team, or start another response step.

Instead of recreating that requirement later in Workflow Automation, you can mention the Datadog app in Slack and describe the workflow from the conversation. The Datadog app can use that conversation as context to help create or update the relevant workflow.

Build workflows with the context behind them

Datadog Workflow Automation lets you build workflows where you identify the need for automation. Bits Chat can use context from across Datadog, while AI coding agents can draw on context from your development environment.

To start building workflows, see the Workflow Automation documentation and explore the Datadog Action Catalog. You can also learn more about our new tools in the Datadog MCP Server setup documentation and create custom agents for your workflows with the Bits Agent Builder documentation.

And if you don’t have a Datadog account, sign up for a 14-day free trial to build operational workflows with Datadog.

CUPED (Controlled-experiment Using Pre-Experiment Data) is a powerful tool that can reduce metric variance and help teams obtain precise experiment results with less data. However, the difference between an experiment’s CUPED-adjusted lift and raw lift can be difficult to explain, especially when an experiment uses many pre-exposure metrics and subject properties.

The CUPED adjustments visualization in Datadog Experiments breaks the difference into a sequence of specific adjustments. It connects each metric lookback or assignment property to an upward or downward adjustment in your estimated lift, helping you understand how CUPED made adjustments and which covariates had the greatest influence.

In this post, we’ll show how to:

• Understand why CUPED changes your lift estimate
• Trace each adjustment in the CUPED waterfall
• Interpret CUPED results with more context

Understand why CUPED changes your lift estimate

Standard experiment analysis calculates lift from metric values observed after subjects enter the experiment. CUPED augments this raw treatment-control difference with adjustments based on information captured before exposure. This is how CUPED reduces variance and produces more precise confidence intervals. But it also changes the point estimate of lift. Sometimes this difference in estimates leads to confusion for experimenters, especially since all the adjustments happen in a black box.

This can be especially tricky to understand because both metrics and subject attributes contribute adjustments. Without a decomposition, the final adjusted lift hides which of these terms changed the estimate and in which direction. The difference between raw and CUPED-adjusted lift is expected behavior, but experimenters still need enough context to determine whether the result matches their understanding of the experiment population.

Trace each adjustment in the CUPED waterfall

The CUPED adjustments visualization lets you move from a summary result to the individual statistical adjustments behind it.

From an experiment’s Metrics table, open the overflow menu for a CUPED-enabled metric and select View CUPED adjustments. Datadog opens a metric-specific view containing the raw result, the adjusted result, the estimated reduction in run time, and a waterfall of the covariate contributions.

This entry point lets you investigate a surprising result without leaving the experiment analysis workflow. You can open a separate visualization for each metric and treatment comparison that you want to examine.

The top of the visualization summarizes the non-CUPED relative lift, the CUPED-adjusted relative lift, and the estimated reduction in experiment run time. The waterfall starts with the raw lift and applies each covariate adjustment in sequence. Every step represents either a pre-exposure metric lookback or an assignment property. The step’s label indicates how much it raises or lowers relative lift, and the final bar shows the CUPED-adjusted result.

In this example, the Purchase Prediction covariate lowers relative lift by approximately 2.8 percentage points. Device Type, Revenue Lookback, Channel, User Persona, and Country make smaller adjustments. The visualization makes it clear that the gap between the two lift estimates is being driven primarily by one covariate rather than by the accumulation of several similarly sized adjustments.

A large assignment-property adjustment may reflect a chance imbalance in the types of subjects assigned to each variant, while a large metric-lookback adjustment may reflect different levels of pre-exposure activity. These contributions do not imply that a covariate caused the treatment effect. They show how CUPED accounted for the observed imbalance when calculating the adjusted estimate.

Interpret CUPED results with more context

The CUPED adjustments visualization makes adjusted lift easier to interpret by connecting the gap from raw lift to the covariates responsible for each statistical adjustment.

Use the visualization when an adjusted result looks surprising, when you need to explain an estimated reduction in experiment run time, or when you want to identify which pre-exposure differences had the greatest effect on the estimate. For details about how Datadog builds covariates, selects adjustment coefficients, and applies CUPED to different metric types, read the Datadog CUPED documentation and the Datadog Experiments documentation.

If you don’t have a Datadog account, sign up for a 14-day free trial to interpret experiment lift with per-covariate CUPED adjustments.