Eppo Release Notes

Successful acquisitions are based on what makes sense for customers. For Eppo’s experimentation customers, there was no better fit than the world’s leading observability platform in Datadog, where product development teams already turn to understand the "why" behind application issues.
Since the deal closed, we’ve been building observability-native feature flagging and a next-gen approach to experiment diagnostics and deep-dives. Building these new products has given us a unique opportunity to build upon what we learned at Eppo and make improvements that would be hard to do in the confines of our "v1". Today, we’re focused on solving what we consider to be the three biggest problems in experimentation.

Catch Errors Sooner

The first problem is that, for all of the advanced statistics available to speed up testing, most of the time lost when running experiments is due to more mundane issues (i.e. bugs) that require discarding results and re-starting. It’s a pain to realize after a few days of runtime that the experiment is broken on an old version of Internet Explorer, there’s a UX degradation on a specific viewport size, or a key instrumentation event is missing on Android. These stop-starts end up taking as much time as the experiment itself.
Datadog Data Observability, for instance, gives us the ability to detect metric instrumentation issues upstream before experiment runtime is wasted

Automate Canary Testing at Scale

The second problem is that commercial tools lack first class support for canary release testing. Experimentation discourse usually focuses on scientific thinking and finding wins, but the largest experimentation programs in the world like Netflix, Uber, or Microsoft are built on a foundation of comprehensive canary testing. By automatically turning code release workflows into randomized experiments, engineers build experimentation muscles and learn statistical thinking. The problem for most companies is that executing canaries in this way is too manual and requires engineers to babysit every release (often leading them to navigate to Datadog or similar products).
Our new Datadog Feature Flags add first-class support for standardizing, automating, and monitoring canary tests

Deep Dive with all of your Data

The third problem is that the amount of insight and intelligence generated per experiment is still lower than it could be. The "state of the art" approach to experiment deep-dives often looks like slicing experiment results by every possible segment, looking for a smoking gun explanation of why the experiment isn’t successful. Hopefully, these teams are at least aware that their "multiple testing" is bound to lead to plenty of false positives. But even when teams correctly identify an underperforming segment, they still need to root-cause to the specific issue, whether it be confusing UX, poorly-aligned personalization, or something less apparent like application slowness.
Datadog Experimentation combines our best-in-class warehouse native approach with other data sources like application vitals or Datadog Real User Monitoring

Partnering with Datadog immediately gave us tools for solving each of these top problems in experimentation. Real-time observability metrics helped us leapfrog into an exciting new suite of experiment diagnostics, so engineers can catch product bugs and experiment issues right as they turn on the test. Statistical canary testing can now be automated based on errors, infra metrics, and product telemetry. And stay tuned for a new spin on experiment deep-dives that weaves the universe of data that Eppo and Datadog bring together, including warehouse metrics, behavioral events, and application vitals.
The Eppo team (and me as CEO) are still 100% focused on experimentation, and Datadog brings a wealth of experience of enterprise support. Today, we’re excited to share that interested teams can request access to our new Datadog Feature Flags as part of Datadog’s Product Preview program.
We’ll be sharing a lot more of what we’re building this fall, including in-person at EXL, MIT CODE, and AWS re:invent. More to come soon!

• Che & the Eppo team

Eppo Is Now Part of Datadog!

This acquisition will drive learning velocity for all Eppo customers, equipping them to expand experimentation across their organizations

Eppo's Founder and CEO, former early data scientist who built experimentation tools and cultures at Airbnb and Webflow

Eppo Is Now Part of Datadog!
I’m excited to announce that Eppo has been acquired by Datadog. This acquisition will drive learning velocity for all Eppo customers, equipping them to expand experimentation across their organizations. With visibility into how any design, product, or technology change impacts the customer experience, teams will be able to connect engineering work to business outcomes—breaking down silos and enabling a culture of velocity and accountability.
There’s much more to come, but I wanted to start by sharing why we’ve taken this step.

Eppo’s Approach

Eppo has always been about learning velocity, with a theory that the companies that learn the fastest, win. This principle is embodied by experiments, ranging from canary testing to A/B testing and automated solutions like bandits. The more experiments a company runs, the higher the company’s learning velocity tends to be.
But there’s nuance in how we entered the experimentation market. Eppo’s vision started from an opinionated take on which experimentation cultures were getting it right, where data-driven behaviors were actually driving business value instead of spinning statistical theater. Summarized, this led us to believe experimentation should be:

• Broadly accessible
• Highly trustworthy
• Used across all important business levers
  Experimentation vendors new and old were failing this test, whether due to outdated worldviews on data teams, a lack of support for critical testing use cases, or a lack of empathy for diverse company contexts. We felt that a modern approach to warehouse metrics, a UX approach rooted in bringing various skillsets together, and a focus on bringing the frontier of statistical methodology to the masses would solve these problems.

Why Datadog

Joining with Datadog offers three major opportunities to move the industry forward into the next era of experimentation and learning velocity. These opportunities are rooted in the same three principles.

• First, we can accelerate Datadog’s movement to becoming more collaborative by making products that are broadly accessible. Datadog began with a focus on engineers, but it has made major strides with product and data teams with features like Product Analytics, Session Replay, and Data Observability. Eppo is excited to add to our DNA of PM-friendly UX, statistical rigor, and warehouse metric management to Datadog’s existing products.
• Second, a combined Datadog and Eppo can make learnings highly trustworthy, with an added pinch of at the fastest speed. With our combined technology, we can strike the right balance of precision and latency. Experimentation needs the most precise data sourced from the warehouse, and it is okay to have higher latency to ensure even 1% impact readings are correct. Product analytics, however, benefit from fluid, real-time event data that can explore at the speed of curiosity. In this category, raw, uncleaned data is fine for detecting large 50% differences. Eppo is a market leader in warehouse-centric composable metrics, and Datadog’s bread and butter is real-time event processing.
• Third, we can drive learning velocity across multiple high-value problems. For example, a cousin of the A/B test is the controlled feature rollout via canary testing. Datadog’s real-time observability with Eppo’s flags and stats engine means that a true end-to-end canary test solution will finally be on the market. Datadog’s AI observability with Eppo’s contextual bandits will allow AI teams to ensemble gen AI foundational models in a state of continuous testing and rebalancing. Stay tuned to learn what is possible with our combined technology, and how we might make multiple Datadog products more statistical.
  And of course, Datadog’s scale means that we can drive experiment velocity across 30,000+ businesses across the world. We’ve already proven that Eppo can drive success at 20-person startups and Fortune 10s, from decades-old retail companies to explosive AI natives. This is an opportunity to set a new de facto standard of experimental building practices across the globe.

Positioned for Agentic Product Development

Eppo was founded in 2021, a time that feels like the Stone Age since gen AI models have become mainstream. As we wrote in our AI manifesto nearly two years ago, the process of deriving ideas, implementing ideas, and measuring idea success has been turbocharged. Eppo has benefited here, with thousands of AI-driven experiments running through Eppo servers each year.
As sci-fi as our AI manifesto felt at the time, our imagination still fell short in one respect. We originally envisioned a human-in-the-loop process of tech workers implementing faster, better. But with the rise of AI agents, it has become clear that some types of product development will become fully closed-loop. Instead of engineers bussing tickets through a queue, AI agents can identify an issue, find its root cause, and implement a fix. And with flags and experiments, the fixes can be safely rolled out with all appropriate metrics measured statistically.

What Happens Next?

Eppo customers will continue to receive the same experience, support, and services. Users will still log in to Eppo, now branded as Eppo by Datadog.
We’ll be building a new vision for experiment-centric product development that combines Datadog’s expertise in observability with Eppo’s expertise in experimentation. We’ll be working closely with the Datadog teams building Product Analytics, Real User Monitoring, and Data Observability.
The goal is to drive learning velocity for our customers, based on the belief that the companies that learn the fastest tend to win. We can accelerate our vision by getting all ideas tested, shipped, and celebrated across Datadog’s 30,000+ customers.

A Leap Ahead with More to Come

Joining forces with Datadog is the quickest path to the future of product development, one that’s more resilient, scientific, and agentic. We’re excited to join a thriving Datadog product culture, and Eppo customers can immediately take advantage of a broader suite that includes Product Analytics, Session Replay, Data Observability, and Application Performance Monitoring. The Eppo team is committed to building as part of Datadog for years—stay tuned for much more to come!

• Che & The Eppo Team

As a Product Manager, every quarter you’re given a goal and you’re responsible for producing a roadmap that meets that target. You and your team put your best ideas together, and you have some confidence because they’re good ideas founded upon user research, but you still don’t really know if you’ll quantitatively hit your number.

Welcome to the life of a modern product manager. It’s a role of gathering inputs, finding creative solutions, technical understanding, strategic prioritization, and a fair bit of persuasion. When experimentation, proven to be one of the most effective ways to achieve data-driven growth, is layered on top, there can be more pressure in how you’re judged based on the impact your team has made. That’s where Experiment Forecasting steps in.

Experiment Forecasting is designed to relieve some of your stress, giving you clarity, control, and confidence as you tackle conflicting demands.

Why Experiment Planning Matters

Experiments are integral to shipping high-quality features that move the needle, but they come with unique challenges. If you’ve experienced any of the following, you’re not alone:

• The “Just Ship It” Urgency: There’s constant pressure to release features quickly, often long before the data from experiments has come in. Do you push something unproven out the door or risk annoying stakeholders by advocating for patience?
• Engineering Bottlenecks: Maybe you’ve jotted down the perfect experiment idea but hit a wall waiting for resources to run the test. Now, it’s a scramble to keep progress going.
• Roadmap Whiplash: Plans change; sometimes slowly, other times at warp speed. Roadmaps can quickly morph into tangled webs that stress everyone involved.
• The “Always Ship More” Mandate: Whether it’s internal KPIs or external competition, there’s immense pressure to keep feature velocity high.

Experiment planning and forecasting allow you to zoom out and solve these issues systematically, bridging the gap between stakeholder expectations, engineering constraints, and the actual impact your team can deliver.

What Is Experiment Forecasting?

Experiment Forecasting is an approach that brings data-driven clarity to your product experimentation strategy. Instead of making wild guesses or operating on gut feel, it offers data-driven predictions. With tools like Eppo’s new Experiment Forecasting feature, you can chart out your future experiments on a timeline, project their potential impact, and assess if your quarterly roadmap is likely to hit your goals. Take control of your roadmap instead of letting it control you.

How Experiment Forecasting Helps You Win

Experiment forecasting changes how Product Managers approach planning and achieve their metric goals. It provides practical insights to create strategies, align teams, and make experiments more effective. Here's how it makes a difference:

• 1. Map Out Your Best Ideas and Gain Strategic Clarity
     Experiment forecasting allows you to measure potential impact against your metrics. By using historical data, you can estimate outcomes and determine if your plans will drive progress or need adjustments. This insight helps you make thoughtful decisions with confidence.
• 1. Spot Gaps and Add Experiments as Needed
     Forecasting helps you assess whether your roadmap includes enough experiments or if you need higher-impact initiatives to meet your goals. These insights enable you to adjust plans proactively rather than play catch-up later.
• 1. Communicate Expectations Clearly to Stakeholders
     Experiment forecasting helps you craft a clear, data-driven narrative for stakeholders. Instead of presenting just a list of projects, you can share forecasts that show how your experiments align with hitting company goals. Quantitative estimates build trust and make setting realistic expectations with leadership and your team easier.
• 1. Jump-Start Collaboration with Eppo’s Experiment Drafts
     With Eppo, you can draft your roadmap experiments, complete with key details like metrics, entry points, and projected impact. The timeline view helps place experiments visually, showing clearly how they’ll fit into your quarter. This makes cross-functional collaboration easier and ensures that, when the quarter begins, your plans are ready and endorsed by your team.

Experiment forecasting improves planning and lays out a path to measurable success. With tools like Eppo, your experiments and strategies have the clarity and alignment needed to drive results consistently.

Experiment Forecasting in Action

Here’s how a quarterly planning session might feel with Experiment Forecasting in place versus without it.

Without Forecasting

You’re creating a roadmap that’s ambitious but lacks clarity. You feel a pit in your stomach because you think these experiments will hit the metric targets, but you’re not positive. Your team plans to evaluate on the fly, knowing you’ll probably need to pivot halfway through.

With Forecasting

You sit down to map out 10 experiments for the quarter. For each, you know its primary metric, the entry point, and the forecasted impact based on past win rates. This lets you objectively benchmark your team’s goals against reality and adjust beforehand. You also clearly communicate key dependencies (engineering and data science needs, launch schedules, etc.) so everyone is aligned before the roadmap even kicks off.

Eliminating guesswork and unpredictability will help you and your team start the quarter with more confidence, alignment, and trust.

Balancing Strategy and Execution

Experiment Forecasting doesn’t just help you plan; it frees you to execute on real impact. It’s one more way to balance the art of product strategy with the science of experimentation while navigating the daily pressures you face. Our goal? To make your job as a PM just a little easier and more effective.

Want to see it in action? Learn more about Eppo’s Experiment Forecasting and how it can fit seamlessly into your product development cycle. With the right tools, the next quarter might just be your most productive yet.

Marketing thrives on making quick, data-driven decisions. Yet, for many marketers, web A/B testing has been a source of frustration with flickering pages, broken elements, and the endless headaches of visual editors. What if you could achieve a seamless experience of engineering-level precision without actually needing engineering?

Now, you can do it with the Eppo-Contentful integration. Teams can now craft and manage content directly within their CMS, rather than relying on clumsy tools to build A/B test changes. Want to run an experiment? Simply flip a switch to turn your new changes into a test variation with no engineering roadblocks and zero stress.

Here’s how it works. The Eppo-Contentful integration requires a small one-time engineering setup. After that, teams use the entry ID of any piece of content in Contentful to define experimental variations, creating a powerful, scalable workflow. By combining two best-in-class tools, this integration unleashes no-code experimentation, empowering marketers to test and iterate without limits.

The Challenges Marketers Face

Experimentation is every great marketer's superpower. Testing new ideas and understanding what works can amplify impact, but common hurdles make it harder to capitalize on this potential:

• Engineering Dependencies: Traditional A/B testing demands engineering support to set up feature flags, modify codebases, and manage backend workflows. Waiting on these resources slows progress.
• Tool Overload: Managing experiments across multiple platforms creates unnecessary complexity and increases the risk of errors.
• Performance Hits: Client-side scripts often cause flickering or slowdowns, disrupting user experience and compromising data accuracy.

These roadblocks can leave marketers feeling stuck, unable to iterate quickly or confidently prove results. That's where the Eppo-Contentful integration comes in.

Why Eppo + Contentful Is a Game-Changer

This integration transforms the way you run experiments. Bringing Eppo's powerful experimentation capabilities directly within Contentful's headless CMS eliminates the need for constant engineering support and keeps everything streamlined in one platform. With everything served from the server, you can test hero images, copy, or even entire layouts seamlessly, all without sacrificing performance or user experience.

What Sets It Apart?

Unlike other solutions, the Eppo-Contentful integration avoids the challenges of performance-hurting client-side scripts. Instead, it relies on a headless CMS with feature flagging to keep your site fast and reliable.
Its benefits include:

• Zero Reliance on Engineering Teams: Set up and run experiments entirely from Contentful's interface without waiting on backend work or sprints.
• Top-Notch Performance: Forget page flickers. Your experiments are embedded directly into your site without slowing it down.
• Effortless Setup: Installing Eppo's Contentful app is as easy as a few clicks, and its intuitive design makes experimentation accessible for marketers of all tech skill levels.

Here's How It Works

Getting started is easy and designed with marketers in mind:

1. Install the app from the Contentful Marketplace and input your Eppo API key.
2. Configure content types you want to test: blog posts, hero images, or banners.
3. Create content variations, such as alternate CTAs, colors, or headlines.
4. Publish and automate via Contentful, where Eppo automatically generates the necessary feature flags in its system.
5. Enable and Go Live with a single click in Eppo, bringing your experiments to life.
6. Track results using Eppo's analytics to find the winning variation backed by complex data.
   This happens in a single workflow, without switching between tools or tedious handoffs.

Real-World Use Cases

Picture these scenarios where the Eppo-Contentful integration solves real marketing challenges:

1. Optimize Landing Page Conversions
   You're launching a major product campaign and want to test different hero images. With the integration, you can create variations directly in Contentful, publish experiments, and identify the winning design while keeping load times blazing fast. There is no need to involve engineering at any point.
2. Maximize Blog Engagement
   Are you struggling with bounce rates? Experiment with blog post headlines or introductory text to see what keeps readers engaged longer. Using this integration, changes can be easily made and tracked across thousands of posts, saving countless hours of manual updates.
3. Test Different Offers for Targeted Audiences
   Want to test a new discount but don’t want users randomly seeing different prices as they go from mobile to desktop? Eppo supports regionally-randomized experiments, called Clustered Experiments. You can implement them via Contentful and analyze them natively. With this method, every user in the test regions will see the same price, but when it comes time to analyze the results, you can compare results user-to-user, accounting for regional variations. Use the integration to tailor messages like banners or CTAs for specific locations. Testing multiple designs or texts doesn't require new code, just a few configurations in Contentful.
4. Refine Call-to-Actions
   Eppo makes it easy to experiment with CTAs like "Get Started" vs. "Learn More" or even entire boxes with varying designs and copy. Consistent across pages but customizable where needed, you can improve clicks and conversions with actionable insights.

Smarter Marketing Starts Here

Great marketing isn't just about big ideas; it's about testing, learning, and iterating. The Eppo-Contentful integration allows you to experiment boldly without compromising speed or site performance, no more waiting or bottlenecks.
Here's why you should try it:

• Launch experiments faster than ever.
• Save time and resources by removing engineering dependencies.
• Make every decision data-driven with robust analytics.
• Scale effortlessly, whether you're managing 10 pages or 10,000.

Start Experimenting Today

Don't wait to revolutionize your marketing approach. Install the Eppo app in Contentful now and see the difference firsthand. Visit the Contentful Marketplace, connect your Eppo API, and create experiments in minutes.

Use gold standard incrementality testing to evaluate the true contribution of marketing to your company's bottom line

Before Eppo, Greg was the CEO of Tech for Campaigns where he led large consumer advertising campaigns and experimentation programs

Learn more about Eppo Geolift at our webinar on November 14th with Bryce Casavant, Senior Data Scientist at WHOOP and Eppo's Marketing Experimentation Team

How do I know where to put my marketing dollars when my data warehouse, advertising partners, and web analytics don’t agree? Privacy changes and black box ad platforms make it harder than ever to find trustworthy marketing signals at the same time that the bar on marketing ROI rises even higher.

As Che wrote in our Series B announcement, it’s never been a greater risk to not know which company initiatives are growth levers versus “low-ROI money pits.” Within marketing, “incrementality testing” is the gold standard to evaluate the true contribution of marketing to a company’s bottom line. A key incrementality strategy is geo-testing or “matched market” tests, where a certain set of regions receives different marketing programs than another, and the differences are evaluated. These tests provide real insights but require specialized statistical knowledge, dedicated operational expertise, and can be tricky to align with a company’s business metrics.

Enter Eppo Geolift: empowering marketers and data teams with the tools they need to make rigorous marketing investment decisions with ease and confidence. Eppo Geolift’s quasi-experimental methodology measures significant changes in your business metrics across geographies using cutting-edge Synthetic Control Methods. Critically, Eppo Geolift uses the same integrations and metric definitions used for lifecycle marketing, product, and AI experiments within Eppo, providing a common language across teams with one place to see all growth initiatives.

Share Geolift with your marketing team or data team

Simple Setup

Without help, launching a geolift program can be complex. Marketers first need to run calculations for which geos to select, for how long to run the test, and at what spend levels the experiment should run. Then, when it comes time to launch, many marketers feel like they’re stepping into a void where six or seven figures can be at stake and hoping to avoid landmines in the process (like turning off Branded Search in your target geos but finding out after that PMax ended up buying all the same keywords!)

To solve this, Eppo guides users through the process of designing a test, launching it, and delivering easy-to-read results. Additionally, Eppo's Protocols offer presets that incorporate statistical best practices and organizational guidelines. Protocols let data scientists empower technical marketers and data analysts to confidently run tests without full-time assistance, enabling teams to move quickly and scale testing.

Always-On Testing for Digital Platforms

Once a program gets off the ground, geolift vets know success lies in maximizing the number of learning opportunities throughout the year. More tests mean understanding Return on Ad Spend (ROAS) at different spend levels, performance across creative types, and time varying factors like seasonality and adstock/decay effects. Eppo Geolift offers automated testing, monitoring, and analysis for major digital platforms including Meta, Google, TikTok, and Pinterest. Our advanced causal inference methodologies minimize holdouts, allowing you to run experiments with minimal disruption to your ongoing campaigns.

Powerful Custom Experiments

Brand buys, inventive programs – Eppo Geolift also offers custom experiments for use cases like measuring offline media or non-geographic initiatives (like search engine optimization). Eppo provides data scientists input into the assumptions used in our power calculator, explicit control over the creation of the synthetic control, and complete flexibility to use one-off metrics as needed.

The Power of Real Business Metrics

Many incrementality options depend either on black-box vendor methodologies or sending your data to an outside third party via API, Google Sheets, or S3 buckets. The external data transfers often mean that the results don’t perfectly align with the business metrics that finance is using.

Eppo Geolift integrates directly with Snowflake, BigQuery, Redshift, and Databricks. You're working with the same trusted metrics your CFO uses—no more discrepancies between lift reports and financial realities. Got a question about a metric? Double-click into the SQL query that drives it.

The Broader Eppo Platform

Geolift is built upon the same platform that our lifecycle marketing and product experiments are built on: easy-to-use design tools, experiment calendaring and sequencing, centralized and vetted metrics definitions, easy report building, and a knowledge base to make sure what’s learned is retained.

Let's get started

Eppo Geolift empowers marketers and data teams to continually optimize their marketing investments for their true business contributions – working off the same metrics as the entire business.

Interested in Geolift? Eppo customers can start designing experiments with us immediately. If you’re not already integrated, we’d love to chat!

Good product teams ship quickly and drive impact. Bad product teams move slowly or they have bad experimentation hygiene, making impact reporting untrustworthy. It can hard for product leaders to discern one type of team from the other. Because there’s so much going on, these leaders are hearing a lot of success stories but they lack the means to verify that teams are operating well.

We built the Performance Scorecard to solve these problems. It is designed to bridge the gap between individual experiments and overall program performance, and ultimately impact. This provides a comprehensive view of your experimentation efforts.

The Performance Scorecard goes beyond traditional metrics by measuring both inputs (how you're experimenting) and outputs (what you're achieving). This dual approach ensures that you can see whether teams are operating in ways that lead to velocity and impact.

Aggregate Impact: Quantifying Your Program's Success

One of the most significant challenges of experimentation programs is demonstrating their overall impact. Previously, teams had to manually compile data every quarter during planning sessions, often relying on a handful of standout experiments to make their case.

As a PM, every quarterly planning cycle, I was compiling the same table of results. I needed to aggregate what experiments we ran and how they impacted our north star metric in front of leadership. But leadership wasn't in the weeds, and this data always seemed new to them. At planning time, this data couldn't break through prior assumptions already formed.

The Performance Scorecard changes this by providing:

• 1. Holdout-based Aggregate Impact: A robust measure of your program's total impact, based on rigorous holdout experiments.
• 1. Bayesian Aggregate Impact Estimate: A new feature that allows teams without holdout capabilities to estimate their overall impact accurately.
• 1. Top experiments: A table view of top experiments for the expected measure shipped in the timeframe specified. With these tools, you can now answer the critical questions: "What is the total value our experimentation program has delivered?" and "What experiments impact our core metrics?" This data is invaluable for securing continued support and resources from leadership. Best of all, this view is always available, making it easy to share on a consistent basis, and not just at planning time. This means you don’t just get to report on success driving impact, you get to celebrate it!

Velocity: Aligning Your Organization on Input Metrics

While outcomes are crucial, the Performance Scorecard also focuses on the inputs that drive those results. Experimentation velocity is one of those key inputs.

Once we decided on a quarterly goal, as the PM I would always advocate for loading up the team’s roadmap with a number of projects I believed would move that metric, with the one I believed would have the biggest impact first. Even if that project failed, the team would have more shots on goal before the quarter was over to move that metric.

With the Performance Scorecard, leaders can track if their teams are also taking enough shots on goals. By tracking experiment velocity, you can:

• Set and monitor organization-wide goals (e.g., running 10 tests per quarter)
• Identify teams that may need additional support or resources
• Encourage a culture of continuous experimentation across your organization

Quality: Ensuring Rigorous Experimentation Practices

The quality of your experiments is just as important an input as their quantity. The Performance Scorecard helps you maintain high standards by tracking key quality metrics. This feature addresses common misconceptions and ensures that your organization is following best practices.

For example, I worked for a leader who suggested running experiments with a 20% control and 80% variant split to get new features to users faster. While well intentioned, this actually led to slower experiments and lagged velocity, as a 50%/50% split gives much more signal and ultimately a faster decision.

The Quality section of the scorecard helps you:

• Monitor experiment design parameters across your organization
• Identify and address potential issues before they impact your results
• Educate stakeholders on the importance of rigorous experimental design

Win-Rate: Testing the Right Ideas

According to research, the success rate of experiments ranges from 8% to 33%. Over a large sample of experiments, this is what most teams should be. Yes we see many teams that fall outside this range.

If a team’s win rate is below what is expected, that indicates that the hypotheses aren’t good enough, and perhaps not founded in good customer insights. On the other end, if a team’s win rate is above this range, the experiments usually don’t have enough impact. These are easy and small ideas that win but don’t move the needle enough to achieve goals set.

With the Win-rate section of the scorecard you can:

• Understand how team win-rate compares to industry benchmarks
• Celebrate teams that are shipping a mix of successful, neutral, and unsuccessful experiments
• Investigate if a team has unusually high or low win-rates

Empowering Program Leaders and Winning Executive Buy-In

The Performance Scorecard is more than just a reporting tool – it's a catalyst for building a true culture of experimentation. By providing program leaders with a comprehensive view of their experimentation efforts, we're enabling them to:

• 1. Evaluate performance across teams and over time
• 1. Identify areas for improvement and optimization
• 1. Demonstrate the value of experimentation to skeptical stakeholders
• 1. Make data-driven decisions about resource allocation and program direction

For executives, the Scorecard offers a clear, holistic view of the experimentation program's impact on the business. This transparency fosters trust and encourages continued investment in data-driven decision-making.

Transforming Experimentation from a Tool to a Strategy

With the introduction of the Performance Scorecard, Eppo is taking experimentation to the next level. We're moving beyond individual tests to create a comprehensive system for measuring, monitoring, and optimizing your entire experimentation program.

By providing insights into aggregate impact, velocity, and quality, the Performance Scorecard empowers you to build a more effective, more efficient, and more impactful experimentation culture. It's not just about running tests anymore – it's about transforming experimentation into a core strategic advantage for your business.

Ready to elevate your experimentation program? Contact us today to learn more about the Performance Scorecard and how it can drive growth for your organization.

Why we raised our Series B, and why experimentation is more important than ever in an era of AI and efficient growth

Eppo's Founder and CEO, former early data scientist who built experimentation tools and cultures at Airbnb and Webflow

Today, Eppo is announcing our $28M series B financing, led by Davis Treybig at Innovation Endeavors with participation from Preeti Rathi at Icon Ventures.

Since our last funding, Eppo has become synonymous with large experimentation ambitions. Category-leading companies like Twitch, DraftKings, and Coinbase use Eppo to supercharge their experimentation. So do generative AI pioneers like Descript and Perplexity. Eppo customers are running experiments across their businesses with use cases spanning product, marketing, and AI.

When looking who to partner with on this round, Davis and the IE team were ultimately a no-brainer among our offers. Davis is the author of the best-researched piece on experimentation from anyone in venture capital, "The Experimentation Gap", outlining its clear connections to AI and use cases beyond digital products. We’ve been partnering with Davis and IE since our seed funding, and have been continuously impressed by their ability to build relationships with researchers on the frontiers of technology, and companies who see where this technology fits in their stacks.

Our Series B comes at an interesting inflection point for the tech industry, a new era of change where experimentation is the clear differentiator between which companies will thrive, and which will struggle.

Here’s why experimentation is more important than ever, and why we raised this Series B:

The two conversations happening in every boardroom

If there are two things every company is discussing right now, they are efficient growth and AI.

Rising interest rates and scarce capital have squeezed companies to the point that even a $500M ARR company with 30% growth trades at only 15x multiples. Teams were stripped down via layoffs, SaaS spend was consolidated, and growth venture investments have cratered. Growth is still an imperative, but must be done with scalable economics instead of growth at all costs. Put another way, it’s never been a greater existential risk to not know which company initiatives are growth levers vs. low-ROI money pits.

The second conversation is around AI. Even as companies clamp down on spend, AI budgets live outside of financial discipline. It’s for good reason: CEOs have seen astronomical increases in efficiency for CoPilot-augmented software development, creative asset development, and knowledge management. CEOs have all taken the time to imagine an AI-native competitor, and how they’d fare against them. GenAI technology presents a clear opportunity to leapfrog competition - or be leapfrogged.

Both of these factors have led to a blossoming in our experimentation space. Running experiments is the simplest path to high conviction on which products, campaigns, and AI strategies are successful, and which need to pivot or wind down. This market landscape is Darwinian: those who experiment, adapt, and swiftly refine their strategies are the ones who succeed. They accelerate their winning initiatives while quickly abandoning the ones that don't work.

Companies now expect more from experimentation

The point of experimentation is to drive velocity, growth, and innovation. Most companies aren’t there yet. They have low horsepower; bottlenecked experimentation stacks that don’t actually power velocity, growth, or innovation.

The biggest change since I started Eppo is companies demanding more from their experimentation investments.

Legacy experimentation vendors can’t deliver the vision. If a company buys a marketing-focused tool like Optimizely, they quickly figure out that the only supported tests are simple website changes. If they buy a feature flagging-centric tool like LaunchDarkly, they realize that the “experimentation” is a shallow coat of paint on a narrow DevOps tool. Teams using these tools end up stunted, spending more time and money on tedious manual efforts and expensive supplementary tools to fill the gaps.

The result is an inadequate trickle of experiments that are never quite trusted either.

In contrast, the tech giants with modern tooling are winning the era of efficient growth and AI. Instead of slowly spinning up button color tests, companies like Microsoft, Netflix, and Eppo customers are running experiments that can generate revenue and change strategic paradigms:

• An AI team saved $5M+ of spend by proving that open source LLM models could match the performance of an expensive GPT model they previously used. Now, all GenAI models are tested for ROI instead of implicitly trusted on brand name.
• At Netflix, simple UX changes are completely automated. Thousands of tests on show artwork are designed, set up, and adjudicated by algorithm.
• At Airbnb, we experimented on a sales team, holding out a random set of markets from their work and seeing if the sales-worked markets grew faster than the holdouts. The team was ultimately disbanded and reassigned, saving headcount cost and increasing strategic focus.
• A company spending tens of millions of dollars a year on YouTube ad campaigns tested whether the spend was doing anything by zeroing out the spend in a select group of geographies and comparing their performance.

In short, these companies are able to experiment pervasively, quickly, and with leadership trusting the results. They’ve built the accessibility and governance required to make any test possible and make experimentation like water: easy, continuous, expected.

Legacy tools like Optimizely or LaunchDarkly look nothing like the workflows that enable market leaders to evaluate large, expensive campaigns, or run all product development through test and learn iterations. With programs and software spend under tight budget scrutiny, the bar is now set much higher.

The age of AI will be an age of experimentation

We didn’t predict the explosion in AI capabilities at the end of 2022, but it created a massive appetite for experiments. There’s a short-term need to evaluate AI model ROI, and a long-term need to evaluate more ideas in general. AB testing is the primary solution for both.

Companies now have a firehose of new GenAI model generations at their fingertips, each reaching new heights and new, higher price tags. A GPT model release gets quickly followed by a new Claude, new Llama, and a host of open source models. As the New York Times aptly put it, AI has a measurement problem: companies have no idea which models are most accurate and provide the best user experience.

With AI capabilities in cloud APIs, the switching cost of these models is near zero. A simple feature flag can be repurposed to a routing system for AI model vendors — frictionless swaps of which API to use. This means that companies with good experiment infrastructure can get results that are more powerful and far cheaper with little effort.

Companies with better infrastructure can go further, multiplexing across an ensemble of gen AI models. Maybe pay up for a premium Claude 3.5 model for high-value users, and save money with open source models on Free tier users. There are wide disparities in price across the LLM clouds, and experimentation gives companies the edge to discern between real performance gains and spending that should be cut.

But there’s an even more interesting long-term trend. GenAI is about to exponentially increase the number of ideas generated and implemented. All of the necessary pieces to produce a new product or a new campaign concept are already levered by AI:

• AI models are great for brainstorming ideas, crowdsourced from our collective intelligence
• AI models can whip up creative assets easily, even strikingly realistic photography
• AI models can implement ideas in code, not just to multiply the output of engineers but even to enable less technical users to implement changes themselves

With just these existing capabilities there will be 10x more product implementations and 10x more marketing campaign concepts, which will all need to be evaluated before they are rolled out.

As the cost of ideas goes to zero, the cost of evaluating these ideas becomes the new bottleneck. Companies hoping to leverage this AI explosion will need experimentation infrastructure that can handle 10x more volume and use cases.

We raised $28M to create experimental companies

Winning companies thrive in an era of change with innovative experimentation. There’s a reason why Jeff Bezos talks about experimentation in every speech, why Netflix ran experiments as early as their DVD mailing days, why Mark Zuckerberg established AB testing on their growth team while still only operating at a handful of colleges. The companies that outcompeted and won their markets are highly experimental.

Our ambition is to change corporate culture everywhere, unleashing their best ideas with a broad experimental mindset. We’re excited to bring more partners and fresh funds to our mission.

Run an experiment in Eppo!

We’d love to show you what we’ve built.

Request access to Eppo and we’ll help you get a few experiments set up.

Join the team

Our team is made up of veteran product builders from Airbnb, Snowflake, Slack, Amazon, and Stitch Fix. We’re on a mission to change corporate culture everywhere. Have a look at our open jobs. We’d love to meet you.

Today, we’re excited to launch Layers. This release combines the flexibility of Eppo feature flags with the structure to easily coordinate concurrent tests.
There are scenarios where you may want to test different variations simultaneously on the same area of your product. However, allowing these experiments to overlap can lead to conflicts and degrade the user experience. Eppo's new Layers functionality solves this by enabling you to create dedicated spaces for running mutually exclusive experiments.

A More Structured Approach

Previous approaches to handling concurrent tests involved complex coordination or creating dependencies between flags. With Layers, you can easily set up a structured environment that keeps your experiments isolated while providing controls over traffic allocation and ordering of experiments.
Each Layer acts as a contained space with the following levels:

• Experiment opt-out rule - set an exclusion group here that always sees the same experience, such as internal users that always see the new experience
• Experiments - use this level to serve concurrent experiments with the ability to set the traffic exposure of each experiment
• Rollout - once an experiment concludes, you can easily select the winning variant and roll it out to all unallocated users
• Default serving rule - set the control variation users see when not exposed to an experiment or rollout

This streamlined workflow minimizes coordination overhead and ensures your experiments don't interfere with each other's results, while giving you the power to exclude users and roll out winning variations as soon as they’re selected.

Rigorous Controls, Simplified Process

Eppo applies the same powerful statistical engines and guardrails to experiments run in Layers as our core experimentation product. You'll get automated diagnostics, traffic balance monitoring, and our full suite of analysis tools to properly measure impact.
Creating a new Layer is simple - just navigate to Configuration, enter details like the Layer name and parameters, and you're ready to start adding experiments. Need to roll out a winning variation? With one click, you can update the Layer's default experience.

Utilize Parameters to Create No-code Experiences

Parameters are the core building blocks that allow you to define the specific elements you want to test variations of within a Layer.
For example, let's say you want to experiment with different messaging headlines on your product's homepage. You could create a Parameter called "headline_text" and set a default value. Then, as you set up each experiment variation within that Layer, you can specify alternative text values to test.
Parameters can accept various data types like strings, numbers, booleans etc. This flexibility enables you to test everything from UI styling and design elements to pricing variants and feature configurations - all within the same Layer construct.
Additionally, this means parameter values can drive experience changes without updating code. This no-code solution makes it easy to increase the number of variations you test and experiments you run to find the optimal solution to your hypothesis.

Experimenting Together

Whether it's a product and marketing team collaborating on a page, or parallel AI experiments, Layers enable your organization to maximize learnings through disciplined, concurrent experimentation.
Interested in using Layers? Eppo customers with Feature Flags can start leveraging this feature immediately. If you're new to Eppo, we invite you to request a demo to see how Layers can enhance your experimentation program.

Finding Eppoʼs new voice

We're excited to introduce Eppoʼs rebrand — a new look that highlights our commitment to transformative innovation, customer partnership, and inspiring the experimentation community.

Eppo was founded on the realization that every company has untapped entrepreneurial potential. Our first product was our world-class Experimentation analysis suite, which makes rigorous statistical inference accessible and automates away tedious manual analysis so that data scientists can focus on driving experimentation culture in their company.
Over the course of the last three years, Eppo has grown significantly - we now process over 5 billion daily assignments for customers, and our headcount has doubled in the last six months alone. With growth comes new ideas... and growing pains. We have expanded to new use cases and products: Feature Flagging for engineers and product teams running experiments, Contextual Bandits for personalization and lifecycle marketing integrations for marketing teams. Our language wasn't expansive enough to resonate with all our new personas.
We also know that our current customers are passionate, even fervent about our approach to rigor and customer-centricity, but it took effort to understand Eppoʼs differentiation in the market.
It was clear to us that we needed a new brand identity to encompass all of this growth and differentiation.

Built on customer beliefs

“Brand is a promise delivered." (What Great Brands Do, Denise Lee Yohn)
We treated this rebrand like any design project — our first question was “What data points should we consider?ˮ. Having led many rebrands in the past, I've seen the common trap of leaders latching onto a famous brand to copy (Apple, Nike, and Uber are the common choices), or projecting what they wish the company to be for them.
We were more interested in reflecting what our customers believed about us, rather than the other way around. We spoke to some of our most enthusiastic customers as well as respected peers and advisors in the experimentation space to understand why they chose (or appreciate) Eppo.
Here are some of the ways they described how they see Eppo in the space:

• “Eppo is built for companies that take data seriously and want to have experimentation has a cultural element. Eppo is rigorous, genuine, and fun.”
• “If Eppo were a car, theyʼd be a colorfully painted vintage Mini. It has quiet confidence, the underlying tech is solid, itʼs lovingly maintained and perfectly designed.”
  What stood out to us was:
• Our customers are transformational leaders who challenge the status quo, push boundaries, and innovate old ways of working.
• Eppo delivers a new way of running a business: one where more ideas are executed, learning velocity is accelerated, and teams operate with agility and efficiency.
  Lastly, we wanted to reflect not just what our promise is but also how it is delivered. We know that just having a tool like Eppo wonʼt create cultural change.
  Our key advantage is the wealth of experimentation expertise at Eppo and the diversity of companies we built it from — and we personally bring this to our customers daily.
  We are close partners with our customers and community, providing guidance and enabling them to make confident decisions and rallying their organization around this new way of working.

Helping transformational leaders win

The new Eppo brand brings the spirit of athleticism to the challenge of transforming a business. It's about the aspiration to win — and in this new age of constant change, the experimenters will win. We're here to give them not just the tools and guidance, but the inspiration and confidence to succeed.
The new Eppo brand brings the spirit of athleticism to the challenge of transforming a business. It's about the aspiration to win — and in this new age of constant change, the experimenters will win. We're here to give them not just the tools and guidance, but the inspiration and confidence to succeed.
Visually, we drew inspiration from classic explorations and games, most notably Conway's Game of Life and FEZ. They both exude dynamism, a sense of precise engineering, and the constant creation of new perspectives via transformations.
‎‎
‎Our new logo nods to that constantly evolving spirit of experimentation, and to the “precisely engineeredˮ nature of Eppo that our customers highlighted. The sense of motion and momentum bolster the promise of transformation for our customers, and our new messaging is clearer and more accessible, giving multiple entry points to new customers to explore our offerings.
‎
As we move forward, we are excited to continue delivering on our promise of culture transformation, innovation, and partnership under the refreshed Eppo brand.

Introducing Perplexity as an Eppo customer

We’re thrilled to welcome Perplexity as an Eppo customer, and are even more excited to announce Perplexity Enterprise Pro free for all Eppo customers!

Perplexity launched their Enterprise Pro plan last month to give teams enhanced data privacy, SOC2 compliance, SSO, and more. To celebrate this milestone, we're offering 3 free months of Perplexity Enterprise Pro for up to 10 seats.

To redeem this offer, please reach out to Sid Sharma ([email protected]) at Eppo.

Eppo powering Perplexity and their next phase of growth

Perplexity is one of the fastest-growing AI companies ever, with top-tier customers like Stripe, Snowflake, Databricks, and Vercel. Eppo is pivotal in enabling Perplexity to test key product features and evaluate their models over the last few months while staying focused on core innovation. As Alexis Weill, Head of Data at Perplexity, puts it:

“Eppo has been instrumental in transforming our company's approach to experimentation. Their platform has helped us to foster a culture of testing and learning. With Eppo, we've been able to significantly scale the number of experiments we run concurrently, allowing us to make data-driven decisions faster."

Contact us to start using Perplexity Enterprise Pro.

A day in the life of an Eppo Customer Data Scientist

Before Eppo, Tyler built the in-house experimentation platform at Big Fish Games. He holds a PhD from the University of Texas at Austin.

“What on earth is a Customer Data Scientist?” I pondered when I stumbled upon the LinkedIn posting that would eventually lead me to my next career move.

Despite my unfamiliarity with the title, I was interested in joining Eppo. I had been following the company for a while—consistently impressed with the keen insights and statistical rigor of its technical content and appreciative of CEO Che Sharma's clear knowledge and explanation of challenges in the field in his appearance on The Data Scientist Show.

So, I applied. After all, I was already a data scientist; adding the word “customer” couldn’t be that different, right?

As I went through the interview process, I learned more about the role. At a high level, customer data scientists work closely with existing and potential customers to empower them to use Eppo effectively. We’re the “white glove service” team that makes Eppo stand out as a trusted strategic partner, not just another generic SaaS provider. When I realized the caliber of experts already on the CDS team, I started to feel like I was joining the experimentation version of the Avengers.

A big part of the role is teaching. This ranges from giving demos on how to use Eppo to giving statistical recommendations on how to analyze an experiment. This part of the job appealed to me because I greatly enjoyed being a teaching assistant in graduate school. I love the challenge of explaining technical concepts in a way that empowers others to succeed–whether that’s helping a student with a homework problem or helping a company with a product decision.

The role also has a hands-on component. CDSs often write SQL queries to help customers understand the data pipeline. They also work on Python projects that automate elements of the onboarding process or demonstrate key statistical concepts.

The multifaceted nature of the CDS role appealed to me, so I decided to accept an offer. I have now been a CDS for over two months, and it has been an incredible experience so far. In this post, I’ll discuss the key aspects of the day-to-day responsibilities of this role.

Explaining some (clearly hilarious) concepts in Bayesian statistics to the editor of this blog

Enabling Customer Success

One of my favorite parts of being a CDS at Eppo is helping customers run trustworthy experiments at scale. As the gold standard for uncovering scientific truth, controlled randomized experiments are the single best way for organizations to quantify the effectiveness of new ideas in terms of key business metrics. An experimentation program that reliably discerns successful ideas from unsuccessful ones undoubtedly yields a significant ROI and democratizes idea generation. This is Eppo’s core value proposition.

Despite the undeniable business value of trustworthy experiments, the road to success is full of traps and pitfalls that often turn an effort to follow the scientific method into statistical theater. Examples include data quality problems, sample ratio mismatch, and p-hacking. Although Eppo’s software is robust against these issues, there are many cases in which attentive support from an experimentation expert makes a huge difference. This is where CDSs come in–they work on the front lines with companies using Eppo and provide hands-on support, ranging from assisting data investigations to providing statistical guidance.

As CDSs at Eppo, we pride ourselves on a detail-oriented white-glove approach to helping customers. If a customer notes that an experiment's results are unexpected, we partner with the customer’s data teams and provide hand-crafted SQL queries to accelerate investigations. If a customer has questions about the statistical methodology, we make ourselves available to meet with them to answer their questions thoroughly. Our detailed and transparent support ultimately fosters a sense of trust in the product and the numbers it reports.

Interacting with multiple data teams

Most data science roles involve working in a specific industry with a particular tech stack. At Eppo, I work with multiple data teams across different industries every day.

Not only has this given me the opportunity to work with many talented data scientists and engineers, but it has also exposed me to a much wider range of data science problems than in any of my previous roles. While my background is in the gaming industry, where I’ve tackled tasks like implementing CUPED from scratch for a smaller in-house platform, my teammate Bertil has run experiments on massive experimentation platforms at Meta and Booking.com. Lukas (who leads the CDS team) built an experimentation tool from scratch at Angi, while Heather (our Solutions Engineer) worked at Optimizely and Twilio Segment for years.

The metrics relevant to a gaming company are quite different from those applicable to, say, a delivery service. Some customers use frequentist statistical methodologies, while others use Bayesian. In some industries, it is ideal to randomize experiments at the user level, while in others, it is better to randomize at a coarser grain. CDSs at Eppo gain hands-on experience with many different flavors of experimentation.

Interacting with multiple data teams is also a great way to stay current on trends in the data world. Customers and prospects often ask about how Eppo integrates with various tools, and truthfully, this was the first I had heard of a few of them because I never encountered them in previous roles. Now that I work as a CDS at Eppo (and can also lean on our incredible collective expertise), I have a better sense of the different tech stacks that companies use and am familiar with a wider range of tools.

A Culture of Continuous Learning

Joining Eppo has been a game-changer for my growth as a data scientist and statistician. As a CDS at Eppo, I am always learning about new data science concepts and technical papers.

Part of this is because of the nature of Eppo’s product. In most companies, data science is a means to an end; data scientists often build models or perform analyses to improve a product. At Eppo, data science largely is the product. A major value proposition of our software is that it enables our customers to leverage the best statistical methods for analyzing experiments. As a result, we make it a priority to stay on top of new developments in statistics and causal inference research. With the advent of larger datasets and more computing power, this has been quite an exciting area of research in the last decade.

One of the best examples of Eppo’s culture of continuous learning is the weekly Eppo Statistics Reading Group discussion. This is an optional meeting in which one of us chooses a technical article to discuss with the team. These discussions are quite casual and welcoming; we have an unofficial rule that nobody can spend more than an hour preparing, which generally results in a laid-back conversation in which we are all learning something new together. I am blown away by how enlightening these discussions have been. If you’d like to get a sense of the topics we have covered, check out Sven Schmit’s reading group summaries on LinkedIn.

I have also been grateful to learn from the experiences of my colleagues at Eppo. Many of the data scientists and engineers worked on internal experimentation platforms across a diverse range of companies before joining Eppo. As a result, the team is quite passionate about the product, as we have first-hand knowledge of the pain points it alleviates. It has been fascinating to hear how my colleagues approached ubiquitous experimentation problems in previous roles as well as domain-specific challenges.

Another aspect of the CDS role that has accelerated my learning is the quality of questions we receive from our customers. Many of the teams we work with include incredibly sharp data scientists who ask about advanced topics such as the tradeoffs associated with using sequential tests, the optimal way to spend alpha across a set of metrics, and how to choose an appropriate Bayesian prior.

Ability to make an impact

As a relatively young company, Eppo is growing rapidly. CDSs have plenty of opportunities to make an impact, given their combination of skills in customer interactions, statistics, and analytics engineering.

It’s satisfying to meet with customers, understand their needs, and propose technical solutions that help shape the product roadmap. In my short time at Eppo, I have already had the opportunity to partner with the engineering team to design the implementation of a new feature. One of my favorite parts of working as a CDS at Eppo is the freedom. When not meeting with customers, I’ve been able to work on projects that align with my interests and add value to the product.

The Eppo team on stage at the iO Theater in Chicago

Flexibility and Work-Life Balance

One of the best parts of working at Eppo is that it is fully remote. I love working remotely because it is so flexible. The lack of a commute adds so much time to my day and saves me from the stress of rush hour traffic. It’s nice to be able to fit in a workout during the day, move over my laundry between meetings, and spend time with my pets. Working remotely also comes with geographic flexibility. As a CDS at Eppo, I can work from anywhere. That means I can live where I want within the US without having to relocate to a city with a high cost of living.

Although working remotely is amazing, you don’t have as much opportunity to socialize in person with co-workers. Fortunately, Eppo makes up for this by having regular team meetups about once a quarter, which are a ton of fun (I just came back from my first meetup in Chicago). A favorite saying among Eppo employees is, take your work seriously, but don’t take yourself too seriously. Our Slack conversations capture this perfectly, as checking my messages typically amounts to sifting through a mixture of statistics insights, silly banter, and hilarious memes. All of this leads to a close-knit team culture despite the physical distance.

Eppo’s culture also encourages a healthy work-life balance. I’ll admit that when I first saw that Eppo has “unlimited PTO,” I had concerns about what that would mean in practice. From what I’ve seen in my two months at Eppo, taking PTO is completely normalized; in fact, our onboarding docs suggest that we take five weeks of PTO per year. I have already taken multiple days off for an out-of-state bachelor party. We also recognize 13 holidays per year in addition to a holiday break the last week of December.

Conclusions

Overall, my journey at Eppo has been a blast so far. A typical day involves answering questions from our savvy customers, nerding out about statistics with my coworkers, and laughing at memes in our Slack channels.

I believe that any experienced data scientist who is passionate about statistics, experimentation, and teaching will find the CDS role a perfect fit. It offers the rare opportunity to learn from a wide array of companies simultaneously and work at the forefront of developments in applied causal inference.

Today, Eppo is excited to launch Contextual Bandits - a powerful tool that lets data, ML, and growth teams customize bandit algorithms that automatically personalize user experiences to optimize outcomes.

Traditional A/B testing has been the gold standard for evidence-based decision-making, and the core of Eppo’s experimentation platform. However, some problems require real-time optimization instead. That’s where Contextual Bandits come in. Eppo’s Contextual Bandits offer an easy way for you to automatically optimize and personalize user experiences in real-time to ensure you’re not leaving money on the table.

Why Use Contextual Bandits?

Few machine learning techniques pique Marketing and Product teams’ interest more than bandit algorithms. These reinforcement learning algorithms balance “exploration” (or learning) with “exploitation” (or optimization) with each incremental user, introducing the efficiency of machine learning in place of the rigor of an A/B test.

The standard bandit algorithms, however, lack one important capability: context. They presume that there is only one single best action (or treatment), and their goal is to zero in on that single optimal choice quickly. Of course, there is rarely a single optimal choice across all your users — preferences often vary. Contextual bandits introduce the ability to consider key information known about a user when predicting which treatment to serve them.

1:1 Personalization, At Scale

A simpler way to think about the most common use case of contextual bandits is that they enable 1:1 personalization (as opposed to broader, rules-based personalization). If we can identify the important characteristics or “features” that might be relevant in determining the optimal user experience, a contextual bandit will help us make a one-to-one match of the right experience for each individual user, at scale, automatically.

There are a few “contextual bandit” solutions commercially available today, but these are usually “off the shelf” models that consider some pre-determined list of characteristics (usually ones that are easily determined on any website) - what browser does the user use, what is their location, time of day, are they a new or returning user?

The real power of contextual bandits, though, is unlocked by building bandits that are specific to each use case. You can use far more informative characteristics, which goes a long way towards actually achieving positive business impact. You can even use contextual bandits as a way to “operationalize” existing AI/ML models, using them as inputs in determining the right user exercise to serve and making sure you’re getting all the value possible out of what you’ve already built.

This is what we’ve built in Eppo Contextual Bandits.

Unmatched Ease and Full Integration

Teams often get excited about bandit algorithms only to find them time-consuming to build, and difficult to validate. Eppo does the hard work for you - just supply actions and their contexts to the Eppo SDK, tell us about the business metric you want to optimize, and let us do the rest.

Eppo’s Contextual Bandits also integrate easily with the rest of your stack and provide a simplified developer experience: use a single SDK for your feature flagging, experimentation, and bandits, and all configuration is handled in a unified UI.

The direct integration between Contextual Bandits and the rest of Eppo’s experimentation platform also allows for direct observation of the true impact your bandit algorithm is having on key business metrics - a key challenge for many teams today.

Proving Business Impact

Your goal isn’t just to run a bandit algorithm, you also want to make sure it actually improves outcomes. However, traditionally it has been difficult to prove the actual ROI or business impact of implementing a bandit algorithm. Generally, bandit algorithms work best optimizing short-term metrics, but business metrics are often measured on a longer timescale.

But there are also more technical challenges to accurately measure the impact of a bandit: observations across users are not independent, as actions the bandit algorithms take today depend on the historic actions and their outcomes: if the bandit decided to take a different action on day 1, that could lead to a very different bandit policy on day 10.

To solve for these challenges, Eppo’s Contextual Bandits are tightly integrated with our experimentation analysis tools and leverage a holdout strategy to measure performance. This lets you rigorously understand performance across any metric you care about, measure guardrail metrics, and conduct deep-dive investigations to understand exactly what is happening under the hood. It also means that you have all the same world-class statistical tools available as any other experiment on Eppo: CUPED, sequential tests, Bayesian analysis, etc.

Where Should You Use Contextual Bandits?

Contextual Bandits are powerful tools for making personalized decisions at scale, without the heft or cold start problem of recommendation systems. There are potential use cases anywhere a more tailored user experience may improve outcomes. Here are some questions to consider when exploring if contextual bandit algorithms are the right tool for you:

• Do you have many options to choose between? (likely tens to hundreds)
• Does the optimization problem have a short-ish timeframe? (weeks instead of months or years)
• Do you have informative data to inform context? (e.g. a logged-in user) If so, Eppo Contextual Bandits can help you shortcut the hard work of building and implementing bandits from scratch and let you get straight to what matters - setting your personalization strategy and driving real business results.

Starting Using Contextual Bandits Now

With the launch of Contextual Bandits, Eppo reaffirms its commitment to empowering teams with the decision-making tools they need to succeed in a competitive digital landscape. We're excited to see how our customers will leverage this new product to achieve unprecedented levels of personalization and efficiency in their optimization efforts. Welcome to the future of data-driven decision making — where the gold standard of randomized controlled experiments meets the cutting edge of machine learning, powered by Eppo.

Eppo customers can start using this feature today. For those considering Eppo, we invite you to request a demo and see how it can enhance your experimentation (and optimization) efforts.

Ensure core metric definitions are in sync across experimentation in Eppo, BI, and other data platforms, all managed via GitHub

Eppo's Founder and CEO, former early data scientist who built experimentation tools and cultures at Airbnb and Webflow

We are excited to launch Eppo Certified Metrics to all customers, Eppo’s native integration with semantic layers like dbt metrics. Now data teams can make sure core metric definitions are in sync across experimentation in Eppo, BI, and other data platforms, all managed via GitHub.

At Eppo, we believe in growing experimentation culture by ending the practice of statistical theater. One of the most common causes of statistical theater is when experiment platforms mistakenly use bad data and bad metric definitions. Eppo’s warehouse-native experiment engine now has a metric layer that’s even easier to sync with the rest of a company’s data platform, by the beauty of semantic layers like dbt metrics.

Why it matters: a story about when SaaS tools got their COVID customer response wrong

I was a data scientist at Webflow when COVID-19 first broke out. We spent March 2020 scrutinizing every data trend to see how Webflow customers would react and if our business would hit a brick wall.

I mostly remember that month by how wrong the analytics tools were. We used a black box SaaS analytics platform that claimed complete health across all subscription SKUs. But since it felt like the world was crashing, we decided that our nascent data team would take a deeper look to make sure.

Lo and behold, the cheaper subscriptions were churning fast. We could see it clearly in the Stripe data we had in the warehouse. The (unnamed) SaaS tool didn’t show the churn because it made several metric definition choices that didn’t match how we see the world:

• Churns weren’t shown in charts until a subscription’s paid period officially ended. Churning on Jan 10 wouldn’t be counted until Jan 31 (or Dec 31 if it was an annual plan).
• Churns weren’t counted until an account’s entire set of subscriptions churned. (Webflow accounts have several subscriptions, one per website and one for the account itself.)
• One class of subscriptions made via Stripe Connect wasn’t included at all due to how Webflow architected them.

An example of how bad metric definitions misled Webflow

‎

‎This problem is endemic to analytics. Each tool and each analyst makes bespoke metric definition choices for their own task. But this anarchic world where everyone makes different choices for “how to count revenue” erodes overall trust in data. If the CEO gets five different numbers for “how many products did we sell last week?”, they are going to wonder how tens of millions in data budget can’t answer the simplest business questions.

But now, there’s finally a good answer for how to unite analysts and data platforms on metric definitions: Enter the semantic layer.

One Definition to Rule Them All

Today, Eppo announces the release of Certified Metrics, our native integration with dbt metrics. Eppo is the first experimentation platform to adopt this growing standard and natively reuse the same metric definitions that underlie customer BI systems and other analysis platforms. For the first time, data teams can focus on modeling metrics and automatically have downstream data platforms use those metrics. All metric definitions are controlled in GitHub for version control and change management.

When dbt acquired Transform, they blessed the MetricFlow common standard for metrics. This enabled products like Eppo to follow suit and reinforce the standard. To illustrate the dbt metric metric schema, consider the benign question, “How many products did we sell last week?”. Here are all of the decisions an analyst or tool has to make:

• What data source best defines a product purchase? Event telemetry? Application databases?
• Are there purchases that should be thrown out, such as returns? Giveaways?
• What time in the purchase lifecycle should be used for falling into “last week”? When the credit card is charged, when the product arrives?
• What dates fall into “last week”? Do weeks start on Sundays or Mondays?

And so on. A semantic layer allows data teams to make each of these choices across all downstream systems. In this world, Webflow would have a “churn initiated” metric that could have revealed the worrisome trends. And now, Eppo would automatically pick up the same churn definition from dbt metrics.

Certified vs. Exploratory Metrics

At Eppo, we believe experimentation requires centralized trust in common standards such as core metric definitions. But experimentation is also a curiosity-driven process, where ad-hoc metrics are constantly examined. A growth experiment might primarily be based on dbt metrics like activation, revenue, and churn. But a growth team might also want to know lower-stakes metrics like “how many people used my new widget.”

Eppo Certified Metrics shines in balancing a git-centric certification flow with a faster in-app process. The GitHub workflow embraces standardization and peer review, while the in-app workflow lets teams operate at the speed of curiosity. In this world, a growth team’s Eppo instance might look like this:

• The team can automatically use core business metrics that have been curated by the data team in dbt metrics and synced into Eppo. These metrics get a “certified” badge to indicate their increased trust and importance.
• To answer a specific hypothesis around visibility of a widget, in seconds, a growth team adds a visibility metric to Eppo’s UI.
• As the widget grows in success and shows that it’s here to stay, the data team “graduates” the original visibility metric to dbt metrics so other teams can make use of it. Eppo automatically generates the yml file and transitions the internal definition to the synced, certified one.

Before Eppo certified metrics, companies had two losing options: either force all metric definitions into Github, slowing teams down with a multi-day review process and swamping the data team’s limited bandwidth… or forget the semantic layer and deal with a wild west of similarly sounding metric definitions.

Experimentation Centered on Trust, Centralization

Our goal at Eppo is to change corporate culture, unleashing internal entrepreneurs with a scientific process centered on customers. Experimentation platforms are uniquely positioned to establish a trusted, centralized process for recognizing great ideas and learnings.

But no amount of advanced statistics, quality hypotheses, or beautiful reports can stand up a culture if we cannot agree on what the underlying data is measuring. With Eppo diagnostics and now certified metrics, Eppo customers can make customer-driven decisions, knowing that data quality and metric definitions are sound.

Thank you to Nick Handel and the dbt team, Martin Tingley and the Netflix team, and everyone else who helped to inspire this project. Stay tuned as we continue to enable experimentation corporate culture everywhere.

Want to try out Certified Metrics? If you’re an Eppo customer, you can use this feature now. If you’d like to talk to our team about using Eppo, simply request a demo.

Today, we’re excited to launch Holdouts. With this release, we’re providing the flexibility of using Eppo to create your holdout, or bringing your own holdout and using Eppo’s analysis tools.
Holdouts are the gold standard for measuring the cumulative impact of an experimentation program, and we’ve developed Eppo Holdouts to address common challenges while maintaining the highest statistical rigor.
Holdouts are a small allocation of traffic that is “held out” of an experiment. This traffic is kept on the control experience and not shown the experiment treatments as they are run. By maintaining a holdout over a long period of time - a quarter, six months, a year, or longer - the holdout group provides a comparative lens between users who saw the experience as it originally started versus those who have been subject to the product’s many changes over time.
The result of a holdout is a more accurate metric impact measurement that aggregates all changes made, versus a method of measuring the impact of each experiment in isolation. This impact tends to be lower than summing up the individual impact of multiple experiments due to the removal of biases such as the “winner’s curse.” Thus, the organization has a much better understanding of how the experimentation program has affected business metrics.
Holdouts will often uncover that the cumulative impact of several experiments is lower than expected.

A More Rigorous Holdout Approach

A frequent misconception about Holdouts concerns their configuration. Typically, a holdout might be a control group (e.g., 10% of traffic) compared with the remaining majority (e.g., 90% of traffic). This approach risks diluting the ability to measure the impact of winning treatments. This is because the non-holdout group's composition is inconsistent, including users who are exposed to losing variations that harm their outcomes. We do not care about measuring those harms in the Holdout analysis as these variations won’t be rolled out. Eppo's solution is to hold back two equally sized groups, with traffic allocated in this way:

• 1. The held-out status quo group: Users consistently exposed to control experiences.
• 1. The held-out winners group: Users who consistently see winning treatments as they are launched.
• 1. The remaining traffic - A large group of users who are exposed to experiments and updates as they are made available

How one experiment with a winning variant gets allocated into the Eppo Holdout

This method reduces error risks and allows for earlier signal by exposing the winners group to winning variants as soon as they are rolled out. This minimizes the duration of the holdout compared to other methods.

How multiple experiments, with Experiment 1 and 3 with winning variants, get allocated into the Eppo Holdout

Simplified Holdout Creation and Analysis

Our approach streamlines the holdout creation process. Setting up a holdout is as straightforward as selecting a date range and specifying the traffic percentage for the holdout. All experiments initiated within this period automatically include the holdout, requiring no additional user intervention.
Eppo applies the same suite of Diagnostics to Holdouts as to experiments. If you’re using Eppo’s Slack Notifications, you’ll be promptly alerted to any issues like traffic imbalances. Moreover, Eppo Holdouts utilize the same analysis tools as experiments, enabling customers to assess their impact on event metrics and key business metrics, including Revenue. This is complemented by a holdout-specific report detailing the influence of each experiment on primary metrics.

Introducing Analysis-only Mode

In addition to the primary Holdouts product, Eppo is also launching an Analysis-Only option. We firmly believe that experimentation is modular, with two jobs to be done: deployment and analysis. While we built a solution for customers who want to do both, we made sure that Eppo Holdouts are also available for those who use their own feature flagging solution for deployments by providing an Analysis-Only mode.
Customers who bring their own holdouts get access to Eppo's comprehensive suite of analysis tools. This includes diagnostics for holdout health, comparisons with business metrics, and detailed holdout reports. Customers can also link their holdouts to experiments within Eppo to illustrate the impact of individual experiments on the holdout.

Measuring Aggregate Impact

We are thrilled to offer these tools to our customers, enabling them to measure not just the impact of individual experiments but also the collective influence of their teams and programs.
Interested in exploring Holdouts? If you're an Eppo customer, you can start using this feature today. For those considering Eppo, we invite you to request a demo and see how it can enhance your experimentation program.

Create visually compelling, fully contextualized PDF reports built to communicate experiment results org-wide.

Today, we are pleased to announce the release of Eppo Reports. From the team that brought you Airbnb’s knowledge repository, Eppo is finally closing the sharing gap of experimentation teams. Eppo customers can now easily create visually compelling, fully contextualized PDF reports that are built to travel across an organization. And because the reports are built in Eppo, they are forever indexed and available to future teams and meta-analysis rollups. Reports are available now in GA across all Eppo customers.

At Eppo, we care about more than experiment automation, we started this company to drive experimentation culture. This is the feeling of meritocracy, when leadership puts customer impact first and enables every team to test ideas without having to win a political process.

To date, we’ve focused on trustworthiness. We are 100% committed to making sure companies avoid statistical theater. Experiment culture’s most common cause of death is pervasive skepticism from tooling that doesn’t address data quality, metric governance, and statistical rigor. We built Eppo to establish a new standard for trustworthiness in an experimentation platform.

But experimentation culture requires more than correct mechanics, it requires constant evangelism. The scientific approach to product needs to be highly visible and celebrated. When done right, experimentation moves leadership’s eyes from inputs (product marketing, shipping) to outputs (metrics, learnings).

The predictable result is enormous time spent curating experiment reports. PMs have to cat-herd a cross-functional team to confidently pull data, organize the right narrative, and scrawl it all out in Google Docs. Inevitably, the teams write in a completely different structure, adding cognitive load to readers. And after all of that time, the reports are usually lost forever in a dark corner of Google Drive, never to be read again.

And after all of that work, it’s still hard to get promoted off a low-trust Google Doc. These reports lack the aesthetic lift that emotionally triggers trust and excitement. The truth is, anyone trying to win hearts and minds could take a page from the McKinsey playbook: dressing up presentation decks is good ROI. Presentation quality matters.

Today, we are launching Eppo Reports, our next foray into helping companies drive experimentation culture. We’ve brought all of the learnings from Airbnb’s Knowledge Repo to experimentation, helping product managers save time and stress while vastly enhancing the brand of any experiment team using Eppo. Now, experiment wins and learnings are easily marketed across an organization, with results indexed in a knowledge base for future teams.

Build reports with blocks ranging from simple document elements to advanced slice-and-dice explorations

• One-click PDF export creates beautiful, high-fidelity artifacts ready to share across the org, in any tool

• Reports become part of an easily searchable repository to build institutional knowledge

Eppo’s reports are built with these ideas in mind:

• Meet people where they are: Reports need to live on the collaboration tools that already exist. As much as we’d love for the CEO to browse Eppo each day, it’s likely that the report will need to render in email, or on a phone in transit.
• Trustworthiness: The silent killer of experimentation culture is leaders not fully believing in the numbers they are presented. Reports cannot amplify misleading information from data pipelines not working, wrong metric definitions, statistical malfeasance, and cherry picking numbers.
• Build a brand: Documents leave an impression on the reader before they’ve even started reading. Like a document using official letterhead, an experiment report can reinforce its trustworthiness with consistent formatting, branded aesthetics, and a narrow focus on what’s important.
• Storytelling: The low-context report reader wants to know: why this experiment matters, what the team did, what the takeaway is. Insights become institutional knowledge when there is a narrative surrounding the charts and numbers.

Eppo reports are built for today’s collaboration tools: email, Google Docs, Notion, Slack. They elevate the brand of experimentation teams with intentional aesthetics and a flexible block system for storytelling. And because Eppo is built to centralize experimentation across product, growth, ML/AI, and marketing, Eppo reports can finally give a consistent, trusted report format across departments.

We’ve always said, our goal at Eppo is to drive corporate culture change at our customers, unleash the internal entrepreneurs, and enable companies to truly be rigorous about how well they understand customer impact. Our team built experimentation infrastructure at Airbnb, Uber, Stitchfix, LinkedIn, and we know well that experimentation culture ultimately builds through people and collaboration.

A special thank you to Annemarie Klaassen and the Vodafone Ziggo team, the folks at Dovetail, and everyone else who helped to inspire this project. Stay tuned as we continue to enable experimentation corporate culture everywhere.

Want to try out Reports? If you’re an Eppo customer, you can use this feature now. If you’d like to talk to our team about using Eppo, simply request a demo.