Eppo Release Notes

Successful acquisitions are based on what makes sense for customers. For Eppo’s experimentation customers, there was no better fit than the world’s leading observability platform in Datadog, where product development teams already turn to understand the "why" behind application issues. Since the deal closed, we’ve been building observability-native feature flagging and a next-gen approach to experiment diagnostics and deep-dives. Building these new products has given us a unique opportunity to build upon what we learned at Eppo and make improvements that would be hard to do in the confines of our "v1". Today, we’re focused on solving what we consider to be the three biggest problems in experimentation.

Catch Errors Sooner

The first problem is that, for all of the advanced statistics available to speed up testing, most of the time lost when running experiments is due to more mundane issues (i.e. bugs) that require discarding results and re-starting. It’s a pain to realize after a few days of runtime that the experiment is broken on an old version of Internet Explorer, there’s a UX degradation on a specific viewport size, or a key instrumentation event is missing on Android. These stop-starts end up taking as much time as the experiment itself. Datadog Data Observability, for instance, gives us the ability to detect metric instrumentation issues upstream before experiment runtime is wasted

Automate Canary Testing at Scale

The second problem is that commercial tools lack first class support for canary release testing. Experimentation discourse usually focuses on scientific thinking and finding wins, but the largest experimentation programs in the world like Netflix, Uber, or Microsoft are built on a foundation of comprehensive canary testing. By automatically turning code release workflows into randomized experiments, engineers build experimentation muscles and learn statistical thinking. The problem for most companies is that executing canaries in this way is too manual and requires engineers to babysit every release (often leading them to navigate to Datadog or similar products). Our new Datadog Feature Flags add first-class support for standardizing, automating, and monitoring canary tests

Deep Dive with all of your Data

The third problem is that the amount of insight and intelligence generated per experiment is still lower than it could be. The "state of the art" approach to experiment deep-dives often looks like slicing experiment results by every possible segment, looking for a smoking gun explanation of why the experiment isn’t successful. Hopefully, these teams are at least aware that their "multiple testing" is bound to lead to plenty of false positives. But even when teams correctly identify an underperforming segment, they still need to root-cause to the specific issue, whether it be confusing UX, poorly-aligned personalization, or something less apparent like application slowness. Datadog Experimentation combines our best-in-class warehouse native approach with other data sources like application vitals or Datadog Real User Monitoring

Partnering with Datadog immediately gave us tools for solving each of these top problems in experimentation. Real-time observability metrics helped us leapfrog into an exciting new suite of experiment diagnostics, so engineers can catch product bugs and experiment issues right as they turn on the test. Statistical canary testing can now be automated based on errors, infra metrics, and product telemetry. And stay tuned for a new spin on experiment deep-dives that weaves the universe of data that Eppo and Datadog bring together, including warehouse metrics, behavioral events, and application vitals. The Eppo team (and me as CEO) are still 100% focused on experimentation, and Datadog brings a wealth of experience of enterprise support. Today, we’re excited to share that interested teams can request access to our new Datadog Feature Flags as part of Datadog’s Product Preview program. We’ll be sharing a lot more of what we’re building this fall, including in-person at EXL, MIT CODE, and AWS re:invent. More to come soon!

• Che & the Eppo team

Eppo Is Now Part of Datadog!

This acquisition will drive learning velocity for all Eppo customers, equipping them to expand experimentation across their organizations

Eppo's Founder and CEO, former early data scientist who built experimentation tools and cultures at Airbnb and Webflow

Eppo Is Now Part of Datadog! I’m excited to announce that Eppo has been acquired by Datadog. This acquisition will drive learning velocity for all Eppo customers, equipping them to expand experimentation across their organizations. With visibility into how any design, product, or technology change impacts the customer experience, teams will be able to connect engineering work to business outcomes—breaking down silos and enabling a culture of velocity and accountability. There’s much more to come, but I wanted to start by sharing why we’ve taken this step.

Eppo’s Approach

Eppo has always been about learning velocity, with a theory that the companies that learn the fastest, win. This principle is embodied by experiments, ranging from canary testing to A/B testing and automated solutions like bandits. The more experiments a company runs, the higher the company’s learning velocity tends to be. But there’s nuance in how we entered the experimentation market. Eppo’s vision started from an opinionated take on which experimentation cultures were getting it right, where data-driven behaviors were actually driving business value instead of spinning statistical theater. Summarized, this led us to believe experimentation should be:

• Broadly accessible
• Highly trustworthy
• Used across all important business levers Experimentation vendors new and old were failing this test, whether due to outdated worldviews on data teams, a lack of support for critical testing use cases, or a lack of empathy for diverse company contexts. We felt that a modern approach to warehouse metrics, a UX approach rooted in bringing various skillsets together, and a focus on bringing the frontier of statistical methodology to the masses would solve these problems.

Why Datadog

Joining with Datadog offers three major opportunities to move the industry forward into the next era of experimentation and learning velocity. These opportunities are rooted in the same three principles.

• First, we can accelerate Datadog’s movement to becoming more collaborative by making products that are broadly accessible. Datadog began with a focus on engineers, but it has made major strides with product and data teams with features like Product Analytics, Session Replay, and Data Observability. Eppo is excited to add to our DNA of PM-friendly UX, statistical rigor, and warehouse metric management to Datadog’s existing products.
• Second, a combined Datadog and Eppo can make learnings highly trustworthy, with an added pinch of at the fastest speed. With our combined technology, we can strike the right balance of precision and latency. Experimentation needs the most precise data sourced from the warehouse, and it is okay to have higher latency to ensure even 1% impact readings are correct. Product analytics, however, benefit from fluid, real-time event data that can explore at the speed of curiosity. In this category, raw, uncleaned data is fine for detecting large 50% differences. Eppo is a market leader in warehouse-centric composable metrics, and Datadog’s bread and butter is real-time event processing.
• Third, we can drive learning velocity across multiple high-value problems. For example, a cousin of the A/B test is the controlled feature rollout via canary testing. Datadog’s real-time observability with Eppo’s flags and stats engine means that a true end-to-end canary test solution will finally be on the market. Datadog’s AI observability with Eppo’s contextual bandits will allow AI teams to ensemble gen AI foundational models in a state of continuous testing and rebalancing. Stay tuned to learn what is possible with our combined technology, and how we might make multiple Datadog products more statistical. And of course, Datadog’s scale means that we can drive experiment velocity across 30,000+ businesses across the world. We’ve already proven that Eppo can drive success at 20-person startups and Fortune 10s, from decades-old retail companies to explosive AI natives. This is an opportunity to set a new de facto standard of experimental building practices across the globe.

Positioned for Agentic Product Development

Eppo was founded in 2021, a time that feels like the Stone Age since gen AI models have become mainstream. As we wrote in our AI manifesto nearly two years ago, the process of deriving ideas, implementing ideas, and measuring idea success has been turbocharged. Eppo has benefited here, with thousands of AI-driven experiments running through Eppo servers each year. As sci-fi as our AI manifesto felt at the time, our imagination still fell short in one respect. We originally envisioned a human-in-the-loop process of tech workers implementing faster, better. But with the rise of AI agents, it has become clear that some types of product development will become fully closed-loop. Instead of engineers bussing tickets through a queue, AI agents can identify an issue, find its root cause, and implement a fix. And with flags and experiments, the fixes can be safely rolled out with all appropriate metrics measured statistically.

What Happens Next?

Eppo customers will continue to receive the same experience, support, and services. Users will still log in to Eppo, now branded as Eppo by Datadog. We’ll be building a new vision for experiment-centric product development that combines Datadog’s expertise in observability with Eppo’s expertise in experimentation. We’ll be working closely with the Datadog teams building Product Analytics, Real User Monitoring, and Data Observability. The goal is to drive learning velocity for our customers, based on the belief that the companies that learn the fastest tend to win. We can accelerate our vision by getting all ideas tested, shipped, and celebrated across Datadog’s 30,000+ customers.

A Leap Ahead with More to Come

Joining forces with Datadog is the quickest path to the future of product development, one that’s more resilient, scientific, and agentic. We’re excited to join a thriving Datadog product culture, and Eppo customers can immediately take advantage of a broader suite that includes Product Analytics, Session Replay, Data Observability, and Application Performance Monitoring. The Eppo team is committed to building as part of Datadog for years—stay tuned for much more to come!

• Che & The Eppo Team

As a Product Manager, every quarter you’re given a goal and you’re responsible for producing a roadmap that meets that target. You and your team put your best ideas together, and you have some confidence because they’re good ideas founded upon user research, but you still don’t really know if you’ll quantitatively hit your number.

Welcome to the life of a modern product manager. It’s a role of gathering inputs, finding creative solutions, technical understanding, strategic prioritization, and a fair bit of persuasion. When experimentation, proven to be one of the most effective ways to achieve data-driven growth, is layered on top, there can be more pressure in how you’re judged based on the impact your team has made. That’s where Experiment Forecasting steps in.

Experiment Forecasting is designed to relieve some of your stress, giving you clarity, control, and confidence as you tackle conflicting demands.

Why Experiment Planning Matters

Experiments are integral to shipping high-quality features that move the needle, but they come with unique challenges. If you’ve experienced any of the following, you’re not alone:

• The “Just Ship It” Urgency: There’s constant pressure to release features quickly, often long before the data from experiments has come in. Do you push something unproven out the door or risk annoying stakeholders by advocating for patience?
• Engineering Bottlenecks: Maybe you’ve jotted down the perfect experiment idea but hit a wall waiting for resources to run the test. Now, it’s a scramble to keep progress going.
• Roadmap Whiplash: Plans change; sometimes slowly, other times at warp speed. Roadmaps can quickly morph into tangled webs that stress everyone involved.
• The “Always Ship More” Mandate: Whether it’s internal KPIs or external competition, there’s immense pressure to keep feature velocity high.

Experiment planning and forecasting allow you to zoom out and solve these issues systematically, bridging the gap between stakeholder expectations, engineering constraints, and the actual impact your team can deliver.

What Is Experiment Forecasting?

Experiment Forecasting is an approach that brings data-driven clarity to your product experimentation strategy. Instead of making wild guesses or operating on gut feel, it offers data-driven predictions. With tools like Eppo’s new Experiment Forecasting feature, you can chart out your future experiments on a timeline, project their potential impact, and assess if your quarterly roadmap is likely to hit your goals. Take control of your roadmap instead of letting it control you.

How Experiment Forecasting Helps You Win

Experiment forecasting changes how Product Managers approach planning and achieve their metric goals. It provides practical insights to create strategies, align teams, and make experiments more effective. Here's how it makes a difference:

• 1. Map Out Your Best Ideas and Gain Strategic Clarity Experiment forecasting allows you to measure potential impact against your metrics. By using historical data, you can estimate outcomes and determine if your plans will drive progress or need adjustments. This insight helps you make thoughtful decisions with confidence.
• 1. Spot Gaps and Add Experiments as Needed Forecasting helps you assess whether your roadmap includes enough experiments or if you need higher-impact initiatives to meet your goals. These insights enable you to adjust plans proactively rather than play catch-up later.
• 1. Communicate Expectations Clearly to Stakeholders Experiment forecasting helps you craft a clear, data-driven narrative for stakeholders. Instead of presenting just a list of projects, you can share forecasts that show how your experiments align with hitting company goals. Quantitative estimates build trust and make setting realistic expectations with leadership and your team easier.
• 1. Jump-Start Collaboration with Eppo’s Experiment Drafts With Eppo, you can draft your roadmap experiments, complete with key details like metrics, entry points, and projected impact. The timeline view helps place experiments visually, showing clearly how they’ll fit into your quarter. This makes cross-functional collaboration easier and ensures that, when the quarter begins, your plans are ready and endorsed by your team.

Experiment forecasting improves planning and lays out a path to measurable success. With tools like Eppo, your experiments and strategies have the clarity and alignment needed to drive results consistently.

Experiment Forecasting in Action

Here’s how a quarterly planning session might feel with Experiment Forecasting in place versus without it.

Without Forecasting

You’re creating a roadmap that’s ambitious but lacks clarity. You feel a pit in your stomach because you think these experiments will hit the metric targets, but you’re not positive. Your team plans to evaluate on the fly, knowing you’ll probably need to pivot halfway through.

With Forecasting

You sit down to map out 10 experiments for the quarter. For each, you know its primary metric, the entry point, and the forecasted impact based on past win rates. This lets you objectively benchmark your team’s goals against reality and adjust beforehand. You also clearly communicate key dependencies (engineering and data science needs, launch schedules, etc.) so everyone is aligned before the roadmap even kicks off.

Eliminating guesswork and unpredictability will help you and your team start the quarter with more confidence, alignment, and trust.

Balancing Strategy and Execution

Experiment Forecasting doesn’t just help you plan; it frees you to execute on real impact. It’s one more way to balance the art of product strategy with the science of experimentation while navigating the daily pressures you face. Our goal? To make your job as a PM just a little easier and more effective.

Want to see it in action? Learn more about Eppo’s Experiment Forecasting and how it can fit seamlessly into your product development cycle. With the right tools, the next quarter might just be your most productive yet.

Marketing thrives on making quick, data-driven decisions. Yet, for many marketers, web A/B testing has been a source of frustration with flickering pages, broken elements, and the endless headaches of visual editors. What if you could achieve a seamless experience of engineering-level precision without actually needing engineering?

Now, you can do it with the Eppo-Contentful integration. Teams can now craft and manage content directly within their CMS, rather than relying on clumsy tools to build A/B test changes. Want to run an experiment? Simply flip a switch to turn your new changes into a test variation with no engineering roadblocks and zero stress.

Here’s how it works. The Eppo-Contentful integration requires a small one-time engineering setup. After that, teams use the entry ID of any piece of content in Contentful to define experimental variations, creating a powerful, scalable workflow. By combining two best-in-class tools, this integration unleashes no-code experimentation, empowering marketers to test and iterate without limits.

The Challenges Marketers Face

Experimentation is every great marketer's superpower. Testing new ideas and understanding what works can amplify impact, but common hurdles make it harder to capitalize on this potential:

• Engineering Dependencies: Traditional A/B testing demands engineering support to set up feature flags, modify codebases, and manage backend workflows. Waiting on these resources slows progress.
• Tool Overload: Managing experiments across multiple platforms creates unnecessary complexity and increases the risk of errors.
• Performance Hits: Client-side scripts often cause flickering or slowdowns, disrupting user experience and compromising data accuracy.

These roadblocks can leave marketers feeling stuck, unable to iterate quickly or confidently prove results. That's where the Eppo-Contentful integration comes in.

Why Eppo + Contentful Is a Game-Changer

This integration transforms the way you run experiments. Bringing Eppo's powerful experimentation capabilities directly within Contentful's headless CMS eliminates the need for constant engineering support and keeps everything streamlined in one platform. With everything served from the server, you can test hero images, copy, or even entire layouts seamlessly, all without sacrificing performance or user experience.

What Sets It Apart?

Unlike other solutions, the Eppo-Contentful integration avoids the challenges of performance-hurting client-side scripts. Instead, it relies on a headless CMS with feature flagging to keep your site fast and reliable. Its benefits include:

• Zero Reliance on Engineering Teams: Set up and run experiments entirely from Contentful's interface without waiting on backend work or sprints.
• Top-Notch Performance: Forget page flickers. Your experiments are embedded directly into your site without slowing it down.
• Effortless Setup: Installing Eppo's Contentful app is as easy as a few clicks, and its intuitive design makes experimentation accessible for marketers of all tech skill levels.

Here's How It Works

Getting started is easy and designed with marketers in mind:

1. Install the app from the Contentful Marketplace and input your Eppo API key.
2. Configure content types you want to test: blog posts, hero images, or banners.
3. Create content variations, such as alternate CTAs, colors, or headlines.
4. Publish and automate via Contentful, where Eppo automatically generates the necessary feature flags in its system.
5. Enable and Go Live with a single click in Eppo, bringing your experiments to life.
6. Track results using Eppo's analytics to find the winning variation backed by complex data. This happens in a single workflow, without switching between tools or tedious handoffs.

Real-World Use Cases

Picture these scenarios where the Eppo-Contentful integration solves real marketing challenges:

1. Optimize Landing Page Conversions You're launching a major product campaign and want to test different hero images. With the integration, you can create variations directly in Contentful, publish experiments, and identify the winning design while keeping load times blazing fast. There is no need to involve engineering at any point.
2. Maximize Blog Engagement Are you struggling with bounce rates? Experiment with blog post headlines or introductory text to see what keeps readers engaged longer. Using this integration, changes can be easily made and tracked across thousands of posts, saving countless hours of manual updates.
3. Test Different Offers for Targeted Audiences Want to test a new discount but don’t want users randomly seeing different prices as they go from mobile to desktop? Eppo supports regionally-randomized experiments, called Clustered Experiments. You can implement them via Contentful and analyze them natively. With this method, every user in the test regions will see the same price, but when it comes time to analyze the results, you can compare results user-to-user, accounting for regional variations. Use the integration to tailor messages like banners or CTAs for specific locations. Testing multiple designs or texts doesn't require new code, just a few configurations in Contentful.
4. Refine Call-to-Actions Eppo makes it easy to experiment with CTAs like "Get Started" vs. "Learn More" or even entire boxes with varying designs and copy. Consistent across pages but customizable where needed, you can improve clicks and conversions with actionable insights.

Smarter Marketing Starts Here

Great marketing isn't just about big ideas; it's about testing, learning, and iterating. The Eppo-Contentful integration allows you to experiment boldly without compromising speed or site performance, no more waiting or bottlenecks. Here's why you should try it:

• Launch experiments faster than ever.
• Save time and resources by removing engineering dependencies.
• Make every decision data-driven with robust analytics.
• Scale effortlessly, whether you're managing 10 pages or 10,000.

Start Experimenting Today

Don't wait to revolutionize your marketing approach. Install the Eppo app in Contentful now and see the difference firsthand. Visit the Contentful Marketplace, connect your Eppo API, and create experiments in minutes.

Use gold standard incrementality testing to evaluate the true contribution of marketing to your company's bottom line

Before Eppo, Greg was the CEO of Tech for Campaigns where he led large consumer advertising campaigns and experimentation programs

Learn more about Eppo Geolift at our webinar on November 14th with Bryce Casavant, Senior Data Scientist at WHOOP and Eppo's Marketing Experimentation Team

How do I know where to put my marketing dollars when my data warehouse, advertising partners, and web analytics don’t agree? Privacy changes and black box ad platforms make it harder than ever to find trustworthy marketing signals at the same time that the bar on marketing ROI rises even higher.

As Che wrote in our Series B announcement, it’s never been a greater risk to not know which company initiatives are growth levers versus “low-ROI money pits.” Within marketing, “incrementality testing” is the gold standard to evaluate the true contribution of marketing to a company’s bottom line. A key incrementality strategy is geo-testing or “matched market” tests, where a certain set of regions receives different marketing programs than another, and the differences are evaluated. These tests provide real insights but require specialized statistical knowledge, dedicated operational expertise, and can be tricky to align with a company’s business metrics.

Enter Eppo Geolift: empowering marketers and data teams with the tools they need to make rigorous marketing investment decisions with ease and confidence. Eppo Geolift’s quasi-experimental methodology measures significant changes in your business metrics across geographies using cutting-edge Synthetic Control Methods. Critically, Eppo Geolift uses the same integrations and metric definitions used for lifecycle marketing, product, and AI experiments within Eppo, providing a common language across teams with one place to see all growth initiatives.

Share Geolift with your marketing team or data team

Simple Setup

Without help, launching a geolift program can be complex. Marketers first need to run calculations for which geos to select, for how long to run the test, and at what spend levels the experiment should run. Then, when it comes time to launch, many marketers feel like they’re stepping into a void where six or seven figures can be at stake and hoping to avoid landmines in the process (like turning off Branded Search in your target geos but finding out after that PMax ended up buying all the same keywords!)

To solve this, Eppo guides users through the process of designing a test, launching it, and delivering easy-to-read results. Additionally, Eppo's Protocols offer presets that incorporate statistical best practices and organizational guidelines. Protocols let data scientists empower technical marketers and data analysts to confidently run tests without full-time assistance, enabling teams to move quickly and scale testing.

Always-On Testing for Digital Platforms

Once a program gets off the ground, geolift vets know success lies in maximizing the number of learning opportunities throughout the year. More tests mean understanding Return on Ad Spend (ROAS) at different spend levels, performance across creative types, and time varying factors like seasonality and adstock/decay effects. Eppo Geolift offers automated testing, monitoring, and analysis for major digital platforms including Meta, Google, TikTok, and Pinterest. Our advanced causal inference methodologies minimize holdouts, allowing you to run experiments with minimal disruption to your ongoing campaigns.

Powerful Custom Experiments

Brand buys, inventive programs – Eppo Geolift also offers custom experiments for use cases like measuring offline media or non-geographic initiatives (like search engine optimization). Eppo provides data scientists input into the assumptions used in our power calculator, explicit control over the creation of the synthetic control, and complete flexibility to use one-off metrics as needed.

The Power of Real Business Metrics

Many incrementality options depend either on black-box vendor methodologies or sending your data to an outside third party via API, Google Sheets, or S3 buckets. The external data transfers often mean that the results don’t perfectly align with the business metrics that finance is using.

Eppo Geolift integrates directly with Snowflake, BigQuery, Redshift, and Databricks. You're working with the same trusted metrics your CFO uses—no more discrepancies between lift reports and financial realities. Got a question about a metric? Double-click into the SQL query that drives it.

The Broader Eppo Platform

Geolift is built upon the same platform that our lifecycle marketing and product experiments are built on: easy-to-use design tools, experiment calendaring and sequencing, centralized and vetted metrics definitions, easy report building, and a knowledge base to make sure what’s learned is retained.

Let's get started

Eppo Geolift empowers marketers and data teams to continually optimize their marketing investments for their true business contributions – working off the same metrics as the entire business.

Interested in Geolift? Eppo customers can start designing experiments with us immediately. If you’re not already integrated, we’d love to chat!

Good product teams ship quickly and drive impact. Bad product teams move slowly or they have bad experimentation hygiene, making impact reporting untrustworthy. It can hard for product leaders to discern one type of team from the other. Because there’s so much going on, these leaders are hearing a lot of success stories but they lack the means to verify that teams are operating well.

We built the Performance Scorecard to solve these problems. It is designed to bridge the gap between individual experiments and overall program performance, and ultimately impact. This provides a comprehensive view of your experimentation efforts.

The Performance Scorecard goes beyond traditional metrics by measuring both inputs (how you're experimenting) and outputs (what you're achieving). This dual approach ensures that you can see whether teams are operating in ways that lead to velocity and impact.

Aggregate Impact: Quantifying Your Program's Success

One of the most significant challenges of experimentation programs is demonstrating their overall impact. Previously, teams had to manually compile data every quarter during planning sessions, often relying on a handful of standout experiments to make their case.

As a PM, every quarterly planning cycle, I was compiling the same table of results. I needed to aggregate what experiments we ran and how they impacted our north star metric in front of leadership. But leadership wasn't in the weeds, and this data always seemed new to them. At planning time, this data couldn't break through prior assumptions already formed.

The Performance Scorecard changes this by providing:

• 1. Holdout-based Aggregate Impact: A robust measure of your program's total impact, based on rigorous holdout experiments.
• 1. Bayesian Aggregate Impact Estimate: A new feature that allows teams without holdout capabilities to estimate their overall impact accurately.
• 1. Top experiments: A table view of top experiments for the expected measure shipped in the timeframe specified. With these tools, you can now answer the critical questions: "What is the total value our experimentation program has delivered?" and "What experiments impact our core metrics?" This data is invaluable for securing continued support and resources from leadership. Best of all, this view is always available, making it easy to share on a consistent basis, and not just at planning time. This means you don’t just get to report on success driving impact, you get to celebrate it!

Velocity: Aligning Your Organization on Input Metrics

While outcomes are crucial, the Performance Scorecard also focuses on the inputs that drive those results. Experimentation velocity is one of those key inputs.

Once we decided on a quarterly goal, as the PM I would always advocate for loading up the team’s roadmap with a number of projects I believed would move that metric, with the one I believed would have the biggest impact first. Even if that project failed, the team would have more shots on goal before the quarter was over to move that metric.

With the Performance Scorecard, leaders can track if their teams are also taking enough shots on goals. By tracking experiment velocity, you can:

• Set and monitor organization-wide goals (e.g., running 10 tests per quarter)
• Identify teams that may need additional support or resources
• Encourage a culture of continuous experimentation across your organization

Quality: Ensuring Rigorous Experimentation Practices

The quality of your experiments is just as important an input as their quantity. The Performance Scorecard helps you maintain high standards by tracking key quality metrics. This feature addresses common misconceptions and ensures that your organization is following best practices.

For example, I worked for a leader who suggested running experiments with a 20% control and 80% variant split to get new features to users faster. While well intentioned, this actually led to slower experiments and lagged velocity, as a 50%/50% split gives much more signal and ultimately a faster decision.

The Quality section of the scorecard helps you:

• Monitor experiment design parameters across your organization
• Identify and address potential issues before they impact your results
• Educate stakeholders on the importance of rigorous experimental design

Win-Rate: Testing the Right Ideas

According to research, the success rate of experiments ranges from 8% to 33%. Over a large sample of experiments, this is what most teams should be. Yes we see many teams that fall outside this range.

If a team’s win rate is below what is expected, that indicates that the hypotheses aren’t good enough, and perhaps not founded in good customer insights. On the other end, if a team’s win rate is above this range, the experiments usually don’t have enough impact. These are easy and small ideas that win but don’t move the needle enough to achieve goals set.

With the Win-rate section of the scorecard you can:

• Understand how team win-rate compares to industry benchmarks
• Celebrate teams that are shipping a mix of successful, neutral, and unsuccessful experiments
• Investigate if a team has unusually high or low win-rates

Empowering Program Leaders and Winning Executive Buy-In

The Performance Scorecard is more than just a reporting tool – it's a catalyst for building a true culture of experimentation. By providing program leaders with a comprehensive view of their experimentation efforts, we're enabling them to:

• 1. Evaluate performance across teams and over time
• 1. Identify areas for improvement and optimization
• 1. Demonstrate the value of experimentation to skeptical stakeholders
• 1. Make data-driven decisions about resource allocation and program direction

For executives, the Scorecard offers a clear, holistic view of the experimentation program's impact on the business. This transparency fosters trust and encourages continued investment in data-driven decision-making.

Transforming Experimentation from a Tool to a Strategy

With the introduction of the Performance Scorecard, Eppo is taking experimentation to the next level. We're moving beyond individual tests to create a comprehensive system for measuring, monitoring, and optimizing your entire experimentation program.

By providing insights into aggregate impact, velocity, and quality, the Performance Scorecard empowers you to build a more effective, more efficient, and more impactful experimentation culture. It's not just about running tests anymore – it's about transforming experimentation into a core strategic advantage for your business.

Ready to elevate your experimentation program? Contact us today to learn more about the Performance Scorecard and how it can drive growth for your organization.

Why we raised our Series B, and why experimentation is more important than ever in an era of AI and efficient growth

Eppo's Founder and CEO, former early data scientist who built experimentation tools and cultures at Airbnb and Webflow

Today, Eppo is announcing our $28M series B financing, led by Davis Treybig at Innovation Endeavors with participation from Preeti Rathi at Icon Ventures.

Since our last funding, Eppo has become synonymous with large experimentation ambitions. Category-leading companies like Twitch, DraftKings, and Coinbase use Eppo to supercharge their experimentation. So do generative AI pioneers like Descript and Perplexity. Eppo customers are running experiments across their businesses with use cases spanning product, marketing, and AI.

When looking who to partner with on this round, Davis and the IE team were ultimately a no-brainer among our offers. Davis is the author of the best-researched piece on experimentation from anyone in venture capital, "The Experimentation Gap", outlining its clear connections to AI and use cases beyond digital products. We’ve been partnering with Davis and IE since our seed funding, and have been continuously impressed by their ability to build relationships with researchers on the frontiers of technology, and companies who see where this technology fits in their stacks.

Our Series B comes at an interesting inflection point for the tech industry, a new era of change where experimentation is the clear differentiator between which companies will thrive, and which will struggle.

Here’s why experimentation is more important than ever, and why we raised this Series B:

The two conversations happening in every boardroom

If there are two things every company is discussing right now, they are efficient growth and AI.

Rising interest rates and scarce capital have squeezed companies to the point that even a $500M ARR company with 30% growth trades at only 15x multiples. Teams were stripped down via layoffs, SaaS spend was consolidated, and growth venture investments have cratered. Growth is still an imperative, but must be done with scalable economics instead of growth at all costs. Put another way, it’s never been a greater existential risk to not know which company initiatives are growth levers vs. low-ROI money pits.

The second conversation is around AI. Even as companies clamp down on spend, AI budgets live outside of financial discipline. It’s for good reason: CEOs have seen astronomical increases in efficiency for CoPilot-augmented software development, creative asset development, and knowledge management. CEOs have all taken the time to imagine an AI-native competitor, and how they’d fare against them. GenAI technology presents a clear opportunity to leapfrog competition - or be leapfrogged.

Both of these factors have led to a blossoming in our experimentation space. Running experiments is the simplest path to high conviction on which products, campaigns, and AI strategies are successful, and which need to pivot or wind down. This market landscape is Darwinian: those who experiment, adapt, and swiftly refine their strategies are the ones who succeed. They accelerate their winning initiatives while quickly abandoning the ones that don't work.

Companies now expect more from experimentation

The point of experimentation is to drive velocity, growth, and innovation. Most companies aren’t there yet. They have low horsepower; bottlenecked experimentation stacks that don’t actually power velocity, growth, or innovation.

The biggest change since I started Eppo is companies demanding more from their experimentation investments.

Legacy experimentation vendors can’t deliver the vision. If a company buys a marketing-focused tool like Optimizely, they quickly figure out that the only supported tests are simple website changes. If they buy a feature flagging-centric tool like LaunchDarkly, they realize that the “experimentation” is a shallow coat of paint on a narrow DevOps tool. Teams using these tools end up stunted, spending more time and money on tedious manual efforts and expensive supplementary tools to fill the gaps.

The result is an inadequate trickle of experiments that are never quite trusted either.

In contrast, the tech giants with modern tooling are winning the era of efficient growth and AI. Instead of slowly spinning up button color tests, companies like Microsoft, Netflix, and Eppo customers are running experiments that can generate revenue and change strategic paradigms:

• An AI team saved $5M+ of spend by proving that open source LLM models could match the performance of an expensive GPT model they previously used. Now, all GenAI models are tested for ROI instead of implicitly trusted on brand name.
• At Netflix, simple UX changes are completely automated. Thousands of tests on show artwork are designed, set up, and adjudicated by algorithm.
• At Airbnb, we experimented on a sales team, holding out a random set of markets from their work and seeing if the sales-worked markets grew faster than the holdouts. The team was ultimately disbanded and reassigned, saving headcount cost and increasing strategic focus.
• A company spending tens of millions of dollars a year on YouTube ad campaigns tested whether the spend was doing anything by zeroing out the spend in a select group of geographies and comparing their performance.

In short, these companies are able to experiment pervasively, quickly, and with leadership trusting the results. They’ve built the accessibility and governance required to make any test possible and make experimentation like water: easy, continuous, expected.

Legacy tools like Optimizely or LaunchDarkly look nothing like the workflows that enable market leaders to evaluate large, expensive campaigns, or run all product development through test and learn iterations. With programs and software spend under tight budget scrutiny, the bar is now set much higher.

The age of AI will be an age of experimentation

We didn’t predict the explosion in AI capabilities at the end of 2022, but it created a massive appetite for experiments. There’s a short-term need to evaluate AI model ROI, and a long-term need to evaluate more ideas in general. AB testing is the primary solution for both.

Companies now have a firehose of new GenAI model generations at their fingertips, each reaching new heights and new, higher price tags. A GPT model release gets quickly followed by a new Claude, new Llama, and a host of open source models. As the New York Times aptly put it, AI has a measurement problem: companies have no idea which models are most accurate and provide the best user experience.

With AI capabilities in cloud APIs, the switching cost of these models is near zero. A simple feature flag can be repurposed to a routing system for AI model vendors — frictionless swaps of which API to use. This means that companies with good experiment infrastructure can get results that are more powerful and far cheaper with little effort.

Companies with better infrastructure can go further, multiplexing across an ensemble of gen AI models. Maybe pay up for a premium Claude 3.5 model for high-value users, and save money with open source models on Free tier users. There are wide disparities in price across the LLM clouds, and experimentation gives companies the edge to discern between real performance gains and spending that should be cut.

But there’s an even more interesting long-term trend. GenAI is about to exponentially increase the number of ideas generated and implemented. All of the necessary pieces to produce a new product or a new campaign concept are already levered by AI:

• AI models are great for brainstorming ideas, crowdsourced from our collective intelligence
• AI models can whip up creative assets easily, even strikingly realistic photography
• AI models can implement ideas in code, not just to multiply the output of engineers but even to enable less technical users to implement changes themselves

With just these existing capabilities there will be 10x more product implementations and 10x more marketing campaign concepts, which will all need to be evaluated before they are rolled out.

As the cost of ideas goes to zero, the cost of evaluating these ideas becomes the new bottleneck. Companies hoping to leverage this AI explosion will need experimentation infrastructure that can handle 10x more volume and use cases.

We raised $28M to create experimental companies

Winning companies thrive in an era of change with innovative experimentation. There’s a reason why Jeff Bezos talks about experimentation in every speech, why Netflix ran experiments as early as their DVD mailing days, why Mark Zuckerberg established AB testing on their growth team while still only operating at a handful of colleges. The companies that outcompeted and won their markets are highly experimental.

Our ambition is to change corporate culture everywhere, unleashing their best ideas with a broad experimental mindset. We’re excited to bring more partners and fresh funds to our mission.

Run an experiment in Eppo!

We’d love to show you what we’ve built.

Request access to Eppo and we’ll help you get a few experiments set up.

Join the team

Our team is made up of veteran product builders from Airbnb, Snowflake, Slack, Amazon, and Stitch Fix. We’re on a mission to change corporate culture everywhere. Have a look at our open jobs. We’d love to meet you.

Today, we’re excited to launch Layers. This release combines the flexibility of Eppo feature flags with the structure to easily coordinate concurrent tests. There are scenarios where you may want to test different variations simultaneously on the same area of your product. However, allowing these experiments to overlap can lead to conflicts and degrade the user experience. Eppo's new Layers functionality solves this by enabling you to create dedicated spaces for running mutually exclusive experiments.

A More Structured Approach

Previous approaches to handling concurrent tests involved complex coordination or creating dependencies between flags. With Layers, you can easily set up a structured environment that keeps your experiments isolated while providing controls over traffic allocation and ordering of experiments. Each Layer acts as a contained space with the following levels:

• Experiment opt-out rule - set an exclusion group here that always sees the same experience, such as internal users that always see the new experience
• Experiments - use this level to serve concurrent experiments with the ability to set the traffic exposure of each experiment
• Rollout - once an experiment concludes, you can easily select the winning variant and roll it out to all unallocated users
• Default serving rule - set the control variation users see when not exposed to an experiment or rollout

This streamlined workflow minimizes coordination overhead and ensures your experiments don't interfere with each other's results, while giving you the power to exclude users and roll out winning variations as soon as they’re selected.

Rigorous Controls, Simplified Process

Eppo applies the same powerful statistical engines and guardrails to experiments run in Layers as our core experimentation product. You'll get automated diagnostics, traffic balance monitoring, and our full suite of analysis tools to properly measure impact. Creating a new Layer is simple - just navigate to Configuration, enter details like the Layer name and parameters, and you're ready to start adding experiments. Need to roll out a winning variation? With one click, you can update the Layer's default experience.

Utilize Parameters to Create No-code Experiences

Parameters are the core building blocks that allow you to define the specific elements you want to test variations of within a Layer. For example, let's say you want to experiment with different messaging headlines on your product's homepage. You could create a Parameter called "headline_text" and set a default value. Then, as you set up each experiment variation within that Layer, you can specify alternative text values to test. Parameters can accept various data types like strings, numbers, booleans etc. This flexibility enables you to test everything from UI styling and design elements to pricing variants and feature configurations - all within the same Layer construct. Additionally, this means parameter values can drive experience changes without updating code. This no-code solution makes it easy to increase the number of variations you test and experiments you run to find the optimal solution to your hypothesis.

Experimenting Together

Whether it's a product and marketing team collaborating on a page, or parallel AI experiments, Layers enable your organization to maximize learnings through disciplined, concurrent experimentation. Interested in using Layers? Eppo customers with Feature Flags can start leveraging this feature immediately. If you're new to Eppo, we invite you to request a demo to see how Layers can enhance your experimentation program.

Finding Eppoʼs new voice

We're excited to introduce Eppoʼs rebrand — a new look that highlights our commitment to transformative innovation, customer partnership, and inspiring the experimentation community.

Eppo was founded on the realization that every company has untapped entrepreneurial potential. Our first product was our world-class Experimentation analysis suite, which makes rigorous statistical inference accessible and automates away tedious manual analysis so that data scientists can focus on driving experimentation culture in their company. Over the course of the last three years, Eppo has grown significantly - we now process over 5 billion daily assignments for customers, and our headcount has doubled in the last six months alone. With growth comes new ideas... and growing pains. We have expanded to new use cases and products: Feature Flagging for engineers and product teams running experiments, Contextual Bandits for personalization and lifecycle marketing integrations for marketing teams. Our language wasn't expansive enough to resonate with all our new personas. We also know that our current customers are passionate, even fervent about our approach to rigor and customer-centricity, but it took effort to understand Eppoʼs differentiation in the market. It was clear to us that we needed a new brand identity to encompass all of this growth and differentiation.

Built on customer beliefs

“Brand is a promise delivered." (What Great Brands Do, Denise Lee Yohn) We treated this rebrand like any design project — our first question was “What data points should we consider?ˮ. Having led many rebrands in the past, I've seen the common trap of leaders latching onto a famous brand to copy (Apple, Nike, and Uber are the common choices), or projecting what they wish the company to be for them. We were more interested in reflecting what our customers believed about us, rather than the other way around. We spoke to some of our most enthusiastic customers as well as respected peers and advisors in the experimentation space to understand why they chose (or appreciate) Eppo. Here are some of the ways they described how they see Eppo in the space:

• “Eppo is built for companies that take data seriously and want to have experimentation has a cultural element. Eppo is rigorous, genuine, and fun.”
• “If Eppo were a car, theyʼd be a colorfully painted vintage Mini. It has quiet confidence, the underlying tech is solid, itʼs lovingly maintained and perfectly designed.” What stood out to us was:
• Our customers are transformational leaders who challenge the status quo, push boundaries, and innovate old ways of working.
• Eppo delivers a new way of running a business: one where more ideas are executed, learning velocity is accelerated, and teams operate with agility and efficiency. Lastly, we wanted to reflect not just what our promise is but also how it is delivered. We know that just having a tool like Eppo wonʼt create cultural change. Our key advantage is the wealth of experimentation expertise at Eppo and the diversity of companies we built it from — and we personally bring this to our customers daily. We are close partners with our customers and community, providing guidance and enabling them to make confident decisions and rallying their organization around this new way of working.

Helping transformational leaders win

The new Eppo brand brings the spirit of athleticism to the challenge of transforming a business. It's about the aspiration to win — and in this new age of constant change, the experimenters will win. We're here to give them not just the tools and guidance, but the inspiration and confidence to succeed. The new Eppo brand brings the spirit of athleticism to the challenge of transforming a business. It's about the aspiration to win — and in this new age of constant change, the experimenters will win. We're here to give them not just the tools and guidance, but the inspiration and confidence to succeed. Visually, we drew inspiration from classic explorations and games, most notably Conway's Game of Life and FEZ. They both exude dynamism, a sense of precise engineering, and the constant creation of new perspectives via transformations. ‎‎ ‎Our new logo nods to that constantly evolving spirit of experimentation, and to the “precisely engineeredˮ nature of Eppo that our customers highlighted. The sense of motion and momentum bolster the promise of transformation for our customers, and our new messaging is clearer and more accessible, giving multiple entry points to new customers to explore our offerings. ‎ As we move forward, we are excited to continue delivering on our promise of culture transformation, innovation, and partnership under the refreshed Eppo brand.

Introducing Perplexity as an Eppo customer

We’re thrilled to welcome Perplexity as an Eppo customer, and are even more excited to announce Perplexity Enterprise Pro free for all Eppo customers!

Perplexity launched their Enterprise Pro plan last month to give teams enhanced data privacy, SOC2 compliance, SSO, and more. To celebrate this milestone, we're offering 3 free months of Perplexity Enterprise Pro for up to 10 seats.

To redeem this offer, please reach out to Sid Sharma ([email protected]) at Eppo.

Eppo powering Perplexity and their next phase of growth

Perplexity is one of the fastest-growing AI companies ever, with top-tier customers like Stripe, Snowflake, Databricks, and Vercel. Eppo is pivotal in enabling Perplexity to test key product features and evaluate their models over the last few months while staying focused on core innovation. As Alexis Weill, Head of Data at Perplexity, puts it:

“Eppo has been instrumental in transforming our company's approach to experimentation. Their platform has helped us to foster a culture of testing and learning. With Eppo, we've been able to significantly scale the number of experiments we run concurrently, allowing us to make data-driven decisions faster."

Contact us to start using Perplexity Enterprise Pro.