By Will Burghes, Head of Professional Services, DV Rockerbox™
In a previous post, we talked about how to read the confidence interval once your incrementality test is done. That's a valuable skill. But the tests that produce the clearest, most useful confidence intervals have something in common: the work that made them trustworthy happened weeks before launch, not during the readout.
The best incrementality tests aren't lucky. They're designed to succeed. Here are five rules we lean on with clients to make that happen every time, before a single dollar of media is paused.
1. Get clear on the decision you're testing for
Before you touch a single design parameter, answer one question: what decision will this test inform? Not “what will we learn?” but what will you do differently depending on the outcome?
A testable decision might be "double the CTV budget if iROAS clears 2.5,” or “pause social if the lift is indistinguishable from zero." It points directly to how the test should be built. Naming the decision up front is what turns a test result into an action, and it's worth doing before anything else on this list.
2. Know the MDE threshold you need, before you design toward it
The minimum detectable effect (MDE) is the smallest lift your test design can reliably observe, given the data volume, test duration and cell sizes. It should be the first thing you calculate, not something you check after the results come in.
If you do the MDE math only after the test, it's easy to read a null result as "the channel doesn't work," when the real story is that the test was never capable of finding a lift as small as the one the channel plausibly delivers.
For example: You plan to spend 10% of your total budget on paid social, but the MDE for your test is 20%. Is it realistic to believe the channel will deliver a 20% revenue increase at that spend level? If you adjust for this before the test launches you still have time to extend the duration or adjust the spend between test and control, rather than once the test is already live.
3. Your holdout group probably needs to be bigger
Once you know how sensitive your test needs to be, size the markets from which you deliberately withhold your ads accordingly. The instinct on every test is to shrink the holdout to minimize business impact. Fewer markets paused; less revenue set aside; a smaller window of reduced spend. It feels like the responsible choice, but is often the opposite.
Your ability to detect a real effect depends heavily on how well you can model a counterfactual from your control. Say a retailer wants to test a channel in just five markets. A footprint that small might only have a revenue correlation of around 0.6 – 0.7 with all of their other markets, which would create a test noisy enough that it could only detect a lift of 20% or more. For most channels, that's an unrealistic bar. A channel could be delivering a strong 10% or 15% lift, and the test would still come back looking like nothing happened.
Scaling the same design up to 20 markets could push that correlation above 0.9. At that level, the test could reliably detect a lift of under 5%.
And what often gets missed is that a bigger holdout costs far less than it seems.
Try the math yourself: multiply the channel’s incremental contribution by its share of total marketing spend, times the share of markets in your holdout, times the fraction of the year the test runs.
Say a channel drives a genuine 30% incremental lift, represents 20% of your total marketing spend, and you hold out 20% of your markets for one month out of twelve:
For a $10 million brand, that's roughly $10K of foregone revenue in exchange for a reliable answer to whether a channel is working. That answer could justify a far larger investment, or it could free up the budget to move elsewhere. Both of those actions could result in a positive business impact far larger than $10K.
A generous holdout is a small price for a real answer.
4. Build the test around your business, not a template
There's no universal test design. The strongest tests are built around what actually drives your numbers. Every business has its own set of confounders that can shift KPI independent of the channel you're testing. Promotions, hero SKUs concentrated in a handful of markets, regional demand differences, competitor activity, seasonality, store openings: all of it is noise your test needs to control for, and that noise looks different for every business.
So before you finalize a design, map the idiosyncrasies: what could plausibly move the metric you’re measuring, in your geographies, during your test window? Then build your controls around those specifics. A geo-holdout that works cleanly for a national ecommerce brand might not work for a physical retailer with regional pricing, a regional promotion calendar or a product concentrated in a few DMAs. The test design isn’t the deliverable; a clean read on incremental impact is. Getting there means mapping your business’s nuances first, then designing the controls that account for them.
5. Execute with the same rigor you designed with
A well-designed test executed loosely gives you the same result as a weak design: an answer you can't fully trust. The execution step gets the least attention and can pose a risk to the test, so it deserves the same care as the design phase.
A few specifics worth building in from day one:
For geo tests, use presence-based ads, not interest-based ads. In interest targeting, platforms serve ads to users who show interest in a region (say by searching for flights to that place), not those who live there, which can quietly erode the line between your test and holdout cells. Presence targeting keeps spend tied to physical location, which is what a geo test depends on.
Watch for channels optimizing their way into your holdout. Many platforms' auto-bidding will chase efficient conversions wherever it finds them, including inside the regions you meant to exclude. Where this is a risk, split the campaign into budget-capped, duplicate campaigns that mirror your test design, one capped campaign per cell, rather than relying on a single campaign's geo exclusions. That keeps the platform from reallocating spend between your test and control cells.
Beyond that, watch for spend changes, out of stock locations or demand outliers that appear in the test, and record start and end dates carefully.
The bottom line
Your ability to make confident decisions at the end of a test reflects the care that went into the test design. The strongest test starts with a clearly named decision, has a realistic MDE, treats a generous holdout as an investment rather than a cost, and builds around the specific dynamics of your business.
Get those things right and you'll have a result worth acting on, whatever it turns out to be.
Next Steps
Contact DV to learn how DV Rockerbox can help you design an incrementality test you can trust.