Strategy & Tracking
Incrementality testing: proving ads actually cause sales
Your platform dashboard counts every booked job that touched an ad, including the homeowners who would have called anyway. Incrementality tests measure only the jobs your spend caused, and advertisers routinely find 30 to 60 percent of attributed conversions were never incremental.
Attribution answers the wrong question. It tells you which ad a homeowner touched, not whether the ad changed their decision. A retargeting pixel takes credit for someone who already had your estimate form half-filled; a brand keyword takes credit for a homeowner who typed your company name after a neighbor's referral. Incrementality testing settles it with a holdout: show ads to one group, withhold them from a matched group, and measure the difference. When eBay ran this on paid search, branded ads returned negative 63 percent (Blake, Nosko, and Tadelis, Econometrica, 2015). The levers below cover geo holdouts, conversion lift, ghost ads, and a holdout a small budget can actually run.
Why attribution over-credits the ads that needed it least
Every ad platform reports conversions, and every platform has a reason to count generously. The model gives credit whenever an ad was served before a sale, whether or not the ad moved the buyer. That systematically over-pays the channels that intercept demand instead of creating it. Branded search and retargeting look brilliant in a dashboard precisely because they reach people already heading to checkout. Measured causally, those channels routinely post an incremental ROAS far below their platform-reported number, often a fraction of it.
The gap is not small. In documented incrementality work, platform-reported ROAS for demand-harvesting channels can overstate true incremental return by half or more. Uber made the point at scale: it turned off about $100 million of a $150 million budget and saw no drop in rider installs. The paid installs simply reappeared as organic. Attribution had been billing for traffic that was coming anyway.
The geo holdout: the workhorse test
A geo holdout splits your market by region. You keep advertising in test markets, pause or hold spend in matched control markets, then compare total sales between them. Because it reads against all sales, not just tracked clicks, it survives cookie loss and works for channels you cannot pixel. Meta's open-source GeoLift and Google's Meridian both build on this design. The output is the number that matters: incremental revenue and incremental ROAS, the lift your spend actually caused rather than the conversions it sat next to.
The discipline is in matching. Control regions must track test regions historically, the test has to run long enough to clear noise, and you size it against a minimum detectable effect so a real lift is not buried in variance. The result usually carries a confidence interval, which is the honest part. WellBuilt designs geo tests so the comparison holds and the readout survives scrutiny, then translates the lift into budget moves you can defend.
What a sound geo holdout needs:
- Control markets whose sales history closely tracks the test markets
- A run length long enough to clear weekly and seasonal noise, typically three to six weeks
- A minimum detectable effect set before launch so you know what lift you can actually see
- A clean pre-period baseline to model the counterfactual against
- Incremental revenue and iROAS as the output, reported with a confidence interval
Ghost ads and PSA tests: holdouts at the user level
Where geo tests split markets, user-level tests split people. The old method is the PSA test: the control group sees an unrelated public-service ad instead of yours, so both groups are equally 'advertised to' and you measure the difference. The cleaner method is ghost ads. The platform runs the auction as normal, records that your ad would have won the impression for a control user, then withholds it. You get a true counterfactual without paying to serve placebo creative, which is why it has become the standard for conversion lift.
This is what Meta Conversion Lift and Google's user-based lift run under the hood: a randomized holdout, usually 5 to 10 percent of the audience, with conversions compared across cells. The honesty cuts both ways. Meta has added an Incremental Attribution setting that optimizes delivery toward the conversions a lift test would credit rather than every attributed one, evidence that even the platforms now separate incremental outcomes from attributed ones.
Attribution tells you which ad a buyer touched. Incrementality tells you whether the ad changed their mind. Only one is worth paying for.
The branded-search and retargeting trap
Two line items fail incrementality tests most often, and they are usually two of the largest. Branded search bids on your own name, then claims the click from someone who would have found you organically. eBay halted branded search in a controlled experiment and watched nearly all the lost paid clicks reappear as free organic visits; the measured return was negative 63 percent. The mechanism is cannibalization: the paid ad takes credit for a click the organic listing would have captured for free.
Retargeting has the same disease. It reaches homeowners who already started an estimate request, so platform ROAS looks spectacular while incremental ROAS collapses. A useful rule of thumb: when a channel's measured incremental ROAS lands far below its platform-reported figure, it is largely harvesting demand you already had, and the budget should be trimmed or moved. Contractors with strong reviews, referrals, and an established local reputation see this most on retargeting, where the ads mostly intercept homeowners already on their way to calling.
Why MMM came back after the cookie
As third-party cookies and signal loss broke user-level tracking, brands rediscovered marketing mix modeling. MMM uses aggregate, top-down data, spend and sales over time, so it needs no individual identifiers and survives a privacy-first world by design. A 2024 eMarketer survey found 53.5 percent of US marketers now use MMM, and the platforms have leaned in: Meta open-sourced Robyn in 2023, and Google released its Meridian model on January 29, 2025.
MMM and experiments are partners, not rivals. Modeling tells you the big-picture contribution of each channel; geo and lift tests calibrate it with ground truth, anchoring the model to a number you proved rather than one you fit. Run on its own, an MMM can drift. Calibrated against holdout results, it becomes a planning tool you can trust between tests. With more than half of US marketers now using MMM, the discipline has moved from research curiosity toward default practice.
A holdout a small budget can actually run
You do not need a large platform-run Conversion Lift study, which carries a substantial minimum spend over the test window, to measure lift. The cheapest experiment is a structured pause. Pick one suspect line item, branded search or retargeting are the usual culprits, and turn it off cleanly for two to four weeks while holding everything else steady. Watch total revenue and organic and direct conversions, not just the platform's number. If overall sales hold while the platform's attributed conversions fall, those conversions were never incremental.
For something more rigorous, a lightweight geo holdout works on modest budgets: split a handful of matched cities, pause the channel in half, and compare. Tools like GeoLift size the test for you. Whatever you choose, decide the success threshold before you start and let the test run its full window. WellBuilt runs these as paired pause-and-geo experiments so the result is causal, not a coincidence of timing.
Key takeaways
- Treat platform-reported ROAS as a ceiling, not a measurement; the incremental number is almost always lower and is the one worth budgeting against.
- Test branded search and retargeting first; they cannibalize organic demand most, and eBay's branded search returned negative 63 percent once measured experimentally.
- Run a geo holdout for channels you cannot pixel, matching control regions to test regions and reporting incremental ROAS with a confidence interval.
- Use ghost-ad conversion lift for user-level tests; it gives a true counterfactual without paying to serve placebo PSA creative.
- Calibrate your marketing mix model against holdout results so planning rests on a number you proved, not one you fit.
SourcesBlake, Nosko & Tadelis, Consumer Heterogeneity and Paid Search Effectiveness, Econometrica, 2015 · WARC / The Hustle, Uber ad spend and attribution fraud (Kevin Frisch), 2021 · Meta Business Help Center, About Conversion Lift and Incremental Attribution, 2024 · Meta GeoLift, open-source geo-based incrementality measurement, 2024 · eMarketer / Snap, US marketer MMM adoption survey (53.5%), July 2024 · Google, Meridian open-source MMM launch announcement, January 29, 2025 · Meta Marketing Science, Robyn open-source MMM, 2023
Questions, answered straight.
What is the difference between attribution and incrementality?
Attribution assigns credit for a conversion to an ad the buyer touched; incrementality measures whether the ad caused the conversion at all. A retargeting ad can be attributed a sale that would have happened anyway, which inflates its reported ROAS. Use attribution to see paths, and incrementality to decide what to fund.
How much non-incremental spend do these tests usually find?
It varies by channel, but the over-counting is large and consistent. Uber switched off two-thirds of its budget with no drop in installs, and eBay's branded search measured a negative return once tested. Start with branded search and retargeting, where the gap between reported and real impact is widest.
Can a small budget run an incrementality test?
Yes. A full platform-run Conversion Lift study carries a steep minimum spend, but a clean two-to-four-week pause of one suspect channel costs nothing extra and reveals whether sales hold without it. A lightweight geo holdout using a tool like GeoLift works on modest budgets too. Decide your success threshold before you start.
Does marketing mix modeling replace incrementality testing?
No, they work together. MMM gives a privacy-safe, top-down read on every channel's contribution, which is why 53.5 percent of US marketers now use it, while geo and lift tests provide the ground-truth lift that calibrates the model. Run experiments to anchor the model, then use the model to plan between tests.
Strategy & Tracking
Want this run for you, not just read about?
Clean tracking and honest attribution, so you know which dollars actually produce revenue.