How to Run a Geo Holdout Test for Paid Media

By Postlet ·

Short answer

A geo holdout test pauses paid media in matched geographic regions while continuing in control regions, then compares conversion rates and revenue between them against a pre-test baseline [2]. The test requires matched markets with similar population, historical conversion rates, and no cross-market contamination [1][2]. Run the holdout for a defined period, then measure the difference in outcomes to determine true incremental lift [5].

Why geo holdouts produce the cleanest causal evidence

Platform ROAS tells you what your ads touched, not what they caused [3]. Geo holdout tests produce the cleanest causal evidence for enterprise paid media programmes because they pause a campaign in matched regions and compare revenue against control regions, producing a causal read that is independent of the platform reporting the numbers [5].

The protocol is simple in description: a campaign is paused in a set of geographic regions for a defined period, while continuing to run in matched control regions [5]. The comparison reveals what the ads actually drove, rather than what would have happened anyway.

Finance teams find geo holdouts the easiest incrementality method to defend because the test design is transparent and the control group provides a clean counterfactual [5].

Select matched geographic regions before anything else

You need a test region and a control region with similar population size, similar historical conversion rates, and no significant organic traffic overlap between them [2]. For US-based campaigns, DMA (designated market area) boundaries provide clean geographic separation [2].

For geo tests, markets must be matched on spend-per-capita, category penetration, historical conversion rate, and seasonal index [1]. The most common mistake is convenience market selection, such as deciding to turn off ads in the Midwest without verifying that region matches the control on these variables [1].

Geographic self-selection produces invalid controls [1]. Choose markets with minimal cross-market traffic and clear geographic separation to avoid contamination between test and control [4].

Audit for leakage across channels and markets

For geo holdouts, verify that no national TV, podcast, or digital channels bled into control markets [1]. The holdout only measures incrementality if the paused channel is actually absent from the holdout region.

Confirm the holdout segment was not receiving ads through other campaigns [1]. If your brand runs multiple campaigns across platforms, check that all of them respect the geographic boundary.

Contamination between test and control is the second most common pitfall after poor market matching [4]. Even small leakage invalidates the comparison by giving the control region partial treatment.

Set the holdout duration and sample size

Geo holdout tests typically run for 30-60 days [2]. Shorter tests risk noise from week-to-week variance, while longer tests increase the cost of paused spend and the risk that external factors change between regions.

One practical approach is to run one geo holdout per quarter on your two largest channels [6]. This balances rigour with the operational cost of pausing spend and managing the test.

Some teams run holdouts for 2-4 weeks [6]. The right duration depends on your conversion cycle and the volume of conversions needed to detect a meaningful difference between test and control regions.

Pause spend in the holdout region and run the test

Run your campaigns at normal spend in the test region, and pause all spend or reduce to near zero in the holdout region for the test period [2]. The holdout region becomes the counterfactual: what happens when the ads are absent.

During the test window, do not adjust targeting, creative, or budget in the control region. Any change to the control introduces a confound that makes it impossible to isolate the effect of the paused spend.

Record the start and end dates, the exact geographic boundaries, and the baseline metrics for both regions before the test begins. You will compare post-test performance against this baseline to calculate lift.

Measure the difference against the pre-test baseline

Compare conversion rates and revenue velocity between regions against a pre-test baseline [2]. The incremental lift is the difference in outcomes between the control region (where ads continued) and the holdout region (where ads were paused), adjusted for any pre-existing differences.

If the control region saw no meaningful difference in conversions or revenue compared to the holdout region, the channel produced little or no incremental lift. If the control region significantly outperformed the holdout, the difference represents the true causal effect of the ads.

Postlet measures holdout results against a matched or synthetic control, with intervals and corrections for multiple tests running at once, so a lucky week is never mistaken for a win [Postlet fact sheet]. The platform re-derives every live conclusion against new data, and when an effect decays, the finding is withdrawn before spend keeps following a result that has stopped holding [Postlet fact sheet].

Common mistakes that invalidate the test

The first mistake is selecting markets for convenience rather than statistical comparability [1]. If the holdout region has a different customer base, seasonality, or historical conversion rate than the control, the test measures those differences rather than ad incrementality.

The second mistake is failing to audit for cross-market contamination [1][4]. National campaigns, podcast ads, or even organic social posts that reach both regions introduce treatment into the holdout and bias the result toward zero.

The third mistake is changing the control region during the test. Budget shifts, creative updates, or targeting changes in the control region make it impossible to attribute differences to the holdout alone.

The fourth mistake is reading results without accounting for natural variance. A difference that could easily occur by chance is not evidence of incrementality.

When to run a geo holdout instead of other incrementality tests

Geo holdout tests are the workhorse method for ecommerce brands [3]. They work when you have enough geographic distribution to split into matched markets, and when the cost of pausing spend in one region is acceptable for the duration of the test.

User-level holdouts managed by the platform are faster to run but require random assignment managed by the platform, never self-selected [1]. They also depend on the platform's reporting, which is the same system you are trying to validate.

Geo holdouts produce a causal read that is independent of the platform reporting the numbers [5], which makes them more defensible when presenting results to finance or executive teams.

Frequently asked questions

How long should a geo holdout test run?
Geo holdout tests typically run for 30-60 days [2], though some teams run them for 2-4 weeks [6]. The right duration depends on your conversion cycle and the volume of conversions needed to detect a meaningful difference. Shorter tests risk noise from week-to-week variance, while longer tests increase the cost of paused spend.
What is the most common mistake in geo holdout test design?
Convenience market selection is the most common mistake [1]. Teams choose a holdout region without verifying it matches the control on spend-per-capita, historical conversion rate, category penetration, and seasonal index. Geographic self-selection produces invalid controls and measures regional differences rather than ad incrementality [1].
How do I prevent contamination between test and control regions?
Verify that no national TV, podcast, or digital channels reach the holdout region [1]. Choose markets with minimal cross-market traffic and clear geographic separation [4]. Confirm the holdout segment is not receiving ads through other campaigns [1]. Even small leakage invalidates the test by giving the control region partial treatment.
Can I run a geo holdout test if I only advertise in one country?
Yes. For US-based campaigns, DMA (designated market area) boundaries provide clean geographic separation [2]. You need at least two comparable regions: one where ads continue and one where they are paused. The regions must have similar population size, historical conversion rates, and no significant organic traffic overlap [2].
What should I measure to determine incrementality?
Compare conversion rates and revenue velocity between the control region and the holdout region against a pre-test baseline [2]. The incremental lift is the difference in outcomes, adjusted for any pre-existing differences. If the control region shows no meaningful difference compared to the holdout, the channel produced little or no incremental lift [5].

Sources

  1. [1] Holdout Test in 2026: Incrementality Measurement That Actually Works
  2. [2] Incrementality Testing for Paid Media Teams: A Field Guide
  3. [3] Incrementality Testing for Paid Media: A Practical Guide
  4. [4] Incrementality Testing in Paid Media Without Wasting Ad Spend - Growth Rocket
  5. [5] Incrementality Testing in Paid Media: A Practical Guide
  6. [6] Incrementality Testing: Geo Holdouts & Lift Studies Guide

Researched and drafted with AI from the sources above. Figures come from those pages or from Postlet's own site; check the source before relying on one.

Want an agent that runs these tests on your own account?