Geo Testing: Unlocking True Incrementality in Marketing and Product Experiments

Experimentation is one of the most reliable ways to understand what truly drives impact. But traditional A/B testing isn’t always possible - especially in marketing campaigns, marketplace environments, or cases where user-level randomization can’t be applied. In these scenarios, geo testing offers a practical and rigorous alternative.

In our latest webinar, Michael Makris (Statsig) and I shared lessons from years of running geo experiments with companies like Uber and leading marketing teams - showing how synthetic controls enable accurate measurement of incremental impact in real-world environments.

Key takeaways:

Geo testing solves situations where A/B tests can’t be run
Instead of randomizing users, geo experiments allocate entire geographic units such as cities, DMAs, or regions to treatment or control. This approach is used when:

• User-level randomization is not possible (e.g., brand or awareness campaigns)
  
  
• Regulations or operational constraints require changes to be applied uniformly within a geography
  
  
• Running two variants side-by-side in the same geography would cannibalize or distort performance (common in marketplace platforms)
  
  

Synthetic controls are essential for isolating true incremental lift
Simply comparing treatment vs. control geos is not enough. Synthetic controls model what would have happened in the treatment geos had the intervention not occurred. This isolates real incremental impact and was a key method Uber used when rolling out major product updates across multiple U.S. cities without disrupting live operations.

Marketing campaigns benefit greatly from geo testing
Attribution models can make campaigns appear successful even when incremental impact is low. In one case, a gaming company observed high ROI for brand search ads in attribution reports - but geo testing revealed almost zero incremental lift in several regions. Geo tests help teams reallocate budget to initiatives that truly move business outcomes.

Experiment design matters
Choosing which geos to allocate to treatment or control is only the beginning. Budget, expected lift (MDE), seasonality, and platform behavior all influence reliability. Unlike user-level A/B tests, expecting very small lifts (0.5-1%) to be detectable is unrealistic - geo testing is most effective when targeting meaningful business effects (5-10%+).

Workflow is standardized and accessible with the right tools
Platforms like Statsig streamline geo test setup - from selecting metrics and duration to running power analysis and generating recommended treatment/control groupings. Analysis then compares observed outcomes in treatment geos against their synthetic controls to estimate lift and statistical confidence.

Running multiple experiments in parallel requires planning
Testing across different platforms or budgets (e.g., Meta at the state level and Google at the DMA level) can lead to overlap and interference. Best practices include staggering experiments, using mutually exclusive geo sets, or splitting by platform (iOS vs. Android) where relevant.

Start small, scale up
Teams familiar with A/B testing can adopt geo testing quickly. Data requirements are often simpler than user-level logging - the main learning curve is in experiment setup and result interpretation. With a few well-run tests, teams build intuition and confidence fast.

In short: Geo testing enables marketing and product teams to measure real-world business impact, reduce wasted spend, and make high-confidence decisions - even when A/B testing is not an option. With the right methodology and tooling, the insights can be transformative.