Just do it — Measuring Complex Customer Journeys

Imagine you own a successful shoe store with branches across the US and an e-commerce store. You invest in advertising both online, on platforms like Facebook, and offline, through bus stops and billboards. Your business is thriving, but you’re uncertain about the effectiveness of your ads for both physical and online stores. Are your online ads driving these sales? Is it the offline ads? Or perhaps a combination of both?

The Dilemma: Measuring Ad Effectiveness

To solve this mystery, you hire analysts to track the customer journey. They use various attribution models like first-touch, last-touch, and multi-touch attribution to determine the impact of your ads. While these methods show some impact from your online ads, they don’t tell the whole story. Here’s why:

Traditional Attribution Models and Their Limitations

Attribution models are tools used to assign credit to marketing touch points along the customer journey. The most common types include first-touch attribution, which credits the first interaction a customer has with your brand; last-touch attribution, which credits the last interaction before the purchase; and multi-touch attribution, which spreads the credit across all touch points leading to the purchase. While these models provide some insights, they have significant limitations.

Complex Customer Journeys

Consider these scenarios, typical in the shoe business example:

• A customer sees an ad at a bus stop, then takes out their phone, searches for your store, and purchases shoes online. Here, traditional models may only attribute the online purchase to the Google search, neglecting the influence of the offline bus stop ad.
• Another customer sees a billboard ad and promptly heads to a physical store to buy shoes. In this case, last-touch attribution may credit the physical store visit, overlooking the initial impact of the billboard ad.
• Yet another customer sees an online ad, decides to visit a store, and makes a purchase there. While the online ad may receive credit for driving the store visit, it might not fully capture the offline purchase made in-store.

These diverse journeys are difficult to capture with traditional attribution models, which often oversimplify complex paths, focusing primarily on digital interactions and overlooking the impact of offline marketing efforts. Moreover, stricter privacy regulations make tracking individual users across multiple touch points more challenging, further reducing the effectiveness of these models. As a result, businesses often rely on gut feelings rather than data-driven insights.

Advanced Solutions: Geo Tests and Marketing Mix Modeling (MMM)

To gain a deeper understanding of your shoe store’s advertising efforts’ effectiveness, you need a better approach. This is where geo tests and Marketing Mix Modeling (MMM) step in.

Geo Tests

In our shoe store example, let’s explore how to determine the effectiveness of your ads using geo tests. The most commonly used tool for this is Meta’s GeoLift package, a Python and R package that helps you plan and analyze tests for all platforms. While there are alternatives like geoX, which is specific to Google and less flexible, GeoLift stands out for its ease of use and broad applicability.

Using GeoLift, you can divide the US DMAs into two comparable groups based on weighted averages, ensuring that your KPI trends, which is usually sales or revenue, is similar in both groups before the test begins. This initial similarity is crucial for accurately measuring the impact of your ads on shoe sales. It’s important to remember that you need to actively advertise on the platform in your test group while not advertising in your control group. If you have any real-world considerations regarding where to test, make sure to incorporate them.

GeoLift also helps you determine the optimal duration for running the test, balancing cost and effect size detection based on your budget, data, and assumptions. You can specify areas to exclude and apply other advanced nuances. For example, in one group of DMAs, you run your Facebook ads as usual to promote your shoe store, while in the other group, you don’t. Alternatively, if you have an ongoing long-term campaign to advertise your shoe store, you can conduct a negative test by turning off ads in one group. In both scenarios, all other marketing efforts for your shoe store remain unchanged, either on or off in all regions, focusing solely on Facebook ads.

Over the next few weeks, you track shoe sales or revenue in both groups. A significant increase in shoe sales in the regions where the ads are running compared to the control group indicates that the Facebook ads are driving those sales. This method effectively isolates the true impact of the ads, as the control group accounts for other factors like seasonal demand fluctuations. This approach allows you to isolate the effect of these Facebook ads on your shoe sales, considering the other campaigns that are ongoing. You learn the incremental effect of these Facebook ads during this period, given the other ads that are already running.

This method isn’t limited to online ads; it can also be applied to offline marketing strategies like bus stop or billboard ads. For instance, you place billboard ads for your shoe store in certain DMAs and not in others, then track total sales or revenue from both your physical branches and e-commerce store. By comparing these KPIs in the control and test groups, you can reveal the ads’ effectiveness, regardless of where or when the shoe purchase occurred.

After running a few geo tests for your shoe store, you’ll likely want to conduct more to maximize your marketing strategy. To do this efficiently, consider running multiple tests simultaneously by dividing the US into different, non-correlated groups. For example, you can run a Facebook ad test using one division and a TikTok ad test using another. Plan the timing of these tests carefully to capture effects during different but important periods, as the impact of ads can vary with demand changes. Additionally, consider what other campaigns are ongoing because you will learn the incremental effect of the ads you are testing on top of these other campaigns. These tests can also be incorporated into Marketing Mix Models (MMMs) for a comprehensive analysis.

MMM

Since geo tests provide only a snapshot of your marketing effectiveness, typically highlighting the impact of your ads on one platform at a specific time, a more comprehensive understanding of your shoe store’s performance requires a Marketing Mix Model (MMM). This model considers all your data — from online and offline ads to sales figures and external factors like seasonality — and breaks it down to reveal the true impact of each marketing channel.

With an MMM, you can uncover insights such as:

• The proportion of your sales driven by Facebook ads, bus stop ads, or any other channel.
• The long-term impact of your marketing efforts, including adstock effects and diminishing returns.
• How to best reallocate your marketing budget to maximize ROI.

To build an MMM, you first need to choose a tool. The main contenders are Robyn by Meta, Meridian by Google, and Orbit by Uber, among others. Each tool has its unique strengths. For example, since shoe stores experience significant seasonal variation, Orbit might be a good choice as it allows the ROI of a channel to change over time. However, Orbit does not account for adstocks, whereas Robyn and Meridian do.

Here’s a “quick” comparison of the three tools:

We suggest selecting two tools that best suit your business needs and testing them. Building the model requires time to collect and input all the data and choose the appropriate parameters. An important step in this process is to incorporate the results of geo tests for your main channels at different times, capturing various phases of your business. You will need around two years of historical data and at least three channels, whether online or offline.

Imagine you’ve gathered two years of data for your shoe store. Each row is a date. Then you need either sales or revenue numbers for each date, and how much you spent on advertising on every platform on each date, including online platforms like Facebook and Google, and offline ads like bus stops and billboards. By feeding this data into your chosen MMM tools and incorporating the results of your geo tests, you can start to see the bigger picture.

For instance, you might discover that your Facebook ads significantly boost sales during the back-to-school season, a crucial period for selling kids’ shoes. Conversely, your bus stop ads might drive higher sales during the summer months when foot traffic is high, as people are out and about more frequently. With these insights, you can adjust your marketing strategy. Perhaps you’ll decide to increase your Facebook ad spend in August and September, while focusing your bus stop ads in June and July. You might also find that certain channels, like Google ads, are consistently underperforming and need to be re-evaluated or phased out in favor of more effective methods.

Making Informed Decisions

By combining geo tests and MMM, you can gain a robust framework to understand the true effectiveness of your marketing efforts. This approach offers a comprehensive understanding of how your ads impact both online and offline sales, enabling you to make data-driven decisions that drive growth and improve ROI for your shoe store.