Year-over-Year Target Transform: A Custom MMM for Highly Seasonal KPIs#
Some KPIs move mostly with the calendar. Think of garden centers in spring, toy stores before Christmas, or travel around a moving holiday such as Easter. For these KPIs the annual cycle dominates the variance. A level-space media mix model (MMM) then spends most of its parameters on the seasonal shape before it can say anything about media.
In this notebook we try a different route. We stop modeling the level and model the year-over-year ratio instead. Last year’s observed KPI becomes the seasonal baseline, so seasonality enters multiplicatively and at no cost: the model needs no Fourier basis, no trend component and no per-unit seasonal profile.
The transform is easy to implement, but it changes what the likelihood can identify. As a result, it also changes what a channel contribution means. We build the model with plain PyMC and PyMC-Marketing components, generate a synthetic panel from it, and compare it out of sample with level-space MMM models on the same data.
The analysis proceeds in these steps:
The Transform
The Reference Map
Synthetic Data
Model Specification
Data Generation
Exploratory Data Analysis
Train Test Split, Prior Predictive Checks and Model Fitting
Model Diagnostics and Posterior Predictive Checks
Parameter Recovery
Media Deep Dive
Level-Space Benchmarks
Out-of-Sample Comparison
A Fixed Seasonal Shape
Conclusion and Next Steps
Note
We build a custom model from PyMC-Marketing components, following the strategy of Custom Models with MMM components and Custom Geo-Hierarchical MMM with Splines. We generate the data from a PyMC model with pymc.do(), as in Generating Synthetic MMM Data. For the standard MMM workflow see Media Mix Modeling (MMM) in Python: End-to-End Example, and for the continuous ranked probability score (CRPS) see Time-Slice-Cross-Validation and Parameter Stability.
Prepare Notebook#
We load the necessary libraries and set the notebook’s configuration.
from collections import Counter
from collections.abc import Sequence
from dataclasses import dataclass
from typing import NamedTuple, Self
import arviz as az
import arviz_plots as azp
import matplotlib.dates as mdates
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import pymc as pm
import pymc.dims as pmd
import pytensor.tensor as pt
import pytensor.xtensor as ptx
import xarray as xr
from matplotlib.lines import Line2D
from pymc import do
from pymc_extras.prior import Prior
from pymc_marketing.metrics import crps
from pymc_marketing.mmm import MMM, GeometricAdstock, LogisticSaturation
az.style.use("arviz-darkgrid")
plt.rcParams["figure.figsize"] = [12, 7]
plt.rcParams["figure.dpi"] = 100
plt.rcParams["figure.facecolor"] = "white"
%config InlineBackend.figure_format = "retina"
seed: int = sum(map(ord, "mmm_yoy_target_transform"))
rng: np.random.Generator = np.random.default_rng(seed=seed)
The Transform#
Let \(y_{u,t}\) be the KPI of unit \(u\) (a store, a region or a product) in week \(t\). Let \(r(t)\) be the reference map: it sends week \(t\) to the week one cycle earlier that we compare it with. Without moving holidays, \(r(t) = t - 52\). We model the ratio
Why should a ratio help? Let’s write the level as the product of a seasonal shape \(S_{u,t}\), a trend \(G_{u,t}\), a lift \(m_{u,t}\) from media and controls, and noise \(\varepsilon_{u,t}\):
Here \(c\) indexes the media channels (tv, search and sponsorship) and \(j\) indexes the controls (discount). The term \(f_{c,u,t}\) is the effect of channel \(c\) after adstock and saturation, \(x_{j,u,t}\) is control \(j\) in its raw units, and \(\gamma_j\) is its coefficient. The lift is a share of the baseline, so \(m_{u,t} = 0.1\) means \(10\%\) more KPI.
Now assume that the shape repeats, \(S_{u,t} = S_{u,r(t)}\), and that the trend grows by a factor \(g_u\) per cycle, \(G_{u,t} = g_u \, G_{u,r(t)}\). Then the shape cancels in the ratio:
The approximation holds to first order when the lifts are small and \(g_u\) is close to one. We take its right-hand side as our model:
In other words, every driver enters as a within-pair contrast: its value in week \(t\) minus its value in the reference week \(r(t)\). A channel can only move this year’s KPI away from “last year’s level times the growth rate” if its spend changed between the two weeks. We recover the level exactly with \(y_{u,t} = y_{u,r(t)} \, z_{u,t}\).
Here are three important remarks about this parametrization:
Seasonality is absorbed multiplicatively. Any part of the seasonal shape that repeats from cycle to cycle is already in \(y_{u,r(t)}\). The model only has to explain the departures from last cycle.
The per-unit baseline collapses to a scalar. \(g_u\) is the typical growth rate of unit \(u\). There is no per-unit seasonal curve, which is where most of the parameters of a level-space panel MMM go.
The noise is proportional, and the reference is noisy too. Additive noise on \(z\) is proportional noise on \(y\), which is usually closer to the truth for sales-like KPIs. But \(y_{u,r(t)}\) is an observation, not a parameter. If the level noise is independent across cycles, the ratio has about twice the noise variance of the level, and it is negatively correlated with the ratio one cycle earlier. The last section of the notebook measures both effects.
The Reference Map#
The naive map \(r(t) = t - 52\) works only if the calendar repeats exactly, and it does not. Moving holidays such as Easter, Ramadan or Chinese New Year shift by a few weeks from one year to the next. With a naive lag, a holiday week is divided by a normal week. This puts a spurious spike into \(z\) exactly where the KPI is largest.
So we anchor \(r\) on the event: this year’s holiday week points to last year’s holiday week. A shoulder of pad weeks on each side of the holiday travels with it.
Let’s see how this works with a concrete example. Last year the holiday fell in week \(13\), and this year it falls in week \(15\), so it moved by \(k = 2\) weeks. With pad=1, the holiday window covers weeks \(12\) to \(14\) last year and weeks \(14\) to \(16\) this year.
The obvious approach drags this year’s window onto last year’s window and keeps the base lag for every other week. This year’s weeks \(14\), \(15\) and \(16\) then point to last year’s weeks \(12\), \(13\) and \(14\). But this year’s weeks \(12\) and \(13\) are normal weeks, and they keep their base references: last year’s weeks \(12\) and \(13\).
This has two defects that are easy to miss:
Duplicated and orphaned references. Last year’s weeks \(12\) and \(13\) are now the reference for two weeks each, and last year’s weeks \(15\) and \(16\) are the reference for no week at all.
A displaced crater. This year’s week \(13\) is a normal week, but its reference is last year’s holiday peak. A normal week divided by a holiday week is a crater in \(z\). This is the same artifact that anchoring should remove, only one week earlier.
In general, dragging leaves \(|k|\) duplicated and \(|k|\) orphaned weeks whenever the event moves. As soon as the move is larger than the shoulder, \(|k| > \text{pad}\), a normal week is also divided by last year’s holiday peak.
The fix is to treat the realignment as a permutation of last year’s weeks, not as a shift. In our example, the two windows together span last year’s weeks \(12\) to \(16\). We rotate this segment by \(k = 2\) weeks. This year’s weeks \(14\), \(15\) and \(16\) still point to last year’s weeks \(12\), \(13\) and \(14\). This year’s weeks \(12\) and \(13\) now point to last year’s weeks \(15\) and \(16\), the two weeks that the holiday window left free.
Now every week of last year is the reference for exactly one week of this year, so \(r\) is a bijection between consecutive cycles. Every holiday week is paired with a holiday week, and the sum of \(y_{u,r(t)}\) over a cycle equals last year’s total. The price is that the pushed-aside weeks pair with a week \(2\,\text{pad} + 1 = 3\) weeks away. On a smooth seasonal shape this costs a few percent, not a holiday-sized spike.
We encode an anchor as a pair of period indices. The builder returns the map together with the two checks that it is a bijection: the prior periods that it uses twice and the ones that it never uses.
class Anchor(NamedTuple):
"""Two consecutive occurrences of a moving event, as period indices.
Attributes
----------
current : int
Period index of an occurrence.
prior : int
Period index of the occurrence one cycle earlier.
"""
current: int
prior: int
@dataclass(frozen=True)
class ReferenceMap:
"""Reference period of every period, with the two checks that the map is a bijection.
Attributes
----------
reference : np.ndarray
``reference[t]`` is the index of the reference period of ``t``. It is negative in the
first cycle, where no reference exists.
duplicated : list of int
Prior periods that are the reference of more than one period.
orphaned : list of int
Prior periods that are the reference of no period.
"""
reference: np.ndarray
duplicated: list[int]
orphaned: list[int]
@classmethod
def from_reference(cls, reference: np.ndarray, period: int) -> Self:
"""Wrap a reference array and find the prior periods that it uses twice or never."""
used = reference[reference >= 0].tolist()
counts = Counter(used)
return cls(
reference=reference,
duplicated=sorted(p for p, n in counts.items() if n > 1),
orphaned=sorted(set(range(len(reference) - period)) - set(used)),
)
@property
def is_bijective(self) -> bool:
"""Whether every prior period is the reference of exactly one period."""
return not self.duplicated and not self.orphaned
The builder starts from the base lag and rotates one prior-cycle segment per anchor.
def build_reference_map(
n_periods: int,
period: int,
anchors: Sequence[Anchor] = (),
pad: int = 0,
) -> ReferenceMap:
"""Map each period to its counterpart one cycle earlier.
Moving events are realigned by rotating the prior-cycle segment that the current and the
prior event windows span, so the map stays a bijection between consecutive cycles.
Parameters
----------
n_periods : int
Length of the full series.
period : int
Base cycle length, for example ``52`` for weekly data.
anchors : sequence of Anchor
Consecutive occurrences of a moving event. ``prior`` must be about one cycle before
``current`` (within half a cycle of ``current - period``), and the windows of different
anchors must not overlap. Both occurrences must lie within the series. The builder raises
``ValueError`` when any of these conditions fails.
pad : int
Number of shoulder periods on each side of an occurrence that travel with it. Must be
non-negative.
Returns
-------
ReferenceMap
The reference of every period, with the duplicated and orphaned prior periods. Both
lists are empty whenever the anchors satisfy the contract above and every current
window fits inside the series. When the last window runs past the end of the series,
part of its rotation is dropped and the affected prior periods appear in ``orphaned``.
"""
if pad < 0:
raise ValueError(f"pad must be non-negative, got {pad}")
reference = np.arange(n_periods) - period
rotated_periods: set[int] = set()
for anchor in anchors:
if not 0 <= anchor.prior < anchor.current < n_periods:
raise ValueError(
f"anchor {tuple(anchor)}: both occurrences must lie within the series of "
f"{n_periods} periods"
)
shift = (anchor.current - period) - anchor.prior
if abs(shift) > period // 2:
raise ValueError(
f"anchor {tuple(anchor)} pairs occurrences {anchor.current - anchor.prior} "
"periods apart, but each anchor must pair an occurrence with the one about one "
"cycle earlier"
)
if shift == 0:
continue
# Prior-cycle segment that both event windows span: (segment,)
segment = np.arange(
min(anchor.prior, anchor.prior + shift) - pad,
max(anchor.prior, anchor.prior + shift) + pad + 1,
)
if segment[0] < 0:
raise ValueError(
f"anchor {tuple(anchor)}: the event window starts before the series"
)
if rotated_periods.intersection(segment.tolist()):
raise ValueError(
f"anchor {tuple(anchor)}: the event window overlaps another anchor's window"
)
rotated_periods.update(segment.tolist())
for base, rotated in zip(segment, np.roll(segment, shift), strict=True):
if base + period < n_periods:
reference[base + period] = rotated
return ReferenceMap.from_reference(reference, period)
To see the defect, we also code the obvious version that drags each window.
def drag_reference_map(
n_periods: int,
period: int,
anchors: Sequence[Anchor] = (),
pad: int = 0,
) -> ReferenceMap:
"""Drag each current event window onto the prior one and keep the base lag elsewhere.
This is the obvious anchoring. We only use it to show its defect.
"""
reference = np.arange(n_periods) - period
# Positions inside an event window, relative to the occurrence: (2 * pad + 1,)
offsets = np.arange(-pad, pad + 1)
for anchor in anchors:
reference[anchor.current + offsets] = anchor.prior + offsets
return ReferenceMap.from_reference(reference, period)
We draw each map as arrows from this year’s weeks to their reference weeks last year. A red arrow pairs a holiday week with a normal week.
Here is the example from above: the holiday moves from week \(13\) to week \(15\), with pad=1. We use a two-year series, so that last year’s weeks are also their period indices.
period = 52
toy_anchor = Anchor(current=period + 15, prior=13)
toy_weeks = np.arange(9, 20)
toy_maps = {
r"naive lag $r(t) = t - 52$": build_reference_map(
n_periods=2 * period, period=period
),
"drag the window": drag_reference_map(
n_periods=2 * period, period=period, anchors=[toy_anchor], pad=1
),
"rotate the segment": build_reference_map(
n_periods=2 * period, period=period, anchors=[toy_anchor], pad=1
),
}
fig, ax = plt.subplots(figsize=(12, 6.5), layout="constrained")
for i, (title, ref_map) in enumerate(toy_maps.items()):
height = 3 * (len(toy_maps) - 1 - i)
draw_reference_pairs(
ax,
ref_map,
anchor=toy_anchor,
period=period,
pad=1,
weeks=toy_weeks,
height=height,
)
ax.text(toy_weeks.mean(), height + 1.4, title, ha="center", fontsize=13)
ax.set(
xticks=toy_weeks,
xlabel="week of the year",
yticks=[0, 1, 3, 4, 6, 7],
yticklabels=["last year", "this year"] * 3,
ylim=(-0.6, 7.9),
)
ax.grid(axis="y", visible=False)
legend_handles = [
Line2D(
[],
[],
marker="o",
linestyle="",
markersize=11,
color="C1",
label="holiday window",
),
Line2D(
[],
[],
marker="o",
linestyle="",
markersize=11,
color="0.7",
label="normal week",
),
Line2D(
[],
[],
marker="o",
linestyle="",
markersize=20,
markerfacecolor="none",
markeredgecolor="black",
markeredgewidth=2,
label="duplicated reference",
),
Line2D(
[],
[],
marker="x",
linestyle="",
markersize=12,
color="black",
markeredgewidth=2.5,
label="orphaned reference",
),
Line2D([], [], color="0.35", linewidth=1.2, label="same kind of week"),
Line2D(
[],
[],
color="crimson",
linewidth=2.2,
label="holiday week paired with a normal week",
),
]
fig.legend(handles=legend_handles, loc="outside lower center", ncols=3)
fig.suptitle(
"Anchoring a Holiday That Moves From Week 13 to Week 15",
fontsize=16,
fontweight="bold",
);
The top diagram is the naive lag. This year’s holiday weeks \(15\) and \(16\) point to normal weeks, and this year’s normal weeks \(12\) and \(13\) point into last year’s holiday window. The middle diagram drags the window. Last year’s weeks \(12\) and \(13\) get two arrows each, weeks \(15\) and \(16\) get none, and the red arrow from this year’s week \(13\) to last year’s peak is the displaced crater. The bottom diagram rotates the segment. Every week of last year gets exactly one arrow, and no arrow mixes a holiday week with a normal week.
Anchor Contract#
The builder takes anchors as pairs of consecutive occurrences, and the rotation only makes sense when the two occurrences are about one cycle apart. The builder checks this condition and raises a ValueError when it fails.
The check matters because a convenient encoding can break the condition without notice. We often store a moving event as one week of the year per cycle. But an event in week \(50\) of one year and in week \(0\) of the next year is two weeks apart, not one year. A map built from that pair would return a two-week ratio that looks like a year-over-year one.
An event window that crosses the boundary between two cycles is not a problem. Take a yearly event that moves two weeks earlier every year: weeks \(3\), \(1\), \(51\), \(49\), \(47\) and \(45\). The third year then holds two occurrences (weeks \(1\) and \(51\)), and one event window crosses into the fourth year. The rotation works on runs of consecutive periods, not on calendar years, so it handles this case.
We pair consecutive occurrences with a small helper.
def consecutive_anchors(occurrences: Sequence[int]) -> list[Anchor]:
"""Pair every occurrence of an event with the occurrence before it."""
return [
Anchor(current=current, prior=prior)
for current, prior in zip(occurrences[1:], occurrences[:-1], strict=True)
]
Next, we write a diagnostic figure that we use twice: for this drifting event and for the holiday calendar of our panel. The top panel shows the lag \(t - r(t)\) of every week. The middle panel shows how often each prior week is the reference of a week, which is one everywhere for a bijection. When we pass the event weeks, a bottom panel marks the pairs that mix an event week with a normal week.
Let’s check the drifting event.
n_demo = 6 * period
# Weeks 3, 1, 51, 49, 47 and 45, with two occurrences in the third year: (occurrence,)
drifting_event = [55, 105, 155, 205, 255, 305]
drifting_map = build_reference_map(
n_periods=n_demo, period=period, anchors=consecutive_anchors(drifting_event), pad=1
)
fig = plot_reference_diagnostics(
{"rotate the segment": drifting_map}, period=period, occurrences=drifting_event
)
fig.suptitle(
"Reference Map for an Event That Drifts Two Weeks Earlier Each Year",
fontsize=16,
fontweight="bold",
);
Every lag stays between \(50\) and \(55\) weeks, so every reference is about one year back, even across the boundary between the third and the fourth year. Every prior week is used exactly once, so the map is a bijection.
Now let’s encode the same kind of event as one week of the year per cycle, with weeks \(48\), \(50\), \(0\), \(2\), \(4\) and \(6\). Week \(0\) of the third year and week \(50\) of the second year are only two weeks apart.
bad_event = [period * cycle + week for cycle, week in enumerate([48, 50, 0, 2, 4, 6])]
try:
build_reference_map(
n_periods=n_demo, period=period, anchors=consecutive_anchors(bad_event), pad=1
)
except ValueError as error:
print(f"ValueError: {error}")
ValueError: anchor (104, 102) pairs occurrences 2 periods apart, but each anchor must pair an occurrence with the one about one cycle earlier
The builder refuses the pair instead of returning a two-week ratio.
Synthetic Data#
We build a weekly panel of four regions over six years. The design favors the ratio model on purpose, so that the comparison isolates the mechanism rather than the tuning:
Inherited seasonal shape: the annual shape carries over from cycle to cycle through the reference map, so we do not draw it again each year.
Moving holiday: the holiday has a three-week profile (the week before, the peak and the week after), and its week of the year moves between weeks \(12\) and \(15\).
Always-on channel: of the three media channels,
sponsorshipruns at a flat plan, so its year-over-year contrast is exactly zero.Proportional noise: the observation noise scales with the level.
We come back to the fairness of this design in the comparison, and with a second panel at the end of the notebook.
n_years = 6
n_dates = n_years * period
date_index = pd.date_range(start="2019-01-07", freq="W-MON", periods=n_dates)
unit_names = ["north", "south", "east", "west"]
channel_columns = ["tv", "search", "sponsorship"]
control_columns = ["discount"]
n_units = len(unit_names)
n_channels = len(channel_columns)
# Maximum adstock lag in weeks.
l_max = 8
coords_data = {
"date": date_index,
"unit": unit_names,
"channel": channel_columns,
"control": control_columns,
}
The holiday falls in a different week of the year in each of the six cycles. Its multiplicative profile lifts the week before by \(15\%\), the peak by \(55\%\) and the week after by \(10\%\).
# Week of the year of the holiday in each cycle: (cycle,)
holiday_week_of_year = [12, 13, 15, 14, 12, 13]
# Period index of the holiday in each cycle: (cycle,)
holiday_index = [
period * cycle + week for cycle, week in enumerate(holiday_week_of_year)
]
# Multiplicative profile of the week before, the peak and the week after: (3,)
holiday_profile = np.array([1.15, 1.55, 1.10])
holiday_offsets = np.arange(-1, 2)
# Every period inside a holiday window: (cycle * 3,)
holiday_weeks = np.add.outer(holiday_index, holiday_offsets).ravel()
print(f"holiday period index per cycle: {holiday_index}")
holiday period index per cycle: [12, 65, 119, 170, 220, 273]
Holiday Calendar#
We anchor each cycle’s holiday to the previous cycle’s holiday, with pad=1. This one-week shoulder on each side matches the width of the holiday profile. For comparison, we also build the naive lag and the dragged map on the same calendar.
holiday_anchors = consecutive_anchors(holiday_index)
calendar_maps = {
"naive lag": build_reference_map(n_periods=n_dates, period=period),
"drag the window": drag_reference_map(
n_periods=n_dates, period=period, anchors=holiday_anchors, pad=1
),
"rotate the segment": build_reference_map(
n_periods=n_dates, period=period, anchors=holiday_anchors, pad=1
),
}
# Reference period of every period, used in the rest of the notebook: (date,)
reference_map = calendar_maps["rotate the segment"].reference
naive_reference_map = calendar_maps["naive lag"].reference
Let’s run the diagnostic figure on the three maps.
fig = plot_reference_diagnostics(
calendar_maps, period=period, occurrences=holiday_index, event_weeks=holiday_weeks
)
fig.suptitle("Reference Maps for the Holiday Calendar", fontsize=16, fontweight="bold");
The naive lag is a bijection, but it pairs holiday weeks with normal weeks \(14\) times, around every holiday that moved. The dragged map aligns every holiday window, but it uses \(7\) prior weeks twice and \(7\) never, and it still mixes \(7\) pairs, one per week of shift. The rotated map is a bijection without mixed pairs.
The lag of the rotated map leaves \(52\) weeks only at the seams. The holiday windows move by \(|k| = 1\) or \(2\) weeks, and the weeks that they push aside move by \(2\,\text{pad} + 1 = 3\) weeks in the other direction. The second anchor is the example of the previous section, with the holiday moving from week \(13\) to week \(15\).
Note
The cost of anchoring
Anchoring moves two groups of weeks. The holiday window moves by \(|k|\), the size of the holiday’s own shift, which is the point. The \(|k|\) weeks that the window pushes aside move by \(2\,\text{pad} + 1\) in the other direction, and that is the cost. So the shoulder should be no wider than the real footprint of the event. For a one-week event, pad=0 moves the pushed-aside weeks by one week. Here the holiday has a three-week profile, so pad=1 is the right width.
Channel Spend#
Media spend comes from a small PyMC model. A multivariate normal over the channels with an LKJ prior on the correlation captures that brands rarely move one channel at a time. Each unit-channel pair has its own intercept and trend, and a softplus keeps the spend positive. The discount control comes from a Beta distribution.
# Time on [0, 1) for the spend trends: (date,)
t_scaled = np.arange(n_dates) / n_dates
with pm.Model(coords=coords_data) as covariates_model:
t_data = pm.Data("t", t_scaled, dims="date")
chol, _, _ = pm.LKJCholeskyCov(
"chol",
n=n_channels,
eta=3,
sd_dist=pm.Exponential.dist(lam=1 / 1.5),
)
spend_intercept = pm.Normal(
"spend_intercept", mu=0.0, sigma=0.7, dims=("unit", "channel")
)
spend_slope = pm.Normal("spend_slope", mu=0.0, sigma=0.7, dims=("unit", "channel"))
# Mean of the latent spend: (date, unit, channel)
spend_mu = pm.Deterministic(
"spend_mu",
spend_intercept + spend_slope * t_data[:, None, None],
dims=("date", "unit", "channel"),
)
spend_raw = pm.MvNormal(
"spend_raw", mu=spend_mu, chol=chol, dims=("date", "unit", "channel")
)
spend = pm.Deterministic(
"spend", pt.softplus(spend_raw), dims=("date", "unit", "channel")
)
discount = pm.Beta(
"discount", alpha=1.2, beta=6.0, dims=("date", "unit", "control")
)
pm.model_to_graphviz(covariates_model)
The graph shows the correlated latent spend spend_raw for every date, unit and channel, its softplus spend, and the independent discount draw.
We draw one realization of spend and discount. Then we hold sponsorship at a flat always-on plan: constant over time, so identical from cycle to cycle by construction. Finally, we scale each channel by its maximum, so that the true saturation parameters and every fitted model use the same input scale.
spend_draw, discount_draw = pm.draw(
[covariates_model["spend"], covariates_model["discount"]], draws=1, random_seed=rng
)
# Flat always-on plan for sponsorship: its mean over time, per unit.
sponsorship = channel_columns.index("sponsorship")
spend_draw[:, :, sponsorship] = spend_draw[:, :, sponsorship].mean(
axis=0, keepdims=True
)
# Raw spend and discount: (date, unit, channel) and (date, unit, control)
channel_data = xr.DataArray(
spend_draw,
dims=("date", "unit", "channel"),
coords={"date": date_index, "unit": unit_names, "channel": channel_columns},
)
control_data = xr.DataArray(
discount_draw,
dims=("date", "unit", "control"),
coords={"date": date_index, "unit": unit_names, "control": control_columns},
)
# Spend scaled by the maximum of each channel over dates and units: (date, unit, channel)
channel_scale = channel_data.max(("date", "unit"))
channel_scaled = channel_data / channel_scale
print(f"channel_scaled: {dict(channel_scaled.sizes)}")
print(f"control_data: {dict(control_data.sizes)}")
channel_scaled: {'date': 312, 'unit': 4, 'channel': 3}
control_data: {'date': 312, 'unit': 4, 'control': 1}
Let’s look at the scaled spend over time.
tv and search move from week to week and follow a different trend in each unit. sponsorship is a flat line at a different level for each unit.
The average spend per unit makes the sponsorship plan explicit.
Each unit has its own sponsorship level, lowest in south and highest in north. Keep this in mind: a constant per unit is exactly what a per-unit intercept also describes.
Here is the key design check. We compare the variability of the spend level with the variability of the year-over-year spend contrast. We only use the modeled weeks: every week after the first cycle except the first \(l_{\max} = 8\), so that adstock has warmed up on both sides of every pair. The level recursion still needs the first \(8\) weeks, so we generate them too.
# Modeled weeks: every week after the first cycle and the adstock warm-up: (obs_date,)
obs_idx_model = np.arange(period + l_max, n_dates)
# Scaled spend as a plain array: (date, unit, channel)
spend_values = channel_scaled.transpose("date", "unit", "channel").to_numpy()
contrast_summary = pd.DataFrame(
{
"std of the spend level": spend_values.std(axis=(0, 1)),
"std of the year-over-year contrast": (
spend_values[obs_idx_model] - spend_values[reference_map[obs_idx_model]]
).std(axis=(0, 1)),
},
index=pd.Index(channel_columns, name="channel"),
)
fig, ax = plt.subplots(figsize=(12, 5), layout="constrained")
contrast_summary.plot.bar(ax=ax, rot=0, color=["C0", "C1"])
ax.legend(loc="upper left", bbox_to_anchor=(1.01, 1.0))
ax.set(xlabel="channel", ylabel="standard deviation (scaled spend)")
fig.suptitle("Spend Level vs Year-over-Year Contrast", fontsize=16, fontweight="bold");
The year-over-year contrasts of tv and search vary even more than their levels, so the ratio model has plenty of signal for these two channels. The contrast of sponsorship is exactly zero, although its level varies across units. The ratio model never sees this channel.
Ground-Truth Parameters#
Next, we fix the true parameters. The saturation beta values are in ratio points: a full swing of tv moves the year-over-year ratio by up to \(0.13\), that is \(13\%\) of the reference level.
adstock_alpha_true = np.array([0.55, 0.25, 0.40])
saturation_lam_true = np.array([2.5, 4.0, 3.0])
saturation_beta_true = np.array([0.13, 0.08, 0.06])
gamma_control_true = np.array([0.18])
intercept_true = np.array([1.03, 1.00, 0.98, 1.05])
z_sigma_true = np.array([0.030, 0.035, 0.028, 0.032])
media_truth = {
"adstock_alpha": adstock_alpha_true,
"saturation_lam": saturation_lam_true,
"saturation_beta": saturation_beta_true,
}
Let’s look at the true media parameters. We reuse these labels in every parameter figure below.
parameter_labels = {
"adstock_alpha": r"adstock decay $\alpha_c$",
"saturation_lam": r"saturation steepness $\lambda_c$",
"saturation_beta": r"saturation maximum $\beta_c$",
}
fig, axes = plt.subplots(ncols=3, figsize=(14, 4.5), layout="constrained")
for ax, (name, values) in zip(axes, media_truth.items(), strict=True):
ax.bar(channel_columns, values, color=["C0", "C1", "C2"])
ax.set_title(parameter_labels[name])
fig.suptitle("Ground-Truth Media Parameters", fontsize=16, fontweight="bold");
tv has the longest carryover and the largest effect. search has the shortest carryover and the steepest saturation. sponsorship has the smallest effect, but we will see that the year-over-year model cannot see it at all.
Model Specification#
We wrap the model in a function, so that the same graph serves three purposes: data generation (with the parameters clamped by pymc.do()), inference on the training window, and prediction on another window. Here is the full model for unit \(u\) and week \(t\):
The symbols mean the following:
\(a_{c,u,t}\) is the spend of channel \(c\), scaled so that the largest value of each channel is \(1\).
\(\tilde{a}_{c,u,t}\) is the adstocked spend. The geometric weights \(w_l(\alpha_c)\) sum to one over \(L = 8\) weeks (
GeometricAdstock).\(\alpha_c\) is the adstock decay, the share of the effect that carries over to the next week.
\(\lambda_c\) and \(\beta_c\) are the steepness and the maximum of the saturation curve (
LogisticSaturation). \(\beta_c\) is in ratio points, so \(\beta_c = 0.1\) means that the channel at full spend lifts the KPI by up to \(10\%\) of the reference level.\(g_u\) is the growth rate of unit \(u\) (
interceptin the code).\(\gamma_j\) is the coefficient of control \(j\) on its raw scale (
gamma_controlin the code).\(\sigma_u\) is the noise scale of the ratio, one per unit (
z_sigmain the code).
Note
Why this parametrization?
The saturation acts on the spend level of each week, and we take the difference afterwards. A difference inside the curve, \(f_c(a_t - a_{r(t)})\), would treat a move from \(0.0\) to \(0.1\) like a move from \(0.9\) to \(1.0\), which contradicts diminishing returns.
The adstock runs over the full contiguous series, and only then do we pick the weeks \(t\) and \(r(t)\). If we picked the weeks first, the carryover of the reference week would come from the history of the current week, which is wrong.
The controls enter linearly, so \(\gamma_j (x_{j,u,t} - x_{j,u,r(t)}) = \gamma_j x_{j,u,t} - \gamma_j x_{j,u,r(t)}\) holds exactly. We use this property when we decompose the KPI later.
The noise is additive on \(z\), so in KPI units its standard deviation is \(\sigma_u \, y_{u,r(t)}\): large in the peak season and small in the trough.
Note
Why a custom model?
The MMM class covers a lot of ground, but the year-over-year transform changes the model itself, not only the preprocessing. Every driver has to enter as a within-pair contrast, and the difference comes after adstock and saturation. Writing the model in PyMC keeps this structure visible, while the PyMC-Marketing components still do the heavy lifting.
We use the following priors:
The adstock prior has mean \(0.25\), so most of the effect lands in the week of the spend, while its tail allows a carryover over several weeks. The steepness prior has mean \(3\). At that value, a channel at full scaled spend reaches about \(90\%\) of its maximum effect, so the data have to decide how strong the diminishing returns are.
The maximum effect has mean \(0.16\) ratio points and puts \(94\%\) of its mass below \(0.38\). Its standard deviation, about \(0.12\), is on the same scale as the true values (\(0.06\) to \(0.13\)). Keep this in mind: the recovery check runs with a prior that already knows the right order of magnitude.
The growth prior is centered at no growth and puts \(94\%\) of its mass between \(0.72\) and \(1.28\), which allows annual growth between \(-28\%\) and \(+28\%\). The control prior is weakly informative on the raw discount share. The noise prior has mean \(0.08\), so it expects noise of a few percent of the reference level and still allows more.
We define each prior once as a Prior, so that the same object feeds the model and the plots.
adstock_alpha_prior = Prior("Beta", alpha=1, beta=3, dims="channel")
saturation_lam_prior = Prior("Gamma", alpha=3, beta=1, dims="channel")
saturation_beta_prior = Prior("HalfNormal", sigma=0.2, dims="channel")
intercept_prior = Prior("Normal", mu=1.0, sigma=0.15, dims="unit")
gamma_control_prior = Prior("Normal", mu=0.0, sigma=0.5, dims="control")
z_sigma_prior = Prior("HalfNormal", sigma=0.1, dims="unit")
adstock = GeometricAdstock(l_max=l_max, priors={"alpha": adstock_alpha_prior})
saturation = LogisticSaturation(
priors={"lam": saturation_lam_prior, "beta": saturation_beta_prior}
)
Let’s visualize the priors with preliz, together with the true values that generate the data.
Every true value sits inside the bulk of its prior, so the priors do not rule out the truth, and they are wide enough to let the data decide.
Now we write the model builder. Note how the adstock and the saturation act on the whole series before we gather the current and the reference weeks.
def build_yoy_model(
obs_idx: np.ndarray,
z_observed: xr.DataArray | None = None,
) -> pm.Model:
"""Build the year-over-year ratio model on an observation window.
Parameters
----------
obs_idx : np.ndarray
Period index of every modeled week, shape ``(obs_date,)``.
z_observed : xr.DataArray, optional
Observed ratio with dims ``(obs_date, unit)``. Without it, the model gets placeholder
values of one, which is what data generation and prediction need.
Returns
-------
pm.Model
The model, with the full series of spend and controls as data.
"""
# Reference period of every modeled week: (obs_date,)
ref_idx = reference_map[obs_idx]
if np.any(ref_idx < 0):
raise ValueError("every modeled week needs a reference week")
coords = {
"date": date_index,
"obs_date": date_index[obs_idx],
"unit": unit_names,
"channel": channel_columns,
"control": control_columns,
}
if z_observed is None:
z_observed = xr.DataArray(
np.ones((len(obs_idx), n_units)),
dims=("obs_date", "unit"),
coords={"obs_date": coords["obs_date"], "unit": unit_names},
)
with pm.Model(coords=coords) as model:
# Full series of scaled spend and raw controls: (date, unit, channel), (date, unit, control)
channel_data_ = pmd.Data("channel_data", channel_scaled)
control_data_ = pmd.Data("control_data", control_data)
# Positions of the current and the reference weeks: (obs_date,)
t_sel = ptx.as_xtensor(obs_idx, dims=("obs_date",))
r_sel = ptx.as_xtensor(ref_idx, dims=("obs_date",))
# Adstock, then saturation, on the whole contiguous series: (date, unit, channel)
f_full = saturation.apply(adstock.apply(channel_data_, core_dim="date"))
# Gather at the current and at the reference week: (obs_date, unit, channel)
f_current = pmd.Deterministic(
"f_current", f_full.isel(date=t_sel), dims=("obs_date", "unit", "channel")
)
f_reference = pmd.Deterministic(
"f_reference", f_full.isel(date=r_sel), dims=("obs_date", "unit", "channel")
)
channel_contribution = pmd.Deterministic(
"channel_contribution",
f_current - f_reference,
dims=("obs_date", "unit", "channel"),
)
# Controls enter as raw differences through a linear coefficient: (obs_date, unit, control)
gamma_control = gamma_control_prior.create_variable("gamma_control", xdist=True)
control_contribution = pmd.Deterministic(
"control_contribution",
gamma_control
* (control_data_.isel(date=t_sel) - control_data_.isel(date=r_sel)),
dims=("obs_date", "unit", "control"),
)
# Growth rate of every unit: (unit,)
intercept = intercept_prior.create_variable("intercept", xdist=True)
# Mean ratio: (obs_date, unit)
mu = pmd.Deterministic(
"mu",
intercept
+ channel_contribution.sum("channel")
+ control_contribution.sum("control"),
dims=("obs_date", "unit"),
)
# Noise scale of the ratio, one per unit: (unit,)
z_sigma = z_sigma_prior.create_variable("z_sigma", xdist=True)
pmd.Normal(
"z", mu=mu, sigma=z_sigma, observed=z_observed, dims=("obs_date", "unit")
)
return model
Data Generation#
Forward Simulation#
We clamp every random variable to its true value with do, then draw one realization of \(z\). This is the strategy of Generating Synthetic MMM Data: the probabilistic model becomes a deterministic forward simulator, except for the observation noise. The generator covers every week after the first cycle, including the adstock warm-up.
# Every week that has a reference: (obs_date,)
obs_idx_all = np.arange(period, n_dates)
generator_model = do(
build_yoy_model(obs_idx_all),
{
**media_truth,
"gamma_control": gamma_control_true,
"intercept": intercept_true,
"z_sigma": z_sigma_true,
},
)
pm.model_to_graphviz(generator_model)
The clamped parameters now appear as Data nodes, and the only random node left is the ratio z. The rest of the graph is the model that we fit later: the adstock and the saturation act on the full series, and f_current and f_reference gather the result at the current and at the reference week.
We draw the ratio and keep the true mean mu too, because it defines the best possible forecast later.
with generator_model:
idata_generated = pm.sample_prior_predictive(
draws=1, var_names=["z", "mu"], random_seed=rng
)
# True ratio and true mean ratio: (obs_date, unit)
z_true = (
idata_generated.prior_predictive["z"]
.sel(chain=0, draw=0)
.transpose("obs_date", "unit")
)
mu_true = idata_generated.prior["mu"].sel(chain=0, draw=0).transpose("obs_date", "unit")
print(f"z_true: {dict(z_true.sizes)}")
Sampling: [z]
z_true: {'obs_date': 260, 'unit': 4}
From Ratios to Levels#
The first cycle is the seed: a smooth annual shape scaled by the size of each unit, with the holiday profile at its first-year position. Every later week comes from the recursion \(y_{u,t} = y_{u,r(t)} \, z_{u,t}\). So the seasonal shape, the trend and the holiday all move forward through the reference map. The moving holiday needs no separate term, because the anchored map carries the whole three-week window from one cycle to the next.
Here are two important remarks about this construction:
The model sees its own draw. The ratio \(y_{u,t} / y_{u,r(t)}\) gives back \(z\) to floating point, because the division undoes the recursion. So we fit the year-over-year model on a draw from its own prior predictive. The level construction only matters for the level-space models and for scoring in KPI units.
The seasonal shape drifts. The recursion chains across cycles, so \(\log y\) at each week of the year is a random walk across years with a step size of about \(\sigma_u\). This is a seasonal random walk, the process for which seasonal differencing is the best predictor. In other words, this panel is the home turf of the transform.
We start with the annual shape and the size of each unit.
week_of_year = np.arange(period)
# Smooth annual shape with a peak in early summer: (period,)
seasonal_shape = (
1.0
+ 0.45 * np.sin(2 * np.pi * (week_of_year - 8) / period)
+ 0.18 * np.cos(4 * np.pi * week_of_year / period)
)
# Weekly KPI of every unit at a shape value of one: (unit,)
unit_size = np.array([1000.0, 700.0, 1350.0, 900.0])
The seed puts the holiday profile into the annual shape and scales it by unit size.
def seed_first_cycle(
shape: np.ndarray, holiday_week: int, sizes: np.ndarray
) -> np.ndarray:
"""Return the levels of the first cycle, shape ``(period, unit)``."""
# Annual shape with the holiday profile at its first-year position: (period,)
seed_shape = shape.copy()
seed_shape[holiday_week + holiday_offsets] *= holiday_profile
return seed_shape[:, None] * sizes[None, :]
The recursion then rolls the seed forward, one week at a time.
def levels_from_ratios(
seed_levels: np.ndarray,
ratios: np.ndarray,
obs_idx: np.ndarray,
reference: np.ndarray,
) -> np.ndarray:
"""Apply ``y[t] = y[r(t)] * z[t]`` forward from the seed, shape ``(date, unit)``."""
levels = np.zeros((len(reference), seed_levels.shape[1]))
levels[: len(seed_levels)] = seed_levels
for position, t in enumerate(obs_idx):
levels[t] = levels[reference[t]] * ratios[position]
return levels
We combine the two helpers to build the KPI of the panel.
# KPI in levels: (date, unit)
y = levels_from_ratios(
seed_levels=seed_first_cycle(seasonal_shape, holiday_week_of_year[0], unit_size),
ratios=z_true.to_numpy(),
obs_idx=obs_idx_all,
reference=reference_map,
)
target_data = xr.DataArray(
y, dims=("date", "unit"), coords={"date": date_index, "unit": unit_names}
)
print(f"y: {y.shape} (date, unit)")
y: (312, 4) (date, unit)
Let’s check the calendar of the generated KPI. A holiday spike should appear exactly once per cycle, at the anchored position, and nowhere else. We measure each week against the mean of its two neighbors.
def neighbor_ratio(levels: np.ndarray) -> np.ndarray:
"""Return each week's level over the mean of its two neighbors, shape ``(date - 2, unit)``."""
return levels[1:-1] / ((levels[:-2] + levels[2:]) / 2)
# Largest neighbor ratio across units for every inner week: (date - 2,)
spike = neighbor_ratio(y).max(axis=1)
spike_threshold = 1.3
# The weeks above the threshold are exactly the holiday weeks.
np.testing.assert_array_equal(
np.flatnonzero(spike > spike_threshold) + 1, holiday_index
)
fig, ax = plt.subplots(figsize=(12, 4.5), layout="constrained")
ax.plot(
date_index[1:-1],
spike,
color="C0",
linewidth=1.2,
label="largest neighbor ratio across units",
)
ax.axhline(
spike_threshold,
color="black",
linestyle="--",
label=f"threshold ${spike_threshold}$",
)
for i, t in enumerate(holiday_index):
ax.axvline(
date_index[t],
color="C3",
linestyle=":",
linewidth=1.5,
label="moving holiday" if i == 0 else None,
)
ax.set(xlabel="date", ylabel="level over neighbor mean")
fig.legend(loc="outside lower center", ncols=3)
fig.suptitle("Holiday Spike Check", fontsize=16, fontweight="bold");
Exactly six weeks pass the threshold, one per cycle, and each one sits on a holiday line. So the anchored map carries the holiday forward without copying it onto a normal week. The smooth stretch in 2019 is the noise-free seed cycle.
Exploratory Data Analysis#
We should always start by looking at the data! Let’s plot the KPI of every unit.
The seasonal amplitude is large and grows with the level: the summer peak is about three times the winter trough. The holiday spike moves between weeks \(12\) and \(15\), in March and April. The units differ in size and growth, and the seasonal shape drifts slowly from year to year, as a seasonal random walk should.
Now let’s look at the same series after the transform, with the anchored map and with the naive lag.
# Observed ratio with the anchored and with the naive map: (obs_date, unit)
z_observed = xr.DataArray(
y[obs_idx_model] / y[reference_map[obs_idx_model]],
dims=("obs_date", "unit"),
coords={"obs_date": date_index[obs_idx_model], "unit": unit_names},
)
z_naive = xr.DataArray(
y[obs_idx_model] / y[naive_reference_map[obs_idx_model]],
dims=("obs_date", "unit"),
coords={"obs_date": date_index[obs_idx_model], "unit": unit_names},
)
fig, axes = plt.subplots(
nrows=2, figsize=(13, 8), sharex=True, sharey=True, layout="constrained"
)
for ax, (title, ratio) in zip(
axes,
[("anchored reference map", z_observed), (r"naive lag $r(t) = t - 52$", z_naive)],
strict=True,
):
for unit in unit_names:
ax.plot(ratio["obs_date"], ratio.sel(unit=unit), linewidth=1.2, label=unit)
for i, t in enumerate(holiday_index[1:]):
ax.axvline(
date_index[t],
color="C3",
linestyle="--",
alpha=0.6,
label="moving holiday" if i == 0 else None,
)
ax.axhline(1.0, color="black", alpha=0.4)
ax.set(title=title, ylabel="$z$")
handles, labels = axes[0].get_legend_handles_labels()
fig.legend(handles, labels, loc="outside lower center", ncols=n_units + 1)
fig.suptitle(
"Year-over-Year Ratio: Anchored and Naive Reference Maps",
fontsize=16,
fontweight="bold",
);
With the anchored map, the ratio fluctuates around the growth rate of each unit. With the naive lag, the same series has a spike and a crater at every moving holiday, because a holiday week is divided by a normal week and the other way around. These are the weeks where the KPI is largest, so a model fitted on the naive ratio would spend its media coefficients on a calendar artifact.
The bar chart below puts a number on the damage.
The naive lag almost doubles the standard deviation of \(z\), and it more than triples its largest deviation from one.
Note
On this panel the comparison is circular, because we generated the levels through the anchored map. The figure shows the size of the artifact that a wrong map produces. It is not evidence that this map is right for real data.
Train Test Split#
We hold out the last \(52\) weeks and train on all modeled weeks before them. Every reference week of the holdout window sits inside the training window, so the holdout prediction only needs observed history, not model output.
n_test = period
# Training and test weeks: (obs_date,)
obs_idx_train = obs_idx_model[:-n_test]
obs_idx_test = obs_idx_model[-n_test:]
train_dates = date_index[obs_idx_train]
test_dates = date_index[obs_idx_test]
z_train = z_observed.isel(obs_date=slice(None, -n_test))
print(
f"train: {train_dates[0].date()} to {train_dates[-1].date()} ({len(obs_idx_train)} weeks)"
)
print(
f"test: {test_dates[0].date()} to {test_dates[-1].date()} ({len(obs_idx_test)} weeks)"
)
train: 2020-03-02 to 2023-12-25 (200 weeks)
test: 2024-01-01 to 2024-12-23 (52 weeks)
Prior Predictive Checks#
We build the model on the training window and look at its graph.
yoy_model = build_yoy_model(obs_idx_train, z_train)
pm.model_to_graphviz(yoy_model)
The graph shows the structure of the transform. The adstock and saturation parameters act on the full series (date), and the result is gathered twice, as f_current and f_reference (obs_date). Their difference, the control contrast and the growth rate intercept add up to mu.
Next, we sample from the prior predictive distribution.
with yoy_model:
yoy_prior = pm.sample_prior_predictive(draws=500, random_seed=rng)
Let’s compare the prior predictive ratio with the observed ratio of north.
The prior \(94\%\) HDI covers ratios from about \(0.6\) to \(1.4\), so before seeing the data the priors allow year-over-year swings of a few tens of percent. This is wide around the observed ratio, but not absurd for a sales KPI.
Model Fitting#
We are now ready to fit the model.
with yoy_model:
yoy_idata = pm.sample(
chains=4,
tune=1_000,
draws=1_000,
target_accept=0.9,
nuts_sampler="nutpie",
random_seed=rng,
)
pm.sample_posterior_predictive(
yoy_idata, random_seed=rng, extend_inferencedata=True
)
Model Diagnostics#
Before looking into the results, let’s check some diagnostics. We start with the number of divergences.
n_divergences = int(yoy_idata["sample_stats"]["diverging"].sum().item())
print(f"Number of divergences: {n_divergences}")
Number of divergences: 0
Next, arviz.diagnose() runs the standard sampler checks in one call: divergences, energy, effective sample size and \(\hat{R}\).
yoy_var_names = [
"adstock_alpha",
"saturation_lam",
"saturation_beta",
"gamma_control",
"intercept",
"z_sigma",
]
sampler_ok = not az.diagnose(yoy_idata, var_names=yoy_var_names)
r_hat_max = max(
float(values.max())
for values in az.rhat(yoy_idata, var_names=yoy_var_names).data_vars.values()
)
print(f"max r_hat (unrounded): {r_hat_max:.4f}")
print(f"sampler checks passed: {sampler_ok}")
Divergences
No divergent transitions found.
E-BFMI
E-BFMI satisfactory for all chains.
ESS
Effective sample size satisfactory for all parameters.
R-hat
R-hat values satisfactory for all parameters.
Processing complete, no problems detected.
max r_hat (unrounded): 1.0029
sampler checks passed: True
Finally, we look at the traces of the media parameters.
trace_plot = azp.plot_trace_dist(
yoy_idata,
var_names=["adstock_alpha", "saturation_lam", "saturation_beta"],
aes={"color": ["channel"]},
figure_kwargs={"figsize": (12, 9), "layout": "constrained"},
)
trace_plot.add_legend("channel", loc="outside lower center", ncols=n_channels)
trace_plot.viz["figure"].item().suptitle(
"Trace Plots of the Media Parameters", fontsize=16, fontweight="bold"
);
We see a good mixing in the traces, consistent with zero divergences and a largest \(\hat{R}\) of \(1.003\). Note that the sponsorship traces wander over the whole range of their priors. This is the first sign of what we discuss in the parameter recovery section.
Posterior Predictive Checks#
As our model and posterior samples are looking good, we can now look into the posterior predictive checks. We compare the posterior predictive ratio with the observed ratio of every unit.
Let’s put a number on the fit: the share of training weeks inside the \(94\%\) HDI of every unit.
in_sample_hdi = az.hdi(z_posterior_predictive, prob=0.94)
in_sample_coverage = (
(z_train >= in_sample_hdi.sel(ci_bound="lower"))
& (z_train <= in_sample_hdi.sel(ci_bound="upper"))
).mean("obs_date")
print(
f"in-sample 94% HDI coverage: {in_sample_coverage.round(3).to_series().to_dict()}"
)
in-sample 94% HDI coverage: {'north': 0.93, 'south': 0.955, 'east': 0.93, 'west': 0.945}
The posterior predictive ratio looks quite reasonable. The in-sample coverage of the \(94\%\) HDI ranges from \(93.0\%\) to \(95.5\%\) across the four units, close to nominal.
Parameter Recovery#
The data-generating process is this model, so the posteriors should concentrate on the true values. Where they do not is the interesting part. We start with the growth rates, the control coefficient and the noise scales.
These three groups live on very different scales, so we divide every posterior draw by its true value. On this scale a ratio of one means perfect recovery, and all parameters fit in one forest plot. We write a small helper around arviz.plot_forest(), and we reuse it for the contributions and for the comparison with the level models.
Let’s compare the growth rates, the control coefficient and the noise scales with their true values.
# True values with the same dims as their posteriors: (unit,) and (control,)
scalar_truth = xr.Dataset(
{
"intercept": ("unit", intercept_true),
"gamma_control": ("control", gamma_control_true),
"z_sigma": ("unit", z_sigma_true),
},
coords={"unit": unit_names, "control": control_columns},
)
# Every posterior draw over its true value: (chain, draw, unit) and (chain, draw, control)
scalar_ratio = yoy_idata.posterior.to_dataset()[list(scalar_truth)] / scalar_truth
fig, _ = plot_posterior_forest(
scalar_ratio,
reference=1.0,
reference_label="true value",
xlabel="posterior / true value",
figsize=(12, 6),
)
fig.suptitle(
"Parameter Recovery: Growth, Control and Noise", fontsize=16, fontweight="bold"
);
Every growth rate sits within half a percent of its true value. gamma_control sits about \(5\%\) below its true value, which stays inside its \(94\%\) HDI. The noise scales follow the realized noise of the training weeks rather than the true values. For west, the posterior sits about \(14\%\) below the true value, outside its \(94\%\) HDI, because the training weeks of west happened to be calm. We come back to this in the coverage section.
Next, let’s compare prior, posterior and truth for the media parameters.
tv and search are recovered: the true values sit inside the bulk of their posteriors for all three parameters, and the posteriors are much narrower than the priors. sponsorship is not recovered. Its posteriors match its priors.
The Always-On Channel#
Why does sponsorship not move? Its spend is identical from cycle to cycle, so \(f_{c,u,t} - f_{c,u,r(t)} = 0\) for every modeled week. The channel drops out of the likelihood entirely, and its posterior is its prior. This is also why the modeled window starts \(l_{\max}\) weeks after the first cycle. Without that warm-up, the reference weeks at the very start of the series would carry a partially built adstock, and the always-on channel would leak a small amount of spurious information.
This is a property of the design, not a bug. But in practice it is one of the main traps of the transform: a user who reads a tight posterior interval on such a channel reads a prior and calls it evidence.
Let’s be precise about what is specific to the transform. A channel whose spend is constant within each unit is collinear with a per-unit intercept in any model, including the level-space MMMs below, which is why we leave it out of them. A plan that repeats from cycle to cycle with any shape is confounded with the seasonal basis of a level model. So the transform does not create the problem. It makes it exact and detectable: the contrast is identically zero, and the ratio of posterior to prior standard deviation says so.
Let’s compute this ratio for every media parameter.
def sd_ratio(var: str) -> pd.Series:
"""Return the posterior standard deviation over the prior one, per channel."""
prior_sd = yoy_prior.prior[var].std(("chain", "draw"))
posterior_sd = yoy_idata.posterior[var].std(("chain", "draw"))
return (posterior_sd / prior_sd).to_series()
A ratio near \(1\) means that the likelihood said nothing. sponsorship sits there for all three parameters (\(1.02\), \(0.98\) and \(1.05\)), and no amount of data collected under a flat plan will move it. The other two channels shrink to between \(0.11\) and \(0.44\) of their prior width. In other words, a year-over-year target model measures plan changes, not programs.
Media Deep Dive#
Now that we are happy with the model fit, we can look into the media and individual channel contributions.
Channel Contributions#
The model works in ratio points, so we first convert every part of \(\mu_{u,t}\) to KPI units. We do this per unit-period: we multiply each ratio-space part by that unit-period’s own reference level \(y_{u,r(t)}\), and only then sum. Averaging the ratio contributions across the panel and scaling the average by one reference factor gives a different number, and the gap grows with the spread of the unit sizes.
The closure check below confirms that the converted parts plus the converted residual give back the observed level \(y_{u,t}\) at every unit-period. This checks the conversion: it fails if the reference level that converts the parts is not the one that defined \(z\), or if we aggregate the parts before the conversion. It does not check how we split \(\mu\) into parts. Any split that sums to \(\mu\) passes, so a constant moved between the baseline and a control would go unnoticed. The split is a modeling decision, and the next subsection makes it explicit.
posterior = yoy_idata.posterior.to_dataset()
# Reference and observed levels of the training weeks, in KPI units: (obs_date, unit)
y_reference_train = xr.DataArray(
y[reference_map[obs_idx_train]],
dims=("obs_date", "unit"),
coords={"obs_date": train_dates, "unit": unit_names},
)
y_observed_train = xr.DataArray(
y[obs_idx_train],
dims=("obs_date", "unit"),
coords={"obs_date": train_dates, "unit": unit_names},
)
# Parts of mu on the contrast basis, in ratio points: (chain, draw, obs_date, unit)
parts_ratio = {
"baseline": posterior["intercept"].broadcast_like(posterior["mu"]),
**{
channel: posterior["channel_contribution"].sel(channel=channel, drop=True)
for channel in channel_columns
},
**{
control: posterior["control_contribution"].sel(control=control, drop=True)
for control in control_columns
},
}
# Every part and the residual, converted with their own unit-period reference level: (chain, draw, obs_date, unit)
parts_kpi = {name: part * y_reference_train for name, part in parts_ratio.items()}
residual_kpi = (z_train - posterior["mu"]) * y_reference_train
# The converted parts and residual give back the observed level at every unit-period.
np.testing.assert_allclose(
(sum(parts_kpi.values()) + residual_kpi)
.mean(("chain", "draw"))
.transpose("obs_date", "unit")
.to_numpy(),
y_observed_train.transpose("obs_date", "unit").to_numpy(),
rtol=1e-8,
)
print("per unit-period closure against the observed level passed")
per unit-period closure against the observed level passed
We summarize each part by its posterior total over the training weeks and the units.
def window_totals(parts: dict[str, xr.DataArray]) -> xr.Dataset:
"""Sum every part over the training weeks and the units, one variable per part."""
return xr.Dataset(
{name: part.sum(("obs_date", "unit")) for name, part in parts.items()}
)
Let’s look at the media and control totals on the contrast basis. The baseline total is much larger, so we print it instead.
# Posterior total of every part: (chain, draw)
contrast_totals = window_totals(parts_kpi)
print(
f"baseline total on the contrast basis: {float(contrast_totals['baseline'].mean()):,.0f}"
)
fig, _ = plot_posterior_forest(
contrast_totals.drop_vars("baseline"),
reference=0.0,
reference_label="zero",
xlabel="total over the training weeks and units (KPI units)",
figsize=(12, 4.5),
)
fig.suptitle("Contrast Basis: Window Totals", fontsize=16, fontweight="bold");
A contrast-basis total can be negative. It is the effect of the change in plan versus last cycle, so a channel whose spend fell relative to the previous cycle contributed negatively: tv and discount total about \(-160\) and \(-220\) KPI units, while search totals about \(300\). This is the correct answer to the question that we asked, but it is easy to misread as a total contribution.
Note also how small these totals are against the baseline total of \(831K\). The contrasts average out over a window in which spend went up in some years and down in others. So the window total says little about a channel, even when every weekly contrast is well measured. sponsorship is exactly zero.
The Level Basis#
The totals above answer the question “what did the change in plan versus last cycle contribute?”. We read them directly off the fitted terms, and the likelihood identifies them.
The other question, “what did this channel contribute in total?”, needs the exact regrouping
for a declared reference spend \(a^\ast\). The regrouping is exact, but the number depends on the choice. With \(a^\ast = 0\), it depends on the shape of \(f_c\) over a range that the data never visited. Under logistic saturation \(f_c(0) = 0\), so the level read is simply \(f_{c,u,t}\), and the baseline absorbs \(-f_{c,u,r(t)}\).
The controls get the same treatment with \(x_j^\ast = 0\). Because they are linear, the level read is \(\gamma_j x_{j,u,t}\) and the baseline absorbs \(-\gamma_j x_{j,u,r(t)}\). Together this gives a decomposition of the KPI into baseline, channels and control, in KPI units:
# Raw control values at the current and at the reference week: (obs_date, unit, control)
control_values = control_data.transpose("date", "unit", "control").to_numpy()
x_current = xr.DataArray(
control_values[obs_idx_train],
dims=("obs_date", "unit", "control"),
coords={"obs_date": train_dates, "unit": unit_names, "control": control_columns},
)
x_reference = xr.DataArray(
control_values[reference_map[obs_idx_train]],
dims=("obs_date", "unit", "control"),
coords={"obs_date": train_dates, "unit": unit_names, "control": control_columns},
)
# Saturated spend at the current and at the reference week: (chain, draw, obs_date, unit, channel)
f_current = posterior["f_current"]
f_reference = posterior["f_reference"]
# Control coefficient: (chain, draw, control)
gamma = posterior["gamma_control"]
# Parts of mu on the level basis with a* = 0 and x* = 0, in KPI units: (chain, draw, obs_date, unit)
level_parts_kpi = {
"baseline": (
posterior["intercept"]
- f_reference.sum("channel")
- (gamma * x_reference).sum("control")
)
* y_reference_train,
**{
channel: f_current.sel(channel=channel, drop=True) * y_reference_train
for channel in channel_columns
},
**{
control: (gamma * x_current).sel(control=control, drop=True) * y_reference_train
for control in control_columns
},
}
# The regrouping is exact: both bases add up to the same total at every unit-period.
np.testing.assert_allclose(
sum(level_parts_kpi.values())
.mean(("chain", "draw"))
.transpose("obs_date", "unit")
.to_numpy(),
sum(parts_kpi.values())
.mean(("chain", "draw"))
.transpose("obs_date", "unit")
.to_numpy(),
rtol=1e-8,
)
print(f"smallest observed discount: {float(control_data.min()):.4f}")
smallest observed discount: 0.0004
The smallest observed discount is close to zero, so \(x_j^\ast = 0\) sits inside the observed range and the control’s level read is not an extrapolation. Here are the totals on the level basis.
# Posterior total of every part: (chain, draw)
level_totals = window_totals(level_parts_kpi)
print(
f"baseline total on the level basis: {float(level_totals['baseline'].mean()):,.0f}"
)
fig, _ = plot_posterior_forest(
level_totals.drop_vars("baseline"),
reference=0.0,
reference_label="zero",
xlabel="total over the training weeks and units (KPI units)",
figsize=(12, 4.5),
)
fig.suptitle(
r"Level Basis ($a^\ast = 0$): Window Totals", fontsize=16, fontweight="bold"
);
The two bases answer two different questions about the same fit, and the totals are far larger on the level basis: tv, search and discount each total between about \(18K\) and \(30K\) KPI units.
Here is an important observation: sponsorship becomes the largest media contributor at about \(90K\), and its \(94\%\) HDI runs from close to zero to more than twice that value. This number comes from the prior, not from the data. The baseline shrinks from \(831K\) to \(669K\), because it hands the levels of the channels and of the control back to them.
Contribution Decomposition#
The level basis gives the familiar decomposition of the KPI into baseline, channels and control. We plot the posterior mean of every part, summed over the units, together with the observed total.
# Posterior mean of every level-basis part, summed over the units: (obs_date,)
decomposition = {
name: part.mean(("chain", "draw")).sum("unit")
for name, part in level_parts_kpi.items()
}
modeled_total = float(sum(part.sum() for part in decomposition.values()))
print(
"share of the modeled KPI: "
+ ", ".join(
f"{name} {float(part.sum()) / modeled_total:.1%}"
for name, part in decomposition.items()
)
)
fig, ax = plt.subplots(figsize=(13, 6), layout="constrained")
ax.stackplot(
train_dates,
*decomposition.values(),
labels=list(decomposition),
colors=["0.75", "C0", "C1", "C2", "C4"],
alpha=0.85,
)
ax.plot(
train_dates,
y_observed_train.sum("unit"),
color="black",
linewidth=1.2,
label="observed KPI",
)
ax.set(xlabel="date", ylabel="KPI, sum over units")
fig.legend(loc="outside lower center", ncols=6)
fig.suptitle(
"Contribution Decomposition on the Level Basis", fontsize=16, fontweight="bold"
);
The baseline carries the seasonal shape and the holiday, and it makes up \(80.4\%\) of the modeled KPI. The channels and the control add a thin layer on top: tv \(3.5\%\), search \(2.1\%\) and discount \(2.9\%\).
The sponsorship slice is the thickest media layer at \(11.0\%\), but we know from the bars above that its size is the prior’s guess. The plot itself does not show this. So we always read a decomposition together with the identification checks.
Saturation Curves#
The year-over-year model has a response curve per channel, in ratio points: \(f_c(\tilde{a})\) is the lift as a share of last year’s level. To express it in KPI units we need a reference level, because the response is proportional to \(y_{u,r(t)}\). The same spend buys more KPI in the peak season than in the trough. So a single curve in KPI units is an approximation: we scale the ratio-point curve by the average reference level of a unit-week in the training window.
Recall that the adstock weights sum to one, so a constant weekly spend \(a\) has the adstocked value \(a\). We can therefore read the horizontal axis as a steady weekly spend.
# Saturation curves in ratio points on a grid of adstocked scaled spend: (chain, draw, x, channel)
curve_parameters = ["saturation_lam", "saturation_beta"]
curve_prior = saturation.sample_curve(
yoy_prior.prior.to_dataset()[curve_parameters], max_value=1.0
)
curve_posterior = saturation.sample_curve(posterior[curve_parameters], max_value=1.0)
# True curve: (x, channel)
curve_truth = saturation.sample_curve(
xr.Dataset(
{
name: (("chain", "draw", "channel"), media_truth[name][None, None, :])
for name in curve_parameters
},
coords={"chain": [0], "draw": [0], "channel": channel_columns},
),
max_value=1.0,
).isel(chain=0, draw=0)
# Average reference level of a unit-week in the training window, in KPI units.
average_reference = float(y_reference_train.mean())
# Range of the scaled spend over the modeled weeks: (quantile, channel)
spend_range = channel_scaled.isel(date=obs_idx_model).quantile(
[0.05, 0.95], dim=("date", "unit")
)
print(f"average reference level: {average_reference:.0f} KPI units per unit-week")
Sampling: []
Sampling: []
Sampling: []
average reference level: 1027 KPI units per unit-week
Let’s plot the prior band, the posterior band and the true curve of every channel.
For tv and search, the posterior band is narrow and contains the true curve over the observed spend range. Beyond that range the band widens, and for tv the posterior mean drifts slightly below the truth at full spend. For sponsorship, the posterior band is the prior band, and the posterior mean is more than twice the true curve. Its spend takes only four values, one per unit, and its year-over-year contrast is zero, so the data cannot shape this curve at all.
Keep in mind that the KPI scale of these curves rests on the average reference level of \(1027\) KPI units per unit-week. In the peak season, the same spend buys about twice as much.
Leverage#
How much does the level read rely on the shape of the curve? The level read is levered off the measured contrast by roughly
For a channel whose spend barely moved from cycle to cycle, this is a large multiplier on a small measured quantity.
# Mean absolute contrast and level read, in ratio points: (channel,)
leverage = pd.DataFrame(
{
"measured contrast": np.abs(posterior["channel_contribution"])
.mean(("chain", "draw", "obs_date", "unit"))
.to_series(),
"level read": np.abs(f_current)
.mean(("chain", "draw", "obs_date", "unit"))
.to_series(),
}
)
leverage["leverage"] = leverage["level read"] / leverage["measured contrast"]
fig, axes = plt.subplots(
ncols=2, figsize=(14, 5), width_ratios=[2, 1], layout="constrained"
)
leverage[["measured contrast", "level read"]].plot.bar(
ax=axes[0], rot=0, color=["C0", "C1"], legend=False
)
axes[0].set(xlabel="channel", ylabel="mean absolute value (ratio points)")
bars = axes[1].bar(
leverage.index,
leverage["leverage"].where(np.isfinite(leverage["leverage"]), 0.0),
color="C2",
)
# Mark the channel whose measured contrast is exactly zero, so that its leverage is infinite.
axes[1].bar_label(
bars,
labels=[
"" if np.isfinite(value) else r"$\infty$" for value in leverage["leverage"]
],
padding=3,
fontsize=16,
)
axes[1].axhline(1.0, color="black", linestyle="--")
axes[1].set(xlabel="channel", ylabel="leverage")
axes[1].margins(y=0.15)
handles, labels = axes[0].get_legend_handles_labels()
fig.legend(
[*handles, Line2D([], [], color="black", linestyle="--")],
[*labels, "level read equal to the measured contrast"],
loc="outside lower center",
ncols=3,
)
fig.suptitle(
"Leverage of the Level Read over the Measured Contrast",
fontsize=16,
fontweight="bold",
);
The leverage is about \(2.6\) for tv, close to one for search and infinite for sponsorship. A value near one means that the level read is about as well measured as the contrast, which is the case for search. A large value means that the level number extrapolates the saturation curve into territory that the data never visited.
For sponsorship, the measured contrast is exactly zero, so there is no defensible total-contribution number for this channel under this parametrization, at any posterior width. Budget optimization and response curves live on the level basis, so they inherit this dependence.
Level-Space Benchmarks#
For the comparison we fit standard multidimensional MMM models on the same panel, in level space, with yearly Fourier seasonality and the same adstock and saturation families. We fit two of them:
Level + Fourier: four yearly harmonics, the discount control and a constant per-unit intercept. This is the usual vanilla comparison. It misses two things that the panel contains: a compounding per-unit trend, and a moving holiday that a smooth annual basis cannot represent.
Level + Fourier + holiday + trend: the same model with a linear trend control and three holiday indicators (the week before, the peak and the week after). These features only use the calendar, which a practitioner would also have.
Both models see the same training window and the same scaled spend as the ratio model, with two differences. They leave out sponsorship, because its spend is constant within each unit: in a level model it is collinear with the per-unit intercept (see the always-on section), and the intercept absorbs its effect. They also start the adstock from zero in the first training week, while the ratio model applies it to the full series. So their carryover is only partly built in the first \(l_{\max} = 8\) of the \(200\) training weeks. This is a small handicap that we accept to keep the models standard.
First, we build the long data frame that the MMM class expects, with the calendar features.
# Media channels of the level models, without the always-on channel.
level_channels = ["tv", "search"]
# Indicator of every week of the holiday window: name -> (date,)
holiday_dummies = {
f"holiday_{label}": np.isin(np.arange(n_dates), np.array(holiday_index) + offset)
for label, offset in zip(["before", "peak", "after"], holiday_offsets, strict=True)
}
level_extra_controls = ["trend", *holiday_dummies]
def make_panel_df(target: xr.DataArray) -> pd.DataFrame:
"""Return the long panel with scaled spend, controls, calendar features and the target."""
calendar_df = pd.DataFrame(
{
"date": date_index,
"trend": np.arange(n_dates) / period,
**{name: values.astype(float) for name, values in holiday_dummies.items()},
}
)
return (
channel_scaled.sel(channel=level_channels)
.to_dataset(dim="channel")
.to_dataframe()
.reset_index()
.merge(
control_data.sel(control="discount", drop=True)
.rename("discount")
.to_dataframe()
.reset_index(),
on=["date", "unit"],
)
.merge(target.rename("y").to_dataframe().reset_index(), on=["date", "unit"])
.merge(calendar_df, on="date")
)
Next, the fitting helper. It uses the same adstock and saturation families as the ratio model, with a wider beta prior because the level model works on the internally scaled KPI. At low spend the logistic saturation is close to linear, so the data pin down the product of its steepness and its maximum better than either of the two. The level models see the media signal through more residual noise than the ratio model, so this trade-off is stronger for them. We therefore sample them with target_accept=0.99 and let nutpie adapt a low-rank mass matrix, which can follow the correlation between the two parameters.
def fit_level_model(
train_df: pd.DataFrame, controls: list[str]
) -> tuple[MMM, xr.DataTree]:
"""Fit a level-space MMM with yearly Fourier seasonality on a training panel."""
mmm = MMM(
date_column="date",
channel_columns=level_channels,
control_columns=controls,
target_column="y",
adstock=GeometricAdstock(
l_max=l_max,
priors={"alpha": Prior("Beta", alpha=1, beta=3, dims="channel")},
),
saturation=LogisticSaturation(
priors={
"lam": Prior("Gamma", alpha=3, beta=1, dims="channel"),
"beta": Prior("HalfNormal", sigma=2, dims="channel"),
}
),
yearly_seasonality=4,
dims=("unit",),
)
X_train = train_df.drop(columns=["y"])
y_train = train_df["y"]
mmm.build_model(X_train, y_train)
# Register the target on its original scale. Without it, the posterior predictive comes back
# on the internal scale and is not comparable with the year-over-year predictions.
mmm.add_original_scale_contribution_variable(var=["y"])
idata = mmm.fit(
X_train,
y_train,
chains=4,
tune=1_000,
draws=1_000,
target_accept=0.99,
nuts_sampler="nutpie",
nuts={"adaptation": "low_rank"},
random_seed=rng,
)
return mmm, idata
We fit both level models on the training weeks.
panel_df = make_panel_df(target_data)
train_df = panel_df[panel_df["date"].isin(train_dates)].reset_index(drop=True)
test_df = panel_df[panel_df["date"].isin(test_dates)].reset_index(drop=True)
level_mmm, level_idata = fit_level_model(train_df, controls=["discount"])
level_full_mmm, level_full_idata = fit_level_model(
train_df, controls=["discount", *level_extra_controls]
)
Before using the level models, let’s run the same sampler checks as for the ratio model.
level_fits = {
"level + Fourier": (level_mmm, level_idata),
"level + Fourier + holiday + trend": (level_full_mmm, level_full_idata),
}
for name, (mmm, idata) in level_fits.items():
print(f"--- {name} ---")
level_var_names = [rv.name for rv in mmm.model.free_RVs]
sampler_ok = not az.diagnose(idata, var_names=level_var_names)
r_hat_max = max(
float(values.max())
for values in az.rhat(idata, var_names=level_var_names).data_vars.values()
)
print(f"max r_hat (unrounded): {r_hat_max:.4f}")
print(f"sampler checks passed: {sampler_ok}\n")
--- level + Fourier ---
Divergences
No divergent transitions found.
E-BFMI
E-BFMI satisfactory for all chains.
ESS
Effective sample size satisfactory for all parameters.
R-hat
R-hat values satisfactory for all parameters.
Processing complete, no problems detected.
max r_hat (unrounded): 1.0090
sampler checks passed: True
--- level + Fourier + holiday + trend ---
Divergences
No divergent transitions found.
E-BFMI
E-BFMI satisfactory for all chains.
ESS
Effective sample size satisfactory for all parameters.
R-hat
R-hat values satisfactory for all parameters.
Processing complete, no problems detected.
max r_hat (unrounded): 1.0045
sampler checks passed: True
Both level models pass all checks, with no divergences and a largest \(\hat{R}\) of \(1.009\) and \(1.005\). Without sponsorship, nothing in them is collinear with the per-unit intercept. With the channel in, the data would pin down little more than the sum of each intercept and the sponsorship effect. The sampler would then wander along a ridge of parameter values that fit equally well. This ridge is the level-space face of the standard deviation ratio of one in the ratio model. Both say that the data carry no information about the channel, but only the ratio model can keep the channel and report this as a clean number.
adstock_alpha and saturation_lam act on the scaled spend, which is the same in all models, so we can compare them across the two model families. saturation_beta lives in ratio points in one family and in internally scaled KPI units in the other, so it is not comparable. The level models have no sponsorship parameters, so we compare tv and search.
# True media parameters for the two parameters that both families share: (channel,)
shared_truth = xr.Dataset(
{
name: ("channel", media_truth[name])
for name in ["adstock_alpha", "saturation_lam"]
},
coords={"channel": channel_columns},
).sel(channel=level_channels)
# Every posterior draw over its true value, per model family: (chain, draw, channel)
cross_model_ratio = {
name: model_posterior.to_dataset()[list(shared_truth)].sel(channel=level_channels)
/ shared_truth
for name, model_posterior in {
"year-over-year": yoy_idata.posterior,
"level + Fourier + holiday + trend": level_full_idata.posterior,
}.items()
}
fig, _ = plot_posterior_forest(
cross_model_ratio,
reference=1.0,
reference_label="true value",
xlabel="posterior / true value",
figsize=(12, 6),
)
fig.suptitle(
"Media Parameters Across the Two Model Families", fontsize=16, fontweight="bold"
);
The intervals of the ratio model are narrower and contain the true values. The full level model has wider intervals, and all four of its means sit below the truth.
Out-of-Sample Comparison#
All models now predict the last \(52\) weeks in KPI units, the only scale on which they are comparable:
Year-over-year: the model predicts \(z\) on the holdout window and multiplies it by the observed reference level \(y_{u,r(t)}\), which sits entirely inside the training window.
Level models: the models predict \(y\) directly, with
include_last_observations=Trueso that their adstock carries over from the training window.
Our main metric is the continuous ranked probability score (CRPS). It scores the whole predictive distribution against the observed value, and lower is better. For a point forecast it reduces to the mean absolute error, so we can read it in KPI units (see Time-Slice-Cross-Validation and Parameter Stability). We also report the RMSE, the MAPE, the coverage and the width of the \(94\%\) HDI, and the share of the squared error that falls in the holiday window.
A comparison of two models alone invites the wrong reading, so we add reference forecasts that locate the gain. Each one is a full predictive distribution, so that every score applies to it:
Noise floor: the true model, that is the true \(\mu\) times the reference level with the true noise. No model can do better than this in expectation.
Seasonal naive: last year’s level times a growth factor, with Gaussian noise on the ratio fitted on the training weeks. We use it with no growth, \(\hat{z} = 1\), and with the mean training ratio of each unit as the growth. Neither version has media or controls.
Whatever the year-over-year model gains beyond the naive forecast is what media and the control buy. Whatever the level models lose against it is what a level-space seasonal basis costs on this panel.
We start with the holdout forecast of the ratio model.
def forecast_yoy(idata: xr.DataTree, levels: np.ndarray) -> xr.DataArray:
"""Return the holdout predictive of the ratio model in KPI units, dims (chain, draw, date, unit)."""
predict_model = build_yoy_model(obs_idx_test)
# Only the free parameters travel to the prediction model. The stored deterministics have the
# shape of the training window and would clash with the holdout window.
free_names = [rv.name for rv in predict_model.free_RVs]
posterior_free = idata.posterior.to_dataset()[free_names]
with predict_model:
forecast = pm.sample_posterior_predictive(
posterior_free, var_names=["z"], random_seed=rng, predictions=True
)
# Observed reference level of every holdout week: (obs_date, unit)
reference = xr.DataArray(
levels[reference_map[obs_idx_test]],
dims=("obs_date", "unit"),
coords={"obs_date": test_dates, "unit": unit_names},
)
return (forecast.predictions["z"] * reference).rename({"obs_date": "date"})
The level models predict in KPI units directly.
def predict_level_model(mmm: MMM, test_df: pd.DataFrame) -> xr.DataArray:
"""Return the holdout predictive of a level-space MMM in KPI units."""
pp = mmm.sample_posterior_predictive(
X=test_df.drop(columns=["y"]),
extend_idata=False,
combined=False,
include_last_observations=True,
var_names=["y", "y_original_scale"],
random_seed=rng,
)
return pp["y_original_scale"]
The reference forecasts are Gaussian in KPI units. We draw them with a separate generator, so that their draws leave the random stream of the fits further below unchanged.
reference_rng = np.random.default_rng(seed=seed + 1)
def normal_forecast(
mean: np.ndarray, sd: np.ndarray, n_draws: int = 4_000
) -> xr.DataArray:
"""Return Gaussian predictive draws over the holdout, dims (chain, draw, date, unit)."""
# Draws around a mean shaped (date, unit): (1, draw, date, unit)
draws = mean + sd * reference_rng.standard_normal((1, n_draws, *mean.shape))
return xr.DataArray(
draws,
dims=("chain", "draw", "date", "unit"),
coords={"date": test_dates, "unit": unit_names},
)
def seasonal_naive(
levels: np.ndarray, z_fit: xr.DataArray, fitted_growth: bool
) -> xr.DataArray:
"""Forecast the holdout as last cycle's level times a growth factor, with Gaussian ratio noise."""
# Observed reference level of every holdout week: (date, unit)
reference = levels[reference_map[obs_idx_test]]
if fitted_growth:
growth = z_fit.mean("obs_date").to_numpy()
scale = z_fit.std("obs_date", ddof=1).to_numpy()
else:
growth = np.ones(n_units)
scale = np.sqrt(((z_fit - 1.0) ** 2).mean("obs_date")).to_numpy()
return normal_forecast(mean=reference * growth, sd=reference * scale)
The scoring helper computes every metric in KPI units for a predictive distribution with dims (chain, draw, date, unit).
# Holdout weeks inside a holiday window: (date,)
is_holiday_test = xr.DataArray(
np.isin(obs_idx_test, holiday_weeks), dims="date", coords={"date": test_dates}
)
def score(pred: xr.DataArray, actual: xr.DataArray) -> pd.Series:
"""Return the holdout scores of a predictive distribution, in KPI units."""
actual = actual.transpose("date", "unit")
# Same date and unit order as the observations, because `crps` works on plain arrays.
pred = pred.sel(
date=actual["date"].to_numpy(), unit=actual["unit"].to_numpy()
).transpose("chain", "draw", "date", "unit")
# Draws on the left, as `crps` expects: (sample, date, unit)
samples = (
pred.stack(sample=("chain", "draw"))
.transpose("sample", "date", "unit")
.to_numpy()
)
# Error of the predictive mean: (date, unit)
error = pred.mean(("chain", "draw")) - actual
hdi = az.hdi(pred, prob=0.94)
lower, upper = hdi.sel(ci_bound="lower"), hdi.sel(ci_bound="upper")
return pd.Series(
{
"CRPS": crps(actual.to_numpy(), samples),
"RMSE": float(np.sqrt((error**2).mean())),
"MAPE (%)": float((np.abs(error) / actual).mean() * 100),
"94% HDI coverage": float(((actual >= lower) & (actual <= upper)).mean()),
"94% HDI width": float((upper - lower).mean()),
"holiday share of squared error": float(
(error**2).where(is_holiday_test).sum() / (error**2).sum()
),
}
)
We show the scores as one bar chart per metric, with the CRPS first.
Now we compute every holdout forecast and score it.
y_test = target_data.sel(date=test_dates)
# Observed reference level of every holdout week: (date, unit)
y_reference_test = y[reference_map[obs_idx_test]]
yoy_pred = forecast_yoy(yoy_idata, levels=y)
level_pred = predict_level_model(level_mmm, test_df)
level_full_pred = predict_level_model(level_full_mmm, test_df)
holdout_forecasts = {
"noise floor (true model)": normal_forecast(
mean=mu_true.sel(obs_date=test_dates).to_numpy() * y_reference_test,
sd=z_sigma_true * y_reference_test,
),
"year-over-year": yoy_pred,
"seasonal naive, fitted growth": seasonal_naive(y, z_train, fitted_growth=True),
"seasonal naive, no growth": seasonal_naive(y, z_train, fitted_growth=False),
"level + Fourier + holiday + trend": level_full_pred,
"level + Fourier": level_pred,
}
comparison = pd.DataFrame(
{name: score(pred, y_test) for name, pred in holdout_forecasts.items()}
).T
Here are the scores.
Let’s read the bar charts from the top.
The ratio model sits at the noise floor. Its CRPS of \(19.47\) is within \(2\%\) of the \(19.12\) of the true model, and the same holds for the RMSE and the MAPE. On this panel it cannot be beaten, so its headline number is the noise level of the data, not a measure of the model.
The reference level does most of the work. With no media at all, last year’s level times a fitted growth rate reaches a CRPS of \(31.14\), and \(36.09\) without the growth. Media and the control buy the remaining distance down to \(19.47\).
The level models pay for their specification. Both lose against the naive forecast with fitted growth, the plain one by a wide margin (\(75.02\) against \(31.14\)). A third of its squared error sits in the holiday window (\(0.33\)), and we look at the rest in the holdout forecasts below. Adding the trend and the holiday indicators brings the CRPS down to \(49.14\).
The rest is the cost of a smooth seasonal basis. The remaining gap between the full level model and the naive forecast is what a level-space seasonal basis pays against a seasonal random walk. The level model estimates one shape from four noisy years, while the ratio model conditions on last year’s realization of it.
Holdout Forecasts#
Let’s look at the holdout forecasts of the ratio model and of the full level model, unit by unit, with the mean of the plain level model for reference.
The ratio model tracks the observed KPI closely, including the holiday spike and the summer peak, and its narrow band follows the seasonal level. The full level model has a wider band and a smoother mean. The plain level model misses the holiday spike in every unit, and its mean is biased in the direction of each unit’s growth: too low for the growing north and west, and too high for the shrinking east. A constant intercept cannot extrapolate a compounding trend.
Coverage#
We read coverage together with the width of the intervals. The intervals of the two level models are \(2.2\) and \(3.4\) times wider than those of the ratio model (\(251.91\) and \(394.93\) against \(117.09\)), and they still fall short of nominal coverage (\(86\%\) and \(90\%\)). A single residual scale per unit has to cover the seasonal peak everywhere, and the misfit of the mean ends up in y_sigma, which widens the intervals without centering them. The intervals of the ratio model scale with \(y_{u,r(t)}\), so they follow the cycle.
The coverage of the ratio model also lands below nominal, at \(89\%\). Under this data-generating process the proportional noise model is exactly right, so there is no structural reason for that. Note that even the true model covers only \(91\%\) of the holdout weeks.
Each z_sigma comes from \(200\) training weeks, and the realized noise in the \(52\) holdout weeks need not match it. Let’s compare the noise scales and the coverage unit by unit.
yoy_hdi = az.hdi(yoy_pred, prob=0.94)
# Holdout weeks inside the 94% HDI of the ratio model: (date, unit)
covered = (y_test >= yoy_hdi.sel(ci_bound="lower")) & (
y_test <= yoy_hdi.sel(ci_bound="upper")
)
# Realized ratio noise of every modeled week: (obs_date, unit)
noise_realized = (z_observed - mu_true.sel(obs_date=z_observed["obs_date"])).to_numpy()
# Noise scales in percent of the reference level: (unit,)
noise_by_unit = 100 * pd.DataFrame(
{
r"true $\sigma_u$": z_sigma_true,
"posterior mean of z_sigma": yoy_idata.posterior["z_sigma"]
.mean(("chain", "draw"))
.to_numpy(),
"realized noise sd, training weeks": noise_realized[:-n_test].std(
axis=0, ddof=1
),
"realized noise sd, holdout weeks": noise_realized[-n_test:].std(
axis=0, ddof=1
),
},
index=pd.Index(unit_names, name="unit"),
)
fig, axes = plt.subplots(
ncols=2, figsize=(15, 5.5), width_ratios=[2, 1], layout="constrained"
)
noise_by_unit.plot.bar(ax=axes[0], rot=0, legend=False)
for container in axes[0].containers:
axes[0].bar_label(container, fmt="%.2f", padding=3, rotation=90, fontsize=12)
axes[0].set(xlabel="unit", ylabel="noise sd (% of the reference level)")
axes[0].margins(y=0.25)
bars = axes[1].bar(unit_names, covered.mean("date").to_numpy(), color="C0")
axes[1].bar_label(bars, fmt="%.2f", label_type="center", color="white", fontsize=12)
axes[1].axhline(0.94, color="black", linestyle="--")
axes[1].set(xlabel="unit", ylabel=r"holdout $94\%$ HDI coverage", ylim=(0, 1.1))
handles, labels = axes[0].get_legend_handles_labels()
fig.legend(
[*handles, Line2D([], [], color="black", linestyle="--")],
[*labels, r"nominal $94\%$"],
loc="outside lower center",
ncols=3,
)
fig.suptitle(
"Noise Scales and Holdout Coverage of the Ratio Model",
fontsize=16,
fontweight="bold",
);
The posterior z_sigma tracks the realized training noise rather than the true value, as it should. The unit with the lowest coverage, west at \(83\%\), is the one whose holdout weeks happened to be noisier than its training weeks (\(3.47\%\) against \(2.74\%\) of the reference level). For that unit the intervals are about a fifth too narrow, and a \(94\%\) interval that is a fifth too narrow covers about \(86\%\) of the time, close to what we see. This is one unlucky draw of \(52\) weeks, not a defect of the noise model.
A Fixed Seasonal Shape#
We generated everything above with the ratio model itself. That is the right way to check parameter recovery, but the wrong way to decide whether to use the transform, because the alternative hypothesis about the data was never on the table. Here it is: the same spend, the same control, the same media response and growth rates, but a fixed seasonal shape that repeats every year, a moving holiday with the same three-week profile, and independent proportional noise at every unit-period:
Here \(s_u\) is the size of unit \(u\) and \(S(t)\) is the fixed annual shape with the holiday. On this panel the ratio model is approximately right in the mean, since \(\exp(\Delta) \approx 1 + \Delta\) for small contrasts. But it is wrong about the noise in two ways:
Twice the variance. The ratio of two independent noisy levels has about twice the noise variance of either level.
Negative correlation. \(\varepsilon_{u,r(t)}\) appears in the denominator of \(z_{u,t}\) and in the numerator of \(z_{u,r(t)}\), so the ratios of consecutive cycles are negatively correlated.
Both effects are measurable. The pushed-aside shoulder weeks now also pay a real price, because a week is divided by the seasonal level of a different week.
We first compute the true media effect on the full series with the same components and the true parameters.
with pm.Model(coords=coords_data) as media_model:
pmd.Deterministic(
"f_full",
saturation.apply(
adstock.apply(pmd.Data("channel_data", channel_scaled), core_dim="date")
),
dims=("date", "unit", "channel"),
)
# True saturated spend on the full series: (date, unit, channel)
f_true = pm.draw(do(media_model, media_truth)["f_full"])
print(f"f_true: {f_true.shape} (date, unit, channel)")
f_true: (312, 4, 3) (date, unit, channel)
Next, we build the fixed shape with the holiday at its position in every cycle, then the mean and the noisy level of the second panel.
# Fixed annual shape, repeated every cycle, with the holiday profile: (date,)
fixed_shape = np.tile(seasonal_shape, n_years)
for t in holiday_index:
fixed_shape[t + holiday_offsets] *= holiday_profile
# Compounding growth of every unit: (date, unit)
growth = intercept_true[None, :] ** (np.arange(n_dates)[:, None] / period)
# Media and control lift on the log scale: (date, unit)
media_and_control = f_true.sum(axis=-1) + (
gamma_control_true * control_data.transpose("date", "unit", "control").to_numpy()
).sum(axis=-1)
# Mean and noisy level of the second panel: (date, unit)
y_fixed_mean = (
unit_size[None, :] * growth * fixed_shape[:, None] * np.exp(media_and_control)
)
y_fixed = y_fixed_mean * (
1 + rng.normal(0.0, z_sigma_true[None, :], size=(n_dates, n_units))
)
target_fixed = xr.DataArray(
y_fixed, dims=("date", "unit"), coords={"date": date_index, "unit": unit_names}
)
# Observed ratio of the second panel: (obs_date, unit)
z_fixed = xr.DataArray(
y_fixed[obs_idx_model] / y_fixed[reference_map[obs_idx_model]],
dims=("obs_date", "unit"),
coords={"obs_date": date_index[obs_idx_model], "unit": unit_names},
)
z_fixed_train = z_fixed.isel(obs_date=slice(None, -n_test))
We fit the ratio model and the full level model on the training weeks of the second panel.
yoy_fixed_model = build_yoy_model(obs_idx_train, z_fixed_train)
with yoy_fixed_model:
yoy_fixed_idata = pm.sample(
chains=4,
tune=1_000,
draws=1_000,
target_accept=0.9,
nuts_sampler="nutpie",
random_seed=rng,
)
panel_fixed_df = make_panel_df(target_fixed)
train_fixed_df = panel_fixed_df[panel_fixed_df["date"].isin(train_dates)].reset_index(
drop=True
)
test_fixed_df = panel_fixed_df[panel_fixed_df["date"].isin(test_dates)].reset_index(
drop=True
)
level_fixed_mmm, level_fixed_idata = fit_level_model(
train_fixed_df, controls=["discount", *level_extra_controls]
)
We run the same sampler checks on both fits.
fixed_fits = {
"year-over-year": (yoy_fixed_idata, yoy_var_names),
"level + Fourier + holiday + trend": (
level_fixed_idata,
[rv.name for rv in level_fixed_mmm.model.free_RVs],
),
}
for name, (idata, var_names) in fixed_fits.items():
print(f"--- {name} ---")
sampler_ok = not az.diagnose(idata, var_names=var_names)
print(f"sampler checks passed: {sampler_ok}\n")
--- year-over-year ---
Divergences
No divergent transitions found.
E-BFMI
E-BFMI satisfactory for all chains.
ESS
Effective sample size satisfactory for all parameters.
R-hat
R-hat values satisfactory for all parameters.
Processing complete, no problems detected.
sampler checks passed: True
--- level + Fourier + holiday + trend ---
Divergences
No divergent transitions found.
E-BFMI
E-BFMI satisfactory for all chains.
ESS
Effective sample size satisfactory for all parameters.
R-hat
R-hat values satisfactory for all parameters.
Processing complete, no problems detected.
sampler checks passed: True
Both fits pass all checks. We score both models against the noise floor and the seasonal naive forecast, as before.
y_fixed_test = target_fixed.sel(date=test_dates)
# True mean of every holdout week: (date, unit)
y_fixed_mean_test = y_fixed_mean[obs_idx_test]
fixed_forecasts = {
"noise floor (true model)": normal_forecast(
mean=y_fixed_mean_test, sd=z_sigma_true * y_fixed_mean_test
),
"year-over-year": forecast_yoy(yoy_fixed_idata, levels=y_fixed),
"seasonal naive, fitted growth": seasonal_naive(
y_fixed, z_fixed_train, fitted_growth=True
),
"level + Fourier + holiday + trend": predict_level_model(
level_fixed_mmm, test_fixed_df
),
}
comparison_fixed = pd.DataFrame(
{name: score(pred, y_fixed_test) for name, pred in fixed_forecasts.items()}
).T
Here are the scores on the second panel.
The picture changes. All models now sit well above the noise floor of \(22.74\). The full level model has the lowest CRPS (\(32.56\) against \(33.08\)) and RMSE (\(58.59\) against \(63.88\)). The ratio model has the lower MAPE (\(3.68\%\) against \(3.98\%\)) and the better calibrated, but wider, intervals (\(96\%\) coverage against \(89\%\)). In other words, the comparison is a draw that the level model wins by a small margin on our main metric.
The squared error of the ratio model concentrates at the peak: its holiday share is \(0.18\), more than twice the \(0.08\) of the noise floor. There are two reasons for this, and both are properties of the transform rather than of this fit:
The reference is noisy. The noise variance of the ratio is roughly doubled, and in KPI units this doubling is largest exactly where the level is largest.
Anchoring realigns the event, not the seasonal level underneath it. On a fixed shape \(S(\cdot)\), the anchored ratio at the holiday is \(S(h_c) / S(h_{c-1})\), which is not one when the week of the holiday moves. The level model with explicit holiday indicators separates the two and has no such term.
The remaining handicaps of the level model are specification choices: a linear trend for compounding growth, a constant noise scale for proportional noise, and one saturation_beta per channel for a media lift that scales with the level. In the data, the lift moves with the season and with the size of the unit, while the level model applies the same response curve to every unit and every week. MMM(link="log") with a multiplicative trend would remove all three. The handicaps of the transform are structural.
What the ratio model pays is visible in its noise parameter and in its residuals. We pair every training residual with the residual of its own reference week, where both exist.
# Training residuals of the ratio model on the second panel: (obs_date, unit)
residual_fixed = (
z_fixed_train - yoy_fixed_idata.posterior["mu"].mean(("chain", "draw"))
).to_numpy()
# Positions of a residual and of the residual of its reference week: (pair,)
position_of = {t: i for i, t in enumerate(obs_idx_train)}
pairs = [
(i, position_of[reference_map[t]])
for i, t in enumerate(obs_idx_train)
if reference_map[t] in position_of
]
current_idx, reference_idx = map(np.array, zip(*pairs, strict=True))
# Noise scales in percent of the reference level: (unit,)
noise_structure = 100 * pd.DataFrame(
{
r"true level noise $\sigma_u$": z_sigma_true,
r"$\sqrt{2}\,\sigma_u$": np.sqrt(2) * z_sigma_true,
"posterior mean of z_sigma": yoy_fixed_idata.posterior["z_sigma"]
.mean(("chain", "draw"))
.to_numpy(),
},
index=pd.Index(unit_names, name="unit"),
)
residual_correlation = pd.Series(
[
np.corrcoef(residual_fixed[current_idx, u], residual_fixed[reference_idx, u])[
0, 1
]
for u in range(n_units)
],
index=unit_names,
)
fig, axes = plt.subplots(
ncols=2, figsize=(15, 5.5), width_ratios=[2, 1], layout="constrained"
)
noise_structure.plot.bar(ax=axes[0], rot=0, legend=False, color=["C0", "C1", "C2"])
for container in axes[0].containers:
axes[0].bar_label(container, fmt="%.2f", padding=3, rotation=90, fontsize=12)
axes[0].set(xlabel="unit", ylabel="noise sd (% of the reference level)")
axes[0].margins(y=0.25)
bars = axes[1].bar(unit_names, residual_correlation, color="C4")
axes[1].bar_label(bars, fmt="%.2f", padding=3, fontsize=12)
axes[1].axhline(-0.5, color="black", linestyle="--")
axes[1].set(
xlabel="unit",
ylabel=r"correlation of residuals at $t$ and $r(t)$",
ylim=(-0.7, 0.05),
)
handles, labels = axes[0].get_legend_handles_labels()
fig.legend(
[*handles, Line2D([], [], color="black", linestyle="--")],
[*labels, r"$-1/2$ for independent level noise"],
loc="outside lower center",
ncols=4,
)
fig.suptitle(
"Noise Structure of the Ratio Model on the Fixed-Shape Panel",
fontsize=16,
fontweight="bold",
);
The posterior z_sigma sits above \(\sqrt{2}\) times the level noise in every unit (\(4.49\%\) to \(6.10\%\) of the reference level, against \(3.96\%\) to \(4.95\%\)). The excess is the misalignment cost described above. The residual at \(t\) is negatively correlated with the residual at \(r(t)\), between \(-0.34\) and \(-0.40\). This is the direction and close to the size of the \(-1/2\) that independent level noise implies, diluted by the misalignment variance.
Neither effect is fatal, but both are mis-specifications of the i.i.d. Normal likelihood that we use here. The posterior intervals of the ratio-space coefficients are somewhat too narrow, and a reference year that was unusually good makes the current year look unusually bad. A likelihood that models \(\varepsilon_{u,t}\) and \(\varepsilon_{u,r(t)}\) jointly, or the log-difference variant of the transform with a moving-average error, would be the principled repair. This is the main reason to treat the home-turf margins above as an upper bound.
Conclusion#
We built the year-over-year target transform as a custom PyMC model on top of PyMC-Marketing components, with an anchored reference map for the moving holiday. Then we compared it out of sample with level-space MMMs on two synthetic panels. On the panel generated from its own assumption, the ratio model reaches the noise floor. On a panel with a fixed seasonal shape and independent noise, a level model with the right calendar terms matches it.
Key Findings#
Seasonality comes at no cost, and so does most of the forecast. On the home-turf panel, the CRPS of the ratio model (\(19.47\)) is within \(2\%\) of the true model (\(19.12\)). But a seasonal naive forecast without media already reaches \(31.14\), so the reference level accounts for most of the gain over the level models (\(49.14\) and \(75.02\)), and media plus the control for the rest. Decompose any headline gain against such a baseline before you attribute it to the MMM.
The failures of the level models are specification failures. A constant intercept cannot extrapolate a compounding trend, and a smooth annual basis cannot represent a moving spike. A trend control and holiday indicators fix most of this inside
MMM(CRPS from \(75.02\) to \(49.14\)). On the fixed-shape panel, the same level model beats the ratio model by a small margin (\(32.56\) against \(33.08\)).The reference map is a load-bearing component, not a preprocessing detail. With a naive lag, the ratio has a spike and a crater at every moving holiday. Dragging the event window aligns the peaks, but it duplicates and orphans reference weeks and still divides normal weeks by holiday weeks. Rotating the prior-cycle segment keeps \(r\) a bijection, at the cost of moving the \(|k|\) pushed-aside weeks by \(2\,\text{pad} + 1\).
The reference is an observation, and its noise comes along. With independent level noise, the ratio carries about twice the variance and a residual correlation across cycles of about \(-0.4\). The i.i.d. likelihood ignores both.
Media enters only as a contrast. The always-on
sponsorshipchannel drops out of the likelihood exactly, and its posterior is its prior (standard deviation ratios between \(0.98\) and \(1.05\)). In a level model the same channel is collinear with the per-unit intercept, so we leave it out of the level models. Only the ratio model can keep it and say cleanly that the data carry no information about it.There are two attribution questions. The contrast basis is identified and answers “what did the change in plan contribute?”. The level basis answers “what did the channel contribute in total?” through an exact regrouping around \(a^\ast = 0\). Its leverage over the measured contrast (about \(2.6\) for
tv, close to one forsearchand infinite forsponsorship) says how much it rests on the shape of the curve.
Model Limitations#
We generated the home-turf panel from the ratio model itself, so its margins show what a level model pays when the assumption of the transform holds. They are not a general ranking.
The i.i.d. Normal likelihood ignores the noise of the reference level and the negative correlation across cycles.
The saturation curves in KPI units rest on an average reference level, because the response of the ratio model scales with last year’s level.
The
saturation_betaprior is on the same scale as the true values, which flatters the recovery check.The reference map assumes fixed-length cycles and consecutive occurrences about one cycle apart, with windows that do not overlap. It refuses other calendars instead of approximating them.
Each panel has a single \(52\)-week holdout, and the scores move with the realized noise of those weeks.
Recommendations#
Use the year-over-year transform when the seasonal shape drifts from year to year and the media plan changes from cycle to cycle. If the seasonal shape is fixed and the calendar is known, a level model with event terms is at least as accurate.
Prefer a level-space model with an explicit seasonal basis when the KPI can touch zero, when the media plan is stable and the question is the total contribution, or when the reference year contains a shock (a stockout, a lockdown) that the current year should not be measured against.
Report the ratio of posterior to prior standard deviation next to the media coefficients, and the leverage next to any level-basis contribution.
Build the reference map with the rotation, check that
duplicatedandorphanedare empty, and keep the shoulder as narrow as the real footprint of the event.Convert ratio-space contributions to KPI units per unit-period before you aggregate them.
Next Steps#
Where would the next iteration go?
A first-class reference map.
build_reference_mapis useful on its own, independently of any model. Every other piece of this notebook consumes it, which argues for a reference map component in PyMC-Marketing rather than a divisor hidden inside a scaler.A better likelihood. Model \(\varepsilon_{u,t}\) and \(\varepsilon_{u,r(t)}\) jointly, or use the log-difference variant of the transform with a moving-average error.
A wider benchmark. Compare the ratio, the log-difference and
MMM(link="log")on the same panels, with time-slice cross-validation as in Time-Slice-Cross-Validation and Parameter Stability instead of a single holdout.Event-relative references. Support overlapping events, and occurrences that are not about one cycle apart, with a reference defined relative to the event rather than to a fixed cycle.
%load_ext watermark
%watermark -n -u -v -iv -w -p pymc_marketing,pytensor
Last updated: Fri, 02 Oct 2026
Python implementation: CPython
Python version : 3.14.2
IPython version : 9.15.0
pymc_marketing: 1.2.0
pytensor : 3.3.2
arviz : 1.2.0
arviz_plots : 1.2.0
matplotlib : 3.11.1
numpy : 2.4.6
pandas : 3.0.5
pymc : 6.3.2
pymc_extras : 0.15.1
pymc_marketing: 1.2.0
pytensor : 3.3.2
xarray : 2026.4.0
Watermark: 2.6.0