- 3simulated campaigns3 weeks each, spring, summer and autumn 2011
- £0true effect on real purchasingthe campaigns never happened
- −£54.7first estimate, autumnper customer-week, interval [−118, −13]
- 5×narrower after one stated changeautumn interval [−23.0, −1.4]
The simulation
The dataset has no promotions, so campaigns were simulated over real customers to test the measurement itself. Because they never happened, their true effect on real purchasing is known: exactly zero. A method that cannot return zero here should not be pointed at a real campaign.
For each simulated campaign: eligible customers are those with a purchase in the 12 weeks before launch; they are split into deciles of pre-period spend and half of each decile is assigned to treatment with seeded hash-based draws; treated customers notionally receive a 5 pound basket voucher; a treated customer who buys in the window redeems on the first basket with probability 0.3; then the lift is estimated.
- EligibleRecent buyersAt least one purchase in the 12 weeks before launch. Anonymous lines excluded.
- AssignStratified 50/50Deciles of pre-period spend; half of each decile to treatment, seeded hash draws.
- Expose£5 basket voucherSimulated. Control receives nothing.
- RedeemProbability 0.3On the first in-window basket of a treated buyer, linked to a real transaction.
- MeasureEstimate the liftOne row per campaign and customer, then the estimator.
- mart_
campaign_ customer
- mart_
Stratifying on pre-period spend guarantees balance on the strongest predictor of future spend, which tightens the estimate without biasing it.
The estimator
Difference-in-differences on weekly net sales per customer: each customer's change from the 12-week baseline to the campaign weeks, averaged for treatment, minus the same for control. Differencing removes stable differences between customers; comparing with control removes what changed for everyone, such as season and trend.
For treatment and control separately, each customer's weekly net sales in the campaign weeks minus the 12-week baseline is averaged. The lift is the treatment average change minus the control average change, with a 95 percent bootstrap interval from 1,000 resamples and a Welch interval for comparison.
The zero test
Spend in this data is extremely skewed: a few wholesale buyers spend hundreds of times the median. On the first real-data run the raw estimator failed its own zero test on the autumn campaign, finding a negative effect for a campaign that did nothing. The spring and summer campaigns passed.
For a simulated campaign whose true effect is zero, the raw estimator gave minus 54.7 pounds per customer-week with an interval from minus 118 to minus 13, which excludes zero. After winsorising, the interval is minus 23.0 to minus 1.4: five times narrower but still just excluding zero. The method was not changed again; the miss is reported.
The size of the miss pointed to outliers dominating the means, so the method was changed once, for that stated reason: each customer's weekly outcome is capped at the 1st and 99th percentiles of the pre-period, which treatment cannot affect. The autumn interval became five times narrower but still just excludes zero. The method was not changed again to make it pass; that would be fitting the method to the result.
What gets reported
Every run validates the method before any estimate is shown, and the results are labelled for what they are.
Three checks run on every run. The zero test: the interval should cover zero; about one in twenty honest intervals will not, so a miss is a warning. Known-effect recovery: a simulated 10 percent uplift is added and the interval should cover it. Permutation placebo: labels are shuffled 200 times and the share of intervals excluding zero should be near 5 percent; the check warns above 10 percent. Estimates are labelled ESTIMATED and simulated, and incremental margin is not computed because the data has no cost information.
Before anyone sees a lift number, someone should have seen the method return zero on a campaign that did nothing.