Run a lift test
You measure one specific change: a budget you raised on a date, a campaign you switched off. Lift compares the days that followed against what would have happened if you had done nothing, and reports the effect with a 95% confidence interval.
What lift measures#
Every time you raise a budget or switch a campaign off, the next question is the same: did it actually make a difference? The raw number for the following fortnight cannot answer it, because your sales have a trend, a season and daily noise anyway.
Lift does this: it fits a model on the data from before the action date, uses it to predict what the following days would have looked like had you done nothing — the counterfactual — and sums the difference between reality and that prediction. That sum is the incremental lift, and it always arrives with a 95% confidence interval.
The unit of analysis is an action, not a channel. For the long-run portfolio picture use the marketing mix model; that tool needs about two years of data, while lift works with about two months.
Lift (incrementality) is not in the Free plan. If the website is on Free you get an upgrade screen instead of the page, with an Upgrade plan button. The docs are not wrong — that website's plan does not include the feature, and plans are set per website.
Define the action#
Tests live at /lift. Creating one means answering two sets of questions: what you did, and what you want the effect measured on and over how long.

The list shows each test's name, action type, channel, date and window length, with either its current status (Scheduled, Queued, Running, Failed) or the finished verdict at the trailing edge. Tests still in flight sort to the top.
The fields that change accuracy#
| Field | Default | Why it matters |
|---|---|---|
| Measurement period | 14 days | only 7, 14 or 28 are accepted. 14 covers two full weekly cycles and is the balanced choice for most volumes |
| Pre-period (days of history) | 56 days | the raw material for the counterfactual. The field's hint recommends at least 42; below 14 nothing can be fitted |
| Use organic demand as a control | on | feeds organic and direct session volume in as a demand proxy, so a market-wide swing is not charged to your action |
| Engine | Ensemble (fast) | the default is fast; Bayesian (rigorous) is slower and stricter about expressing uncertainty |
If you are unsure whether to believe the results, create a test on a date where nothing happened. The verdict should come back Neutral. That is exactly what the product's own gate checks, and it is the simplest way to decide how much to trust the rest of your tests.
How the counterfactual is built#
The default engine (Ensemble (fast)) fits a couple of candidate models on the pre-period — a structural time series with a random-walk level and a weekly season, and a "same day last week" seasonal-naive baseline — and keeps whichever scores the lowest error on a held-out tail of the pre-period. It then refits on the full pre-period and forecasts the days after the action.
The width of the band comes from a block bootstrap of the residuals: the model is refitted dozens of times on synthetic series so that both parameter uncertainty and future noise land inside the interval. Seven-day blocks preserve the weekly autocorrelation.
The Bayesian (rigorous) engine fits the same shape — random-walk level, weekly seasonality, regression on the control — as a Bayesian model and simulates the counterfactual from the posterior. It is slower, and when daily volume is low its interval widens honestly.
Both engines take any calendar seasonality events — Nowruz, Ramadan, Muharram and the rest — that fall inside the test window as exogenous controls, so a holiday surge is not charged to your action. The seasonality calendar explains where those come from.
The placebo-calibrated engine (v2)#
Every new or re-run test is measured with this engine, and there is no switch to turn it off. Tests that already finished keep the engine they ran with.
The idea is simple: before trusting a result, check how often the method claims an effect when there is none. The engine repeats the same test on dates every week through your past year, where you made no change, and records how far its prediction missed each time. Those misses set the interval and the p-value of your real test. The Calibration line in the result header says where those misses came from:
| Calibration | Meaning |
|---|---|
| Own history (N placebo tests) | the interval was checked against past dates on this website when nothing was changed |
| Pooled network | this website's history is too short, so the interval was checked against placebo errors pooled from similar websites (anonymous aggregates) |
| Model only | not enough history here or in the network; the interval is the model's own estimate and the grade is C at best |
- The counterfactual is the average of five forecasting models instead of a pick of one, which is steadier when one model misbehaves.
- Weeks with a real spend change, an outage or a holiday your test window does not share are left out of the placebo tests.
- Significant requires both a p-value at or below 5% and an interval that excludes zero.
- When the window holds a calendar event the pre-period never saw (a promotion run only on the test days, say), its effect cannot be separated from the ad effect. The Test validity card names it and the grade is capped at C.
- Days with missing or incomplete data (a tracker outage, or days before the tracking code was installed) are counted in Test validity as Days with missing or incomplete data. When one falls inside the test window, Outage in the window reads Yes and the grade drops to C.
- Days the KPI dropped to zero while the tracking code kept recording visits (the business was closed, or checkout broke) count as missing data too, listed as Days the KPI dropped to zero while the tracking code kept firing (possible closure). They are never read as an effect of your change; when they cover most of the test window the test is not scored.
- One exception: after a change that cuts spend (a stop, a removed channel or a budget decrease), purchases falling to almost zero may be the effect of the cut itself. Those days stay in the result, the grade drops to C, and the page warns: "Purchases fell to almost zero after this change while other traffic continued. If the business was closed, this is not an ad effect."
- Likewise, when all of your site's traffic drops on the day your ad spend is cut (an ad-driven shop that switches every campaign off), the drop is read as the effect of the cut, not as a broken tracking code. Those days stay in the result, the grade drops to C, and the page warns that a tracking outage at the same moment could not be told apart.
Counterfactual controls#
Under v2 the Counterfactual controls field replaces the organic switch:
| Option | When to use it |
|---|---|
| External factors only (recommended) | the default. Calendar events are the only controls; organic traffic is measured as an outcome |
| On-site organic as a control (may understate the effect) | the old behaviour, kept for comparison. Ads also drive organic visits, so the result is a direct effect only and is labelled so |
| No controls | a plain before/after forecast |
The On-site organic (halo) card reports the action's effect on organic and direct traffic. A positive value is a halo (the ads also lifted organic); a negative value means the paid channel took over demand that would have come anyway.
Check sensitivity before you test#
In the new-test drawer, Test sensitivity answers "can this test see the effect I expect?" before you spend anything. Enter the Planned spend change (%) and select Check sensitivity. You get:
- the smallest lift this website can detect in the chosen window (MDE);
- the chance of detecting your planned change, when a Marketing Mix model run has measured the channel;
- the shortest of 7, 14 or 28 days that reaches 80% power, and the change size needed for it.
Quality grade and false discoveries#
Every v2 result carries a grade from Quality A to Quality D:
| Grade | Meaning |
|---|---|
| A | calibrated, little placebo bias, clean window, a material spend change |
| B | calibrated, with minor issues |
| C | model-based, or the spend moved before the test; use it as directional |
| D | unreliable; never sent to the Marketing Mix model |
The Test validity card lists what the grade was built from. In the test list, a ⚠ next to a significant result means it does not survive a false-discovery check across this website's tests of the last 180 days. When you run many tests, a few will look significant by chance.
Other spend changes around the test#
When the website records spend changes (Incrementality → Activities), the result lists Other spend changes around this test: material changes (at least 10% of total daily spend) made during the test window or in the 14 days before it.
- During the window: the lift includes their effect, so the test cannot credit it all to your action. Such a test is never graded A.
- Just before the window: the baseline moved, so the test counts as contaminated.
- Measured by this test: an automatic test measures several changes together. Each row shows its share of the joint lift and how much of that share the data resolved (Resolved by data). A low value means the split mostly follows prior estimates.
Activities you typed in yourself (a promotion, a price change, an influencer post …) are listed with their kind, whatever their size. The change you are testing (same channel and campaign within 7 days) and dismissed activities are never listed.
Why a test sits on Scheduled#
If the measurement window has not finished, the test does not run immediately — it waits on Scheduled. It starts automatically once the last day of the window has passed, plus 2 days of settling for conversions that arrive late (server-side orders, delayed attribution).
That is deliberate. Scoring a window that has not finished compares empty days against a full counterfactual, and the result is a fake negative lift. You do not need to do anything; the test queues itself.
Read the result#
The verdict and the interval#
The top of the result drawer summarises the action, with the verdict badge at its trailing edge:
| Verdict | What it means |
|---|---|
| Positive | the whole 95% interval on the lift sits above zero |
| Negative | the whole interval sits below zero |
| Mixed | the interval excludes zero, but strongly positive and strongly negative days are both common — the effect is not uniform |
| Decline · not significant / Increase · not significant | the interval contains zero, but the estimate moved by 1% or more: the badge names its direction in a muted colour, and the sentence below gives the estimate, the p, the 95% range and "read it as a likely decline (increase), not a proven one" |
| No clear change | the interval contains zero and the estimate moved by less than 1% |
| Neutral | a test that was not scored (refused, for example too many invalid days); the reason is shown instead of a result |
Below it, a plain-language sentence says the same thing, next to Significant (p=…) or Not significant (p=…). That p is a two-sided p-value tested at the 5% level (α = 0.05), the same test as the 95% interval. The "Significant" label follows the verdict: it appears only on a Positive or Negative result, whose 95% interval excludes zero. A result whose interval includes zero always reads "Not significant", even when its p looks small, because an interval that still includes zero cannot rule out "no effect" — but its estimate is still stated: a −25.6% result reads "Decline · not significant", never "Neutral". When the test is underpowered the sentence adds the smallest change it can reliably detect. The fit MAPE beside it reports the counterfactual's quality on the pre-period: the lower it is, the closer the prediction tracked your site's real behaviour.
A neutral verdict is a result, not a failure#
This is the most important sentence on the page. Neutral means the change you observed fits within that series' natural variation. The model deliberately declines to claim an effect it cannot defend.
Neutral does not always mean the same thing, and the Detectability card separates the cases:
| Label | Minimum detectable effect | How to read it |
|---|---|---|
| High confidence | under 5% | the test could see a small effect; neutral probably means there was none |
| Medium confidence | under 10% | neutral is meaningful, but small effects would have slipped through |
| Low — wide bands | under 20% | only large effects were visible |
| Underpowered | over 20% | the test never had the power; neutral says nothing about the action |
So an Underpowered test with a Neutral verdict means "we don't know", not "it did nothing". Fix it with one of: a longer measurement period, a longer pre-period, or a metric with more daily volume (add_to_cart rather than purchase when orders are sparse).
The effect cards#
- Incremental lift — the summed difference between reality and the counterfactual across the window, with the relative percentage and the 95% CI.
- Daily avg (all) and Daily avg (organic) — the cannibalization view. If the total rose while organic fell by about as much, the action mostly moved demand around.
- Incremental revenue — modelled directly for a revenue metric. For a conversion metric it reads Estimated revenue (purchases × average order value): the lift multiplied by the average order value of the pre-test window.
- Ad spend (window), Incremental ROAS and CPA — all based on the spend of the channel and campaign you named in the definition. If your role may not see cost, these three cards do not appear.
The chart and the daily table#

In Incremental effect over time, the solid line is Observed, the dashed line is the Counterfactual, the shaded area is the 95% prediction band, and the thin line is Organic. The vertical marker sits on the action date. Where the solid line rises above the shaded band, the lift is real; inside the band is what would have happened anyway.
The Only show after the action checkbox drops the pre-period so you can concentrate on the measurement window. Daily incremental lift (post-period) gives the same thing as numbers day by day, and Advanced data puts the raw pre and post totals side by side for all traffic and for organic.
What lift cannot tell you#
The counterfactual extends the pre-action trend. Anything else that happens at exactly the same time as your action is charged to your action: a price change, another channel's campaign, a press mention, an outage.
Three guards are in place — the organic demand control, the calendar seasonality events, and the neutral-by-default verdict — but none of them replaces your own discipline:
- Change one thing at a time and record its date precisely.
- For simultaneous actions on several channels, create separate tests with separate channels.
- The control assumes the action does not affect organic itself. If your campaign visibly lifts brand search, that assumption weakens — turn the control off and compare both readings.
A finished test can be Edited (saving re-runs it), Duplicated so the definition is ready for the next action, Re-run when data arrived late, or exported from the Export menu as a PDF report, CSV (data) or PNG (chart).
A significant lift test on a named channel is the best possible calibration input for the MMM. Enter its ROI and standard error in the Calibration experiments (anchor the model to lift tests) card and the next model run ties your causal measurement to the observed history.
Frequently asked questions#
Does a neutral verdict mean the test failed?
No. Neutral means the change you saw fits inside the counterfactual's confidence band, so it cannot be credited to the action. That is a real result, and it is what stops you spending against random noise. The detectability card tells you how large an effect the test could have seen at all.
Why is my test stuck on Scheduled?
Because the measurement window has not finished. A test runs automatically once every day of its window has elapsed, plus 2 days of settling for conversions that arrive late. Scoring a half-finished window produces a fake negative lift.
How long should the pre-period be?
The form starts at 56 days and the field's own hint recommends at least 42. Below 14 days nothing can be fitted and the test closes with a not-enough-data message. A longer history means a more accurate counterfactual and a narrower interval.
What does the placebo-calibrated engine change?
The interval and the p-value stop coming from the model's own opinion of its error. They come from dozens of fake tests on dates where nothing happened in your own past year, so "significant" means what it says on your data. It needs about a year of history; with less, the result is labelled model-based.
How is this different from the marketing mix model?
Lift measures one action over a short window and works with roughly two months of data; the marketing mix model measures the whole channel portfolio over roughly two years. They are complementary, and a lift result can be fed to the MMM as a calibration prior.
Thanks — your feedback helps us improve the docs.