Data & spreadsheets · course 42 of 105

A/B Test Maths: Is the Lift Real?

Read an A/B test result like an analyst: conversion rates, the two kinds of lift, the two-proportion z-test step by step, and how many visitors a test needs before you run it.

4 modules

Try a question

Not on sale yet. When sales open: $4.99 for this course, paid once, or every course for $9.99 a month. Prices

Preview

What it teaches

The numbers behind an A/B test result. Conversion rate, and the two different “lifts” (a difference in percentage points, and a percentage of the control’s rate); the two-proportion z-test step by step (pooled rate, standard error, z, two-sided p-value, significant at 5% or not); then planning: how many visitors each group needs for a baseline, a lift, a significance level and a power, and the confidence interval for the difference. The AI asks 3 questions, places you, shows a short card per module, then gives you one question at a time.

Not a statistics degree, and it can’t tell you whether a real experiment was run properly or whether to ship a change.

How long

44 items. About 25-35 rounds for a confident analyst up to about 55-85 for a beginner, plus a 12-item final: 2-3 sessions (estimate, not timed with a real student). The tutor gives a one-line save code at the end of each session.

From the answer key · M1, row C7

A: 300 of 10,000 visitors converted. B: 345 of 10,000 converted. What is B’s relative lift over A, in %?

Think first, then show the answer

Answer15%

Working300 ÷ 10,000 = 3%; 345 ÷ 10,000 = 3.45%; (3.45% − 3%) ÷ 3% = 0.15 = 15%.

The wrong answer it catches13.04% → divided by B’s rate; relative lift is measured against the control, A.

Conversion rate = conversions ÷ visitors, written as a percentage. “the percentage of visitors who complete a desired action—such as making a purchase, signing up for a newsletter, or submitting a form”. Wikipedia’s A/B page: “has a 5% response rate (50 of the 1,000 people emailed used the code to buy a product)”.
Relative lift = (variant − control) ÷ control, shown as a percentage of the control’s rate. A glossary (as Relative Uplift) defines it as “The percentage increase compared to control”. GrowthBook’s docs write it as 100% × (μt − μc) ÷ μc, where μt and μc “are the averages in the treatment and baseline variations respectively”.

The modules

4 modules, drilled one item at a time.

M0
The words
The words: control and variant, conversion rate, lift, percentage points, null hypothesis, p-value, significance level, peeking, power
M1
Rates and lift
Conversion rate, absolute lift in percentage points, relative lift, and working back from a lift to a rate
M2
The two-proportion z-test
The two-proportion z-test: pooled rate, standard error, z, two-sided p-value, and is the lead significant at 5%
M3
Sample size and intervals
How many visitors each group needs, and the 95% interval and margin of error for the difference

Before it teaches, it asks

1
What do you work on most (product, marketing, growth, something else), and have you read the results of an A/B test before?
2
A converts 4% and B converts 5%. What is B’s relative lift over A, in %?
3
A: 150 of 2,000 visitors converted. B: 130 of 1,000 converted. What is the pooled conversion rate, in %, to 2 decimals?

What’s exact, and what isn’t

Show

Straight from the course’s own notes.

The words (M0): Every card line is built from quotes (every Answer is written from them), checked word for word on saved copies of the pages (all read 2026-10-13): the NIST/SEMATECH e-Handbook of Statistical Methods (sections 7.1.3 and 7.1.3.1), six Wikipedia pages (A/B testing, Treatment and control groups, Conversion marketing, Percentage point, Two-proportion Z-test, Statistical power), GrowthBook’s documentation, a vendor’s A/B glossary (Lyftio) and a 2016 sample-size paper in PubMed Central (Kim). Where a source uses its own word (“uplift”, “defects”), the card says so.
The z-test (M2): The pooled two-proportion z-test of NIST section 7.3.3. Every key is computed by script and rounded half up as its question says. p-values come from math.erfc, which a script checks against Python’s statistics.NormalDist and a numerical integral. The two-sided 5% cutoff is 1.96.
Sample size and intervals (M3): One sample-size formula (Kim 2016, two equal groups) with the z-values printed on the card (1.96, 0.84, 1.645, 1.282, from Kim and NIST), and Wikipedia’s interval for the difference (each group’s own rate). Each rests on one page, so a script also checks them against exact binomial arithmetic that uses no page: at the sample sizes the course gives as answers, the z-test has 79.9% to 90.0% power (the targets are 80% and 90%), and at 300 visitors per group its false-positive rate is 5.0% and the 95% interval covers the true difference 94.6% to 94.9% of the time.
Computed by script: Every M1, M2 and M3 key (M1 in exact fractions). Every listed wrong answer is recomputed from the slip it names and must differ from the key. M0's Answers are written from quoted lines, not computed.
Honest limits: Large-sample, two-group, converted-or-not results only. Sample-size formulas come in more than one version (Kim’s own page shows its formula giving 291 and the software G*Power 294 for one example), so a calculator that uses another one can give a different number from this course’s key. The keys also use the card’s rounded z-values (1.96, not 1.95996); a spreadsheet’s exact z-values land within 0.2% (at most 8 visitors) of them, which the tutor accepts once without counting a miss. The course drills numbers you are given; it can’t judge a real experiment and it doesn’t teach any analytics tool.
Not in it yet: One-sided tests, more than two variants and corrections for many comparisons, averages and revenue (t-tests), Bayesian results, sequential testing and other ways around peeking, small-sample exact tests, continuity corrections, minimum detectable effect, and checks on how a real test was set up.