A/B Test Significance

Did variant B really beat A, or is it noise? An A/B test turns two conversion counts into a clear yes-or-no on statistical significance.

A/B Test SignificanceLive

two-proportion z-test

Controls

Presets

A visitors

A conversions

B visitors

B conversions

An A/B test compares two conversion rates to see if a difference is real or just noise. The two-proportion z-test pools the data to estimate the standard error, then a p-value below 0.05 signals a statistically significant difference. Bigger samples separate the two curves and make small lifts detectable.

▶ Run in Python

Data Inspector

Rate A10.00%

Rate B12.50%

Lift+25.0%

p-value0.0769

Resultnot significant

Governing equation

z = \frac{p ^ _{B} - p ^ _{A}}{p ^ ( 1 - p ^ ) ( \frac{1}{n _{A}} + \frac{1}{n _{B}} )} = \frac{0.025}{0.113 \cdot 0.887 ( \frac{1}{1000} + \frac{1}{1000} )} = 1.77

Reading this result: B looks 25.0% different, but with only 2,000 visitors p=0.077 — the sample is too small to call it real yet.

Runs locally in your browser — free forever. Scale to the cloud when reality gets heavy.

or unlock everything with Pro →

★ Sign in to save this setup

Save your tuned setup, or drop this simulation into your own site, docs, or course page.

How it works

An A/B test compares the conversion rates of two variants with a two-proportion z-test. It pools the samples to estimate the standard error of the difference, then reports a p-value: below 0.05 means the observed lift is unlikely to be chance. Larger samples separate the two distributions and let you detect smaller true differences.

✦

Ask the AI about this model

The math, the assumptions, real-world uses, or a code translation — explained for this exact simulation.

More Statistics simulations

Hypothesis Test Linear Regression Central Limit Theorem Confidence Intervals Bayesian Inference Bootstrap PCA

Frequently asked questions

Is this A/B test significance calculator tool really free?▾

Yes. A/B Test Significance runs entirely in your browser using your device's own compute, so local use is free forever. You only pay Compute Tokens if you scale a job to the cloud.

Do I need to install anything?▾

No. Everything runs client-side in a modern browser — no downloads, no license, no account required to start.

Can I save or share my simulation?▾

Create a free account to save projects, and use a shareable embed or minted DOI to publish a live, interactive version anywhere.

How accurate are the results?▾

The solver uses established numerical methods, but results are for research and educational purposes and should be validated against experiment or professional review before you rely on them.