A/B Test Calculator: Statistical Significance and Sample Size

An A/B test calculator tells you whether the difference between two versions of a page, email or ad is real or just noise, and how much traffic a test needs before you can trust it. It works as both an A/B test significance calculator and an A/B split test planner, with two modes: a significance test for results you already have, and a sample size planner for tests you are about to start, including how many days the test will take at your traffic.

How to Use the A/B Test Calculator

Choose a mode with the buttons at the top: Test significance or Sample size planner. Both modes share the confidence setting. Everything runs in your browser and nothing you type is sent anywhere.

Test significance mode

• Visitors in A (control): users or sessions that saw the original version. Use the same unit for both versions.
• Conversions in A: how many of them completed the goal, such as a purchase or form submission.
• Visitors in B (variant) and conversions in B: the same for the new version.
• Confidence: 90%, 95% (the default) or 99%. Higher confidence means fewer false winners but needs more data.

It returns the conversion rate of A and B, the relative uplift of B over A, and a two-sided p-value. The note under the results says whether the difference is significant at your confidence level, whether B beat A or lost to it, and the z-score behind the call. If it is not significant, the note tells you to keep the test running or treat it as a tie.

Sample size planner mode

• Baseline conversion rate (%): the current conversion rate of the page or step you are testing.
• Smallest lift worth detecting (relative %): the minimum detectable effect. A 15% relative lift on a 3% baseline means detecting a move to 3.45%.
• Confidence: as above.
• Daily visitors to the test (optional): total daily traffic entering the test across both versions.

It returns visitors needed per variant, total visitors, and the number of days the test will take at your daily traffic. The planner uses 80% statistical power, a standard choice that means the test has an 80% chance of detecting a real lift of the size you entered.

How the Significance Test Works

The significance mode runs a two-proportion z-test: it compares the two conversion rates, measures the gap in units of standard error, and converts that into a p-value. In formula form:

• Conversion rate A = conversions A / visitors A, and the same for B
• Pooled rate = (conversions A + conversions B) / (visitors A + visitors B)
• Standard error = square root of [pooled rate x (1 minus pooled rate) x (1 / visitors A + 1 / visitors B)]
• z = (rate B minus rate A) / standard error
• p-value (two-sided) = 2 x (1 minus the normal cumulative probability of the absolute z)

This is what makes it an A/B test statistical significance calculator rather than a simple rate comparison. The result is significant when the p-value is below 1 minus your confidence level: below 0.05 at 95% confidence, below 0.10 at 90% and below 0.01 at 99%. Because the test is two-sided, it also catches variants that are significantly worse, which is often the most valuable result a test produces.

For sample size, the planner uses the standard formula for comparing two proportions: the z-value for your confidence level and the z-value for 80% power, combined with the variance at the baseline and target rates, divided by the squared difference between them.

Worked Example: A Winner and a Tie

Example numbers, not benchmarks. Two checkout tests each send 10,000 visitors to each version, with a 3% baseline.

Test 2 shows a 10% uplift, which sounds worth shipping, but with this much traffic a gap that size appears by chance often enough that the calculator will not call it. Test 1 has a larger gap on the same traffic and clears the 95% bar. Neither result tells you the exact size of the true lift; a 20% observed uplift could be a smaller real lift that got lucky.

What Is Statistical Significance in an A/B Test?

Statistical significance means the difference you observed would be unlikely if the two versions actually performed the same. At 95% confidence you accept a 5% chance of declaring a winner when there is no real difference, which is called a false positive.

Significance is about reliability, not size or value. A tiny lift can be significant on huge traffic, and a large lift can fail to reach significance on small traffic. The business question is always two questions: is the difference real, and is it big enough to matter?

What the P-Value Does and Does Not Tell You

The p-value is the probability of seeing a difference at least as large as yours if A and B truly converted at the same rate. A p-value of 0.02 means a gap this size would show up about 2% of the time by chance alone.

It is not the probability that B is better, and it is not the probability that you are wrong. It also says nothing about how large the true lift is. Treat it as a filter that keeps noise out, and use the uplift and your revenue numbers to decide whether the change is worth the effort.

How the A/B Test Sample Size Calculator Works

An A/B test sample size calculator tells you how many visitors each version needs before a test can reliably detect a lift of a given size. You set three things: the baseline conversion rate, the smallest lift worth detecting, and the confidence level. The planner fixes power at 80%.

The smaller the baseline rate and the smaller the lift, the more visitors you need, and the relationship is steep: halving the lift you want to detect roughly quadruples the sample size. The table shows the planner's output for a 3% baseline at 95% confidence.

Confidence matters too. At the same 3% baseline and 15% lift, the planner needs 19,057 visitors per variant at 90% confidence and 35,999 at 99%.

Choosing a Minimum Detectable Effect

Pick the smallest lift that would be worth the cost of building and keeping the change, then check whether your traffic can detect it in a reasonable time. If the planner says a 10% lift needs four months, you have three options: test bolder changes that could plausibly move conversion by more, test higher up the funnel where the baseline rate is larger, or accept 90% confidence for low-risk changes.

What you should not do is pick a large lift just to make the sample size look small. The test will then be underpowered for the realistic lifts most changes produce, and most of your tests will end as ties.

How Long Should an A/B Test Run?

An A/B test should run until it reaches the planned sample size and for at least one full week, ideally two, so that every weekday and weekend pattern is included. Used as an A/B test duration calculator, the days output in the planner gives the minimum based on traffic; round it up to whole weeks.

Shoppers on a Monday behave differently from shoppers on a Saturday, and paydays, email sends and promotions all shift conversion rates. A test that runs Tuesday to Friday measures a slice of your customers, not all of them. Avoid starting tests during a sale unless the sale is what you are testing.

Why Tests Fail to Reach Significance

Most tests that never reach significance are underpowered, testing changes too small to detect with the available traffic. Other common causes:

• The change is invisible. A button shade or a minor headline tweak rarely changes behavior enough to measure.
• The goal is too far away. Testing a homepage change against completed purchases dilutes the effect through every step in between. Measure the next step as well.
• Traffic is split across too many variants. Each extra variant divides the sample.
• The versions really perform the same. A clean tie is a valid result. It tells you the change does not matter and frees you to test something else.

A/B Testing Mistakes That Produce False Winners

• Peeking and stopping early. Checking results every day and stopping the moment p drops below 0.05 inflates false positives well beyond 5%. Decide the sample size first and judge the test once it is reached.
• Sample ratio mismatch. If you split traffic 50/50 but one version received noticeably more visitors, something in the setup is broken, such as redirects, bots or tracking. Fix it before trusting any result.
• Testing many variants without adjustment. This calculator compares two versions. Comparing several variants against the control raises the chance that one wins by luck, so use a stricter confidence level or a tool built for multiple comparisons.
• Changing the test mid-flight. Editing a variant, changing traffic allocation or launching a big campaign halfway through mixes two different tests in one dataset.
• Novelty effects. Returning visitors sometimes click on something just because it is new. Running for whole weeks helps the effect wash out.

Frequentist vs Bayesian A/B Testing

This calculator is frequentist: it asks how surprising your data would be if there were no difference, and it answers with a p-value and a fixed confidence threshold. Bayesian A/B testing instead estimates the probability that B is better than A, using prior assumptions plus the data.

Bayesian outputs such as "92% chance B is better" are easier to explain to stakeholders, and several testing platforms use them. They do not remove the need for enough traffic, and peeking still causes problems if you stop on the first good number. For a clear yes or no on a finished two-version test, the frequentist approach here is the standard method.

Turning Test Results Into Revenue

A significant result is the start, not the end. Ship the winner, then check that the lift holds in your normal analytics over the following weeks, and work out what it is worth with the conversion rate calculator, which converts a rate change into extra conversions and revenue.

If your tests keep ending as ties, or you do not have the traffic to test at all, the problem is usually what you are testing rather than the statistics. Our conversion rate optimization work starts with analytics, session recordings and checkout audits to find changes large enough to measure, and uses this same sample size math to decide which tests are worth running. To discuss a testing program, contact us. Other calculators are on our free marketing tools page.