Ab Test Calculator Sample Size

Estimate how many users you need before launching an A/B test with confidence. This calculator uses a standard two-proportion sample size formula so you can plan around baseline conversion rate, minimum detectable effect, significance level, statistical power, test sidedness, and available traffic.

Sample Size Calculator

How to use an AB test calculator sample size tool the right way

An AB test calculator sample size tool answers one of the most important questions in experimentation: how much traffic do you need before a result can be trusted? Many teams launch tests too early, see a short-term lift, and stop the experiment long before enough evidence has accumulated. Others choose a target effect that is unrealistically tiny for their traffic level, which leads to tests that run for months without reaching significance. A well-built calculator prevents both mistakes by translating your business assumptions into a concrete audience requirement.

At a practical level, sample size is driven by five variables: your baseline conversion rate, your minimum detectable effect, your significance level, your desired statistical power, and the number of variants. The smaller the effect you want to detect, the larger the required sample. The higher your desired confidence and power, the larger the required sample. When you add more variants, your traffic is split more ways, so the total traffic requirement goes up even if the per-variant requirement stays similar. That is why sample size planning should happen before a test is designed, not after it is launched.

What each input means

• Baseline conversion rate: your current expected conversion probability under the control experience. If your page converts at 5%, your baseline is 0.05.
• Minimum detectable effect: the smallest change that would matter to the business. This can be set as a relative uplift, such as 20%, or an absolute lift, such as 1 percentage point.
• Significance level: also called alpha, it controls your false positive risk. A 95% confidence threshold corresponds to alpha = 0.05.
• Power: the probability that your test will detect a real effect if it exists. A common target is 80%.
• Sidedness: two-sided tests can detect either improvement or decline, while one-sided tests only look in one direction.
• Variants: total experiences in the experiment, including the control. More variants mean more traffic needed overall.

Why sample size matters more than most marketers think

Underpowered tests are one of the biggest reasons experimentation programs produce noisy or contradictory results. Suppose your baseline conversion rate is 5% and you hope to detect a 20% relative uplift, which means an increase to 6%. That difference is only one percentage point in absolute terms. Even though the relative change looks large, random variation can easily create a one-point swing in small samples. The role of sample size planning is to make sure your data can separate signal from noise.

This matters because an A/B test is not just a dashboard exercise. It affects design decisions, product roadmaps, media spend, and revenue forecasts. If a team stops a test after only a few hundred visitors because the treatment is “winning,” they expose themselves to regression to the mean and false discovery. If they wait for an appropriately sized sample, their final readout is far more likely to generalize to future users.

The calculator on this page uses a standard normal approximation for the difference in two conversion proportions. In plain English, it estimates how many observations per group are needed so that the expected treatment lift stands out relative to the natural variability of binary conversion data. While advanced settings such as sequential testing, Bayesian monitoring, uneven allocation, and multiple comparison corrections can be layered on later, this baseline calculation is the starting point used by many professional experimentation teams.

Confidence, alpha, and false positives

Significance level and confidence are two ways of describing the same tolerance for false alarms. If you choose alpha = 0.05, you are saying that under repeated testing, you accept about a 5% chance of declaring a result significant when there is actually no true difference. This is not a guarantee about one specific test, but a long-run operating characteristic of the procedure.

Confidence level	Alpha	Two-sided critical z value	Interpretation in testing
90%	0.10	1.645	Requires less evidence than 95%, so tests need fewer users but accept a higher false positive risk.
95%	0.05	1.960	The most common default for growth, product, and conversion rate optimization teams.
99%	0.01	2.576	Very strict standard that increases sample size substantially to reduce false positives.

The table above uses standard critical values from the normal distribution. As confidence increases, the threshold for claiming a win becomes more demanding. That greater rigor is valuable when the business cost of a wrong decision is high, but it also means you need more traffic.

Power and false negatives

Power is the counterpart to significance. While alpha controls false positives, power controls false negatives. An 80% power target means that if the true effect is at least as large as your chosen minimum detectable effect, your experiment has an 80% chance of detecting it. A low-powered test may fail to find a meaningful improvement even when the treatment truly works.

This is why experimentation leaders often say that the minimum detectable effect should be “worth detecting.” If your business would only act on a lift of at least 10% or 20%, then set the calculator to that threshold. If you insist on detecting tiny changes such as 1% relative improvement on a low-converting page, your traffic needs may become unrealistic.

Example sample size planning scenarios

The exact result depends on the assumptions you enter, but the patterns are consistent. Lower baseline rates usually need more traffic for the same relative lift because conversions are rarer events. Smaller effects need much more traffic. Higher confidence and power both increase required sample size. The following scenarios illustrate realistic planning outcomes for a two-variant test using 95% confidence and 80% power.

Baseline conversion rate	Target effect	Interpretation	Approximate sample per variant
5.0%	+20% relative lift to 6.0%	Common ecommerce or lead-gen planning case	About 8,100 users per variant
10.0%	+10% relative lift to 11.0%	Moderate improvement on a stronger funnel	About 14,700 users per variant
20.0%	+10% relative lift to 22.0%	Absolute difference is 2 points, which is easier to detect than a tiny lift	About 6,500 users per variant
2.0%	+25% relative lift to 2.5%	Low-conversion experiences often need a lot of traffic	About 15,200 users per variant

These figures are not arbitrary. They reflect the real math behind two-proportion power analysis. Notice how a baseline of 10% with a one-point absolute lift can require more traffic than a baseline of 20% with a two-point lift, even though both are meaningful changes. The reason is simple: the signal-to-noise ratio changes with both the underlying probability and the effect size.

Absolute lift versus relative uplift

A common source of confusion is whether to plan around absolute or relative change. If your baseline conversion rate is 5%, then:

• An absolute lift of 1 percentage point means moving from 5% to 6%.
• A relative uplift of 20% also means moving from 5% to 6%.
• A relative uplift of 10% means moving from 5% to 5.5%.

For low baseline rates, seemingly small absolute lifts can represent large relative changes. For high baseline rates, even modest relative improvements can become meaningful absolute differences. The calculator lets you use either framing because different teams think in different units. Growth teams often discuss relative uplifts, while product and analytics teams often prefer absolute percentage-point changes.

How to choose a realistic minimum detectable effect

Your minimum detectable effect should not be a guess copied from another team. It should come from business value, implementation effort, and expected variance. A helpful decision framework is:

1. Estimate the smallest lift that would justify rollout costs.
2. Check whether that lift is plausible given past experiments and the nature of the change.
3. Run the sample size calculation and compare the traffic requirement with your expected audience.
4. If the required duration is too long, increase the effect threshold, simplify the test, or focus on a higher-conversion segment.

For example, if a homepage redesign would take weeks of engineering time, a 1% relative lift may not justify the cost. But for a pricing page or checkout flow, even a small lift can have outsized revenue impact. The calculator helps you quantify whether your expected gain is measurable within a practical time frame.

Balanced allocation is usually the default assumption in sample size calculators. If your test uses uneven traffic splits or if you plan frequent peeking without a proper sequential framework, your actual evidence threshold can change.

Common mistakes that create bad test decisions

• Stopping early after a short-term winner appears: early swings are common and often disappear as sample accumulates.
• Choosing an unrealistically small MDE: this can force tests to run so long that seasonal shifts contaminate the result.
• Ignoring traffic allocation across multiple variants: adding a third or fourth version slows learning unless traffic is abundant.
• Mixing users of very different intent into one test: segmentation can improve interpretability when behavior varies widely.
• Focusing only on primary conversion: guardrail metrics such as bounce rate, refund rate, or page speed may change too.

When the standard formula is enough and when it is not

The standard fixed-horizon sample size formula is ideal for many website experiments with binary outcomes such as signup, purchase, or click-through. It is especially useful in planning conversations because it is transparent and fast. However, there are situations where you may want a more advanced design:

• Sequential testing: if you plan to look at results repeatedly and stop when evidence crosses a boundary.
• Bayesian experimentation: if your team prefers posterior probabilities and expected loss frameworks.
• Multiple comparisons: if you run many variants or inspect many metrics, error control may need adjustment.
• Continuous outcomes: revenue per user, time on page, or order value use different variance assumptions than binary conversion.
• Clustered or repeated observations: for example, account-level tests where one account generates multiple user sessions.

Even in those cases, the simple sample size calculator remains useful as a first-pass planning tool. It provides a credible directional estimate and keeps teams from launching tests that are obviously underpowered from the start.

How to interpret the calculator output

Once you click Calculate Sample Size, focus on four outputs: required sample per variant, total sample across all variants, expected treatment conversion rate, and estimated duration. Required sample per variant tells you the minimum audience each experience should receive. Total sample is useful for media and traffic planning. Expected treatment rate lets stakeholders see the business target in plain language. Estimated duration turns statistics into scheduling reality.

If the duration is too long, do not simply lower the confidence threshold to force a faster answer. Instead, revisit the experiment design. You might test a bolder treatment, narrow the audience to higher-intent users, reduce the number of variants, or prioritize a page with stronger baseline conversion. Good experimentation is not just about mathematical correctness. It is about aligning statistical rigor with business practicality.

Useful external references

For readers who want deeper statistical background, these sources are strong starting points:

• NIST Engineering Statistics Handbook for core hypothesis testing concepts and distribution-based decision thresholds.
• Penn State Statistics Online for formal treatment of inference for proportions and test planning.
• UC Berkeley Statistics for broader academic resources on probability, inference, and statistical reasoning.

Final planning checklist for reliable A/B tests

1. Use a trustworthy estimate of baseline conversion rate from recent, relevant traffic.
2. Set an MDE that is both meaningful to the business and plausible for the treatment.
3. Default to 95% confidence and 80% power unless there is a strong reason to change them.
4. Assume a two-sided test unless your organization has a documented rationale for one-sided testing.
5. Include all variants in total traffic planning, not just the control and a single treatment.
6. Run the test long enough to reach the planned sample and to cover normal weekday or weekend behavior cycles.
7. Review secondary metrics and implementation quality before declaring a rollout winner.

When used properly, an AB test calculator sample size tool turns experimentation from opinion into disciplined decision-making. Instead of asking whether a dashboard looks promising today, you ask whether the planned design has enough evidence to answer the question at all. That shift is what separates a mature testing program from one driven by anecdote. Start with the numbers, commit to an evidence threshold before launch, and let the data reach the sample it truly needs.