AB Test Sample Calculator
Estimate how many users you need in each variant before launching an A/B test. Enter your baseline conversion rate, the minimum detectable uplift you care about, your confidence level, and desired power to get a defensible sample size target.
Sample Size Calculator for A/B Testing
Built for conversion-rate experiments using a two-sample proportion test with equal traffic allocation.
Expert Guide: How to Use an AB Test Sample Calculator Correctly
An A/B test sample calculator helps you decide how many observations you need before you can trust an experimental result. In practical terms, it tells you how many users, sessions, visitors, or leads should be included in your control group and your variant group so your test has a realistic chance of finding a true effect. Without sample size planning, teams often stop too early, call winners too quickly, and make product or marketing decisions based on noise instead of signal.
For conversion experiments, the core problem is straightforward: you have a current conversion rate and you want to know how much traffic is required to detect a meaningful change. That change is usually expressed as a minimum detectable effect, often shortened to MDE. If your current checkout conversion rate is 5.0% and you want to detect a relative uplift of 10%, your target variant rate is 5.5%. A good calculator translates that business question into a sample requirement.
Quick takeaway: Smaller effects require larger samples. Higher confidence requires larger samples. Higher statistical power requires larger samples. Lower baseline conversion rates also tend to require more users when the absolute difference you care about is small.
What the calculator is actually estimating
This calculator estimates sample size for a two-group test of proportions. That means it is designed for outcomes like conversion versus no conversion, signup versus no signup, click versus no click, or purchase versus no purchase. The tool assumes equal traffic allocation between control and variant and uses a normal approximation commonly applied in A/B testing workflows.
The calculation depends on five key ideas:
- Baseline conversion rate: your current expected conversion rate in the control group.
- Minimum detectable uplift: the smallest relative improvement worth detecting.
- Confidence level: your tolerance for false positives. At 95% confidence, the nominal significance level is 5% for a two-sided test.
- Statistical power: your probability of detecting a true effect when it really exists. Many teams use 80% as a default.
- One-sided or two-sided testing: two-sided testing is more conservative and is standard for most experiments.
Why sample size matters so much in experimentation
Many failed experimentation programs do not fail because testing is a bad idea. They fail because teams underpower tests. An underpowered test creates two expensive problems. First, it may miss a genuine improvement, which creates a false negative. Second, it can exaggerate early random swings and tempt teams into stopping the test too soon. This is especially common when daily dashboards show unstable win rates in the first few days.
Sample planning creates discipline. When you know in advance that a test needs, for example, 31,000 users per variant, you are much less likely to believe a dramatic but unstable result after only 2,000 users. You also gain a more realistic sense of whether an experiment is feasible. Some tests simply are not worth running if the needed runtime is too long or if the effect size is too small to matter commercially.
Understanding confidence level and false positives
Confidence level is the complement of your chosen significance level. A 95% confidence level usually corresponds to a 5% significance threshold for a two-sided test. In plain language, if there were truly no difference between A and B and you repeated the test many times, about 5 out of 100 such tests could appear significant just by chance.
| Confidence Level | Alpha for Two-Sided Test | Approximate Critical Z-Score | Interpretation |
|---|---|---|---|
| 90% | 0.10 | 1.645 | Less conservative, smaller sample size, higher false-positive tolerance |
| 95% | 0.05 | 1.960 | Common business default balancing rigor and practicality |
| 99% | 0.01 | 2.576 | Very conservative, much larger sample size requirement |
Those Z-scores are standard values used in statistical testing. When confidence rises from 95% to 99%, required sample sizes can increase substantially, especially when the target effect is small. This is why experimentation teams often reserve 99% thresholds for extremely high-risk decisions rather than using them universally.
Understanding statistical power and false negatives
Power is the probability that your experiment will detect a true effect of the size you specified. If your test has 80% power, that means it has an 80% chance of finding a statistically significant result when the true effect is exactly your minimum detectable effect. The remaining 20% represents Type II error, also called beta.
In business terms, low power means you may overlook changes that are genuinely helpful. Power depends on the effect size you care about, the variability of the metric, and the amount of traffic you can collect. Increasing power from 80% to 90% is often desirable for strategically important experiments, but you should expect a meaningful increase in sample size.
Minimum detectable effect is the business lever that matters most
Teams often focus too much on confidence and not enough on MDE. The MDE is where strategy enters the math. If you set the MDE too small, your required sample size may explode, turning a simple landing-page test into a multi-month effort. If you set it too large, you might miss worthwhile gains. The best MDE is usually the smallest change that would justify implementation cost, engineering effort, creative updates, and organizational attention.
Suppose your baseline conversion rate is 5%. A 10% relative uplift means detecting a move to 5.5%. A 20% relative uplift means detecting a move to 6.0%. Because the second difference is larger in absolute terms, it requires fewer observations. This is one reason experienced experimentation teams prioritize bold, high-contrast test ideas instead of tiny cosmetic changes.
| Baseline Rate | Relative Uplift | Target Variant Rate | Typical Effect on Needed Sample |
|---|---|---|---|
| 5.0% | 5% | 5.25% | Very large sample requirement |
| 5.0% | 10% | 5.50% | Large sample requirement |
| 5.0% | 20% | 6.00% | Moderate sample requirement |
| 5.0% | 30% | 6.50% | Meaningfully smaller sample requirement |
How to use this calculator step by step
- Enter your current baseline conversion rate as a percentage.
- Choose the smallest relative lift that would matter to the business.
- Select a confidence level, usually 95% for routine experiments.
- Select your desired power, commonly 80% or 90%.
- Choose two-sided testing unless you have a very strong pre-registered directional hypothesis.
- Enter estimated daily visitors per variant to get a runtime estimate.
- Click calculate and review per-variant sample size, total sample size, target variant conversion rate, and estimated duration.
If your required sample is too high, you have three realistic options: accept a larger MDE, increase traffic allocation, or redesign the experiment to produce a stronger contrast between the control and the variant. In many cases, better test design beats waiting months for a tiny detectable effect.
Examples of sample planning scenarios
Below are illustrative examples using standard proportion test assumptions. These numbers are representative and designed to show how quickly required traffic grows when effect sizes shrink.
| Baseline Conversion | MDE Relative Uplift | Confidence | Power | Approx. Sample per Variant |
|---|---|---|---|---|
| 5.0% | 10% | 95% | 80% | 31,000+ |
| 5.0% | 20% | 95% | 80% | 8,000+ |
| 10.0% | 10% | 95% | 80% | 14,000+ |
| 20.0% | 10% | 95% | 80% | 6,000+ |
These examples reveal a pattern many teams underestimate. Detecting small changes at low baseline rates is expensive. If your funnel step converts at 2% and you only care about a 5% relative lift, your experiment may need an enormous amount of traffic. That is not a defect in the calculator. It reflects the reality of statistical uncertainty.
Common mistakes when using an AB test sample calculator
- Using unrealistic baseline rates: If your baseline is stale or seasonal, your sample estimate may be wrong from the start.
- Confusing relative and absolute uplift: A rise from 5% to 6% is a 1 percentage point gain but a 20% relative uplift.
- Stopping early after significance appears: Peeking repeatedly can inflate false-positive risk.
- Ignoring practical significance: A statistically significant result can still be too small to matter financially.
- Testing too many variants: More arms generally require more total traffic and more careful multiple-comparison handling.
- Applying the wrong metric type: This calculator is best suited to binary conversion outcomes, not revenue per user with heavy skew.
When should you use one-sided versus two-sided testing?
Two-sided testing asks whether the variant is different from the control in either direction. One-sided testing asks whether the variant is better in only one direction. Because one-sided tests place all alpha in one tail, they can require less sample. However, they should be used cautiously. In most product, UX, and conversion experiments, a two-sided framework is safer because a change can either help or hurt performance. If you decide on a one-sided test, that decision should be made before the experiment begins.
What this calculator does not cover
No sample calculator should be treated as magic. This one is intended for standard fixed-horizon planning for binary outcomes. It does not automatically adjust for sequential testing, multiple metrics, novelty effects, unequal allocation, cluster randomization, or complex Bayesian designs. If your company runs mature experimentation programs with heavy metric guardrails or adaptive stopping rules, you may need a more specialized framework.
It also does not solve measurement quality issues. If your event instrumentation is broken, if attribution windows are inconsistent, or if bot traffic contaminates sessions, your result quality will suffer regardless of sample size. Good statistics cannot rescue poor data collection.
How to interpret the result responsibly
The output should be treated as a planning number, not a guarantee. Real traffic can be unbalanced. Actual baseline rates can drift. User behavior can change because of seasonality, promotions, holidays, or channel mix shifts. The best use of a sample-size estimate is to set a minimum runtime expectation and define decision rules before launch.
As a practical rule, do not run tests only until the sample target is reached if you have strong day-of-week effects. Try to cover full weekly cycles where possible so both variants experience comparable traffic patterns. A seven-day or fourteen-day minimum runtime is often sensible for consumer products with clear weekday and weekend differences.
Authoritative sources for deeper statistical reading
If you want to study the underlying concepts in more depth, these sources are strong starting points:
- NIST Engineering Statistics Handbook (.gov)
- Penn State Online Statistics Program (.edu)
- U.S. Census Bureau Statistical Tutorials (.gov)
Final advice for marketers, product managers, and CRO teams
A strong A/B testing culture starts with realistic expectations. Not every test needs to detect tiny lifts. In many organizations, the highest-return experimentation strategy is to test fewer ideas but make each one more strategically meaningful. Pair a sensible confidence threshold with a realistic power target, define a business-relevant MDE, and commit to a preplanned sample size before launch. If you do that consistently, this AB test sample calculator becomes more than a planning widget. It becomes a quality-control system for better decisions.
Use the calculator above as your first checkpoint. If the required sample looks achievable, move forward with confidence. If it looks too large, treat that as a signal to revisit the hypothesis, increase the expected contrast, or reconsider the value of the test. Good experimentation is not only about statistics. It is about using statistics to prioritize the right work.