A/B Testing Sample Size Calculator
Estimate how many visitors you need per variation before launching an experiment. Enter your baseline conversion rate, minimum detectable effect, confidence level, power, and test sidedness to calculate a statistically grounded sample size for a two sample conversion test.
Calculator
Enter your assumptions and click Calculate Sample Size to see required visitors, expected conversions, and a planning chart.
How an A/B testing sample size calculator works
An A/B testing sample size calculator helps marketers, growth teams, product managers, and analysts decide how much traffic is required before an experiment can detect a meaningful difference. Instead of guessing how long a test should run, a calculator uses statistical assumptions to estimate the number of observations needed for a reliable result. In practical terms, this means fewer false wins, fewer missed opportunities, and a better balance between speed and rigor.
For most website and product experiments, the core outcome is a conversion rate. You may want to know whether a new headline increases signup rate, whether a revised checkout flow reduces cart abandonment, or whether a new pricing page improves purchases. In all of these cases, your observed result is a proportion: conversions divided by visitors. Because conversion data follows binomial behavior, sample size formulas are designed around comparing two proportions.
The calculator above uses five central inputs: your baseline conversion rate, the minimum detectable effect, confidence level, statistical power, and traffic allocation. These values are not just technical settings. They define how ambitious, conservative, or realistic your testing program will be.
1. Baseline conversion rate
The baseline conversion rate is your current best estimate of how often users convert. If your current landing page converts 5 out of every 100 visitors, your baseline is 5%. This value matters because lower conversion rates generally require larger sample sizes. Detecting a 1 percentage point change from 1% to 2% is statistically harder than detecting a 1 percentage point change from 15% to 16%.
2. Minimum detectable effect
The minimum detectable effect, often shortened to MDE, is the smallest change worth detecting. If your baseline is 5% and you enter a 1 percentage point uplift, you are planning to detect a lift from 5% to 6%. A smaller MDE always requires more traffic because you are trying to distinguish a subtler difference from random noise.
Teams often make the mistake of setting an unrealistically small MDE because they want precision. The tradeoff is time. If your traffic volume is limited, choosing an MDE that is too small can create tests that run for weeks or months and still fail to conclude cleanly.
3. Confidence level and alpha
A 95% confidence level corresponds to a 5% significance threshold, commonly called alpha = 0.05. This setting controls the chance of a false positive, meaning the chance that you conclude a variant is different when the observed difference is actually due to random variation. A stricter confidence threshold such as 99% lowers false positives, but increases the required sample size.
4. Statistical power
Power is the probability that your test will correctly detect a real effect of the specified size. An 80% power target is a common standard. It implies that if the true uplift equals your chosen MDE, your experiment has an 80% chance of finding a statistically significant difference. Raising power to 90% or 95% is more conservative, but it also means more required traffic.
5. Allocation ratio
Sample size grows when traffic is unevenly split. A 50 / 50 split minimizes total required visitors for a two arm test because both groups contribute equally to precision. If you route only 25% of users to the variant, you may reduce exposure risk, but you will usually need more total traffic before the test reaches the same sensitivity.
Why sample size matters in real experimentation programs
Running experiments without adequate sample size creates fragile decision making. A test can look promising after a few hundred visitors and then regress to the mean once additional traffic arrives. This happens because early results are highly volatile. Without enough observations, random fluctuations can resemble real performance differences.
On the other hand, collecting far more data than necessary is also costly. It slows decision cycles, delays launches, and can prevent teams from testing enough ideas. A strong calculator finds the middle ground: enough traffic to support reliable inference, but not so much that the organization loses momentum.
- Too small a sample size increases false negatives and unstable results.
- Too large a sample size increases opportunity cost and testing cycle time.
- Well planned sample sizes improve trust in experiment reporting.
- Consistent assumptions create repeatable experimentation standards across teams.
Reference assumptions and what they imply
The table below shows how required visitors per variant change as baseline rate and desired uplift change under common assumptions: 95% confidence, 80% power, and a balanced 50 / 50 split. These are rounded planning values using standard two proportion test assumptions.
| Baseline conversion | Target variant conversion | Absolute uplift | Approx. visitors per variant | Approx. total visitors |
|---|---|---|---|---|
| 2% | 2.5% | 0.5 percentage points | 13,500 | 27,000 |
| 5% | 6% | 1.0 percentage point | 8,100 | 16,200 |
| 10% | 11.5% | 1.5 percentage points | 6,500 | 13,000 |
| 20% | 22% | 2.0 percentage points | 6,000 | 12,000 |
Notice that sample size does not scale in a simple straight line. It depends on both the baseline rate and the size of the effect you hope to detect. The practical takeaway is that tiny changes require large audiences. If your site receives only a few thousand qualified visitors each month, trying to detect very small uplifts may not be feasible.
What formula is typically used?
For binary outcomes such as conversions, many calculators use a normal approximation for the difference between two independent proportions. The rough structure of the sample size formula combines:
- The average expected conversion behavior across the two groups.
- The chosen alpha threshold derived from the confidence level.
- The chosen beta threshold derived from statistical power.
- The gap between control and variant conversion rates.
In simple terms, the formula asks: how much random variation should we expect, and how many observations do we need so that a difference of this size stands out from that variation with acceptable certainty?
The calculator on this page applies a standard two sample proportion approach. It is appropriate for planning many common website experiments such as signup rate, add to cart rate, purchase rate, click through rate, or lead form completion.
Recommended defaults for most teams
If you are not sure where to start, a practical default setup is 95% confidence, 80% power, a two-sided test, and a 50 / 50 traffic split. These defaults are widely used because they offer a good compromise between rigor and speed. You can tighten standards later if your organization works in a high risk domain such as healthcare, public policy, or regulated financial products.
| Decision setting | More conservative choice | Effect on required sample size | Best use case |
|---|---|---|---|
| Confidence level | 99% instead of 95% | Higher | When false positives are very costly |
| Power | 90% instead of 80% | Higher | When missing real wins is costly |
| Test type | Two-sided instead of one-sided | Higher | When either uplift or decline matters |
| Traffic split | 50 / 50 instead of 75 / 25 | Lower total sample size | When you can expose both groups equally |
Common mistakes when estimating sample size
Using a guessed baseline
If your baseline conversion rate comes from outdated or noisy data, your estimate may be off. Use recent, representative traffic whenever possible. If the metric is highly seasonal, consider using a rolling average across a relevant time period.
Choosing an MDE that is too optimistic
Some teams choose a 20% or 30% relative lift because it produces a smaller sample size. This can be misleading if most of your past wins were much smaller. Your MDE should reflect business relevance and historical realism, not just convenience.
Stopping the test early
Peeking at results before you hit the planned sample size can inflate false positive risk. If your testing platform does not support sequential methods, it is better to commit to a target sample size before launching.
Ignoring quality and segmentation
Traffic quality matters. If your audience contains several user segments with very different behaviors, your overall baseline may not represent the segment you care about. Segment specific tests often require their own sample size planning.
How to interpret the calculator output
After calculation, you will see required visitors for the control group, required visitors for the variant group, total visitors, and expected conversions at the planning assumptions. Think of these as design targets, not guaranteed outcomes. Real experiments can underperform the planned effect, and data quality issues such as bot traffic, duplicate users, or tracking gaps can reduce usable sample.
- Visitors per control: the estimated number of users needed in the original experience.
- Visitors per variant: the estimated number of users needed in the test experience.
- Total visitors: the combined audience required across both arms.
- Expected conversions: an intuitive estimate of event counts under the planning scenario.
Good experimentation practices beyond sample size
Sample size is only one part of a strong experiment design. You should also define a primary metric, keep implementation clean, maintain random assignment, and avoid changing targeting rules halfway through the test. If multiple teams run experiments at once, monitor overlap and interaction effects. In addition, document your assumptions before launch so stakeholders understand what the test was designed to detect.
It is also wise to estimate test duration. For example, if your calculator says you need 20,000 total visitors and your eligible page receives 2,000 visitors per day, the test will need about 10 days at full traffic, not counting guardrails such as weekday and weekend balancing. This helps avoid the classic problem of launching a test that could never finish within a useful business window.
Authoritative resources for statistical testing and sample size
If you want deeper technical grounding, the following references are strong places to continue:
- NIST Engineering Statistics Handbook for foundational guidance on hypothesis testing, proportions, and experimental analysis.
- Penn State STAT Online for university level explanations of statistical inference and power concepts.
- U.S. Census Bureau guidance on sample size for practical intuition on precision and planning.
Final takeaway
An A/B testing sample size calculator brings discipline to experimentation. It converts strategic decisions into measurable test requirements. By defining a realistic baseline, meaningful MDE, sensible confidence threshold, and adequate power, you can launch experiments with far greater clarity. The result is not just better statistics. It is better product decision making, faster learning, and more confidence that winning variants are actually worth shipping.