Ab Testing Calculate Sample Size

Estimate how many users you need per variation before launching an experiment. This premium calculator uses baseline conversion rate, minimum detectable effect, confidence level, and statistical power to compute a practical sample size for two variant A/B tests.

Expert Guide: How to Calculate Sample Size for A/B Testing

Learning how to calculate sample size for A/B testing is one of the most important skills in experimentation, conversion rate optimization, and product analytics. Many teams can design clever variant ideas, but far fewer know how many users they need before they can trust the outcome. That gap leads to false wins, false losses, and expensive business decisions based on noisy data.

Sample size answers a practical question: How much traffic do we need to detect a real difference between variant A and variant B? If the sample is too small, your test will be underpowered. Underpowered tests miss real improvements and often create unstable result swings. If the sample is too large, you may delay decisions unnecessarily and tie up valuable traffic that could be used for other experiments.

For standard conversion based A/B tests, the core inputs are simple:

• Baseline conversion rate: your current expected conversion rate for the control.
• Minimum detectable effect, often called MDE: the smallest relative uplift worth detecting.
• Significance level: commonly 5%, equivalent to 95% confidence for a two-sided test.
• Statistical power: commonly 80% or 90%, which helps reduce false negatives.
• Traffic split: how users are divided between the control and treatment variants.

Why sample size matters so much

A/B testing is built on probability. Every conversion rate you see in a live experiment is only an estimate of the true conversion rate. Because of random variation, short tests can produce dramatic but misleading outcomes. A variant may look 20% better after a few hundred users and then fade back to almost no difference after several thousand more. Sample size planning protects your decision process from that volatility.

A good sample size target does not guarantee a winning experiment. It gives your test enough statistical resolution to detect the level of improvement you care about.

Suppose your current checkout conversion rate is 10% and your team believes a new experience could improve conversion by at least 15% relative. That means you are looking for a lift from 10.0% to 11.5%, or a 1.5 percentage point absolute difference. Detecting a 1.5 point difference is substantially easier than detecting a 0.3 point difference. This is why the MDE has such a major impact on required traffic.

The core statistical idea

For a two sample test of proportions, sample size is based on the amount of separation between the control conversion rate and the expected treatment conversion rate, adjusted by the standard normal critical values for your selected significance and power. In plain English, the calculator asks how much natural randomness exists in your conversion data and how large a true effect must be before you can distinguish it from that noise with acceptable confidence.

The usual assumptions for a simple A/B sample size calculator are:

1. Each user is counted once and assigned independently.
2. The metric is binary, such as converted or did not convert.
3. The experiment is analyzed as a comparison of two proportions.
4. Users are exposed consistently to their assigned experience.
5. No major tracking errors or sample ratio mismatch distort the data.

How to interpret the key inputs

Baseline conversion rate should be based on recent, relevant, stable data. If your site converts around 4% during the last month, use that rather than a long term average from a different season or campaign mix.

Minimum detectable uplift should reflect business value, not wishful thinking. If a 2% relative uplift would not meaningfully affect revenue or profit, you may not want to design a test around detecting such a tiny difference. A smaller MDE always means a larger sample size, often dramatically larger.

Significance level controls false positives. A 5% significance level is common because it balances caution and speed reasonably well for many business experiments. Lowering alpha to 1% can be valuable in high stakes settings, but it increases required traffic.

Power controls false negatives. With 80% power, your test has a strong chance to detect the chosen MDE if the effect is truly present. Moving to 90% power increases reliability but also increases sample size.

Real world benchmark table: how often false signals appear in small samples

Scenario	Typical setup	Statistical implication	Operational impact
Low traffic test stopped early	Few hundred users per variant, repeated peeking	Greatly elevated risk of false positives beyond nominal 5%	Teams launch changes that do not reproduce
Properly powered test	80% to 90% power, preplanned sample size, controlled stopping rules	Closer alignment with expected Type I and Type II error rates	More stable win rates and higher decision quality
Tiny MDE chosen without enough traffic	Seeks 1% to 3% relative lift on low baseline metrics	Required sample can become impractically large	Test durations stretch for weeks or months

In digital experimentation programs, the biggest mistake is often not the formula itself. It is setting unrealistic expectations. Teams frequently want to detect very small uplifts on low base conversion rates while only having modest weekly traffic. The result is a test that never reaches informative scale. In those cases, the right answer may be to raise the MDE, combine related funnel steps, improve traffic quality, or test larger product changes.

What the sample size number actually means

If the calculator tells you that you need 8,000 users per variant, that means your experiment should generally continue until both the control and treatment arms reach about that count, assuming your setup remains valid. It does not mean you should stop the moment one variant crosses the threshold if the other has not. Nor does it mean the result is automatically trustworthy if instrumentation quality is poor.

It is also important to distinguish between users, sessions, and pageviews. Most A/B tests should randomize and measure at the user level when possible. Using pageviews can inflate effective sample size if the same person generates multiple observations, because those observations are not fully independent.

Comparison table: approximate per variant sample sizes at 95% confidence and 80% power

Baseline conversion rate	Relative uplift	Expected treatment rate	Approximate users per variant
5.0%	10%	5.5%	31,000+
5.0%	20%	6.0%	8,200+
10.0%	10%	11.0%	14,700+
10.0%	15%	11.5%	6,800+
20.0%	10%	22.0%	6,500+

These values are rounded planning figures for two group tests of proportions and show a pattern every growth team should understand: when the detectable difference gets smaller, sample size rises sharply. That relationship is one reason strategic prioritization matters. If a design change is only likely to move conversion by a tiny amount, you should ask whether the expected business impact justifies the long run time.

Common mistakes when calculating A/B test sample size

• Using a stale baseline: seasonality, promotions, and traffic source changes can shift conversion materially.
• Choosing an unrealistically small MDE: this is often the fastest path to impossible test duration.
• Ignoring traffic allocation: a 50/50 split is most efficient for two variant testing; heavily uneven splits require more total users.
• Stopping as soon as a dashboard turns green: peeking and early stopping can distort your actual error rates.
• Not accounting for exclusions: bots, QA users, repeat enrollment issues, and analytics filters reduce effective sample.

How traffic split affects sample size

For a simple two variant test, equal allocation is statistically efficient. If you send 90% of traffic to control and only 10% to treatment, your total required audience increases because the smaller arm becomes the bottleneck. Unequal allocation can still make sense when the treatment is risky or expensive, but you should understand the cost in duration and precision.

Should you use one-sided or two-sided tests?

Most business A/B tests use a two-sided test because they care whether the new variant is either better or worse. A one-sided test can produce a smaller required sample, but it is only justified if you truly would ignore evidence of harm in the opposite direction, which is uncommon in product and conversion work. In practice, two-sided planning is usually the safer default.

Authority sources for experimentation and statistical practice

If you want to go deeper, review these high quality public sources:

• U.S. Census Bureau for survey methodology concepts, sampling fundamentals, and statistical communication.
• Penn State Department of Statistics for accessible lessons on hypothesis testing, power, and sample size.
• National Institute of Standards and Technology for engineering statistics and measurement guidance relevant to experimental design.

Practical planning workflow for marketers and product teams

1. Estimate the current baseline conversion rate from recent data.
2. Define the smallest uplift that would matter financially.
3. Select significance and power based on decision risk.
4. Calculate users required per variant.
5. Translate that traffic requirement into days or weeks using expected eligible visitors per day.
6. Run the test cleanly without changing targeting, logging, or goal definitions midstream.
7. Check data quality before reading the business outcome.

Teams that follow this workflow consistently tend to make better decisions because they separate planning from interpretation. They decide in advance what evidence threshold is acceptable and then hold themselves to it. That discipline is especially important in organizations where many stakeholders are watching the same dashboard and hoping for a quick win.

Final takeaway

To calculate sample size for A/B testing well, do not focus only on the formula. Focus on decision quality. Start with a credible baseline, choose a meaningful MDE, keep confidence and power reasonable, and respect the traffic realities of your experiment. A calculator gives you the mathematical target, but sound experimentation practice is what turns that target into a trustworthy result.

Use the calculator above to estimate the required sample size per variant, compare alternative MDE assumptions, and understand how long your experiment may need to run. If the number looks surprisingly large, that is not a failure of the method. It is useful evidence that the effect you want to detect is small relative to the natural noise in your conversion data.