A B Test Sample Size Calculator

Estimate how many visitors or users you need in each variation before launching an A/B test. This calculator uses a standard two-sample proportion approach for conversion-rate experiments, helping teams plan tests with better statistical confidence, power, and minimum detectable effect assumptions.

Calculator

Results

What this calculator helps you decide

• How many users you need per variant before interpreting results.
• How confidence level and power affect sample size requirements.
• How baseline conversion rate and detectable uplift change test duration.
• Why smaller effects require much larger samples.

Expert Guide: How an A/B Test Sample Size Calculator Improves Experiment Quality

An A/B test sample size calculator is one of the most important planning tools in experimentation, conversion rate optimization, product analytics, and performance marketing. Before a team launches a split test, it should know how many users, sessions, or visitors are needed to detect a meaningful difference between a control and a variant. Without that estimate, teams often stop tests too early, overreact to random fluctuations, or celebrate “wins” that disappear when the experiment is repeated.

In practical terms, sample size planning answers a simple question: how much data do we need before we can trust the result? The answer depends on your baseline conversion rate, the minimum detectable effect you care about, your significance threshold, your desired statistical power, and how traffic is split across variants. These inputs are not just academic. They directly affect test cost, duration, and decision quality.

What sample size means in an A/B test

Sample size is the number of observations needed in each group to detect a true difference with a specified probability. In many web experiments, the observation is a user or session, and the outcome is binary: converted or did not convert. If your control converts at 5.0% and you want to detect whether a variant improves performance to 5.5%, the calculator estimates how many users are needed in the control and variant groups to reliably distinguish that difference from noise.

A good sample size estimate protects against two common problems. The first is a false positive, where random chance makes a variant look better than it really is. The second is a false negative, where a real improvement exists but the test did not collect enough data to detect it. Significance level helps control the first problem; power helps control the second.

The core inputs explained

• Baseline conversion rate: Your current conversion rate in the control experience. If your checkout converts at 4%, that baseline anchors the rest of the math.
• Minimum detectable effect (MDE): The smallest relative uplift worth detecting. If your baseline is 5% and your MDE is 10%, you are planning for a target variant rate of 5.5%.
• Significance level: Commonly set to 0.05 for a 95% confidence threshold. Lower alpha means stricter evidence and therefore larger required sample sizes.
• Power: Commonly 80% or 90%. Higher power means a better chance of detecting a true lift, but it also increases the required sample size.
• Test type: A two-tailed test checks for any difference, whether positive or negative. A one-tailed test only checks for a difference in one direction and usually requires a smaller sample, but it should only be used when a decrease is genuinely irrelevant.
• Traffic allocation: A balanced 50/50 split is generally the most efficient. Uneven allocation often requires a larger total sample to achieve the same sensitivity.

Why small uplifts require large samples

The most surprising lesson for many teams is how quickly sample size grows when the effect size gets smaller. Detecting a 30% uplift from a 5% baseline is much easier than detecting a 5% uplift from the same baseline. That is because the absolute difference between rates becomes tiny. A 5% baseline with a 5% relative uplift only moves to 5.25%, which can be difficult to separate from normal random variation unless the test collects a large amount of traffic.

This is exactly why experimentation teams should define business relevance before running the test. If a change needs six months of traffic to detect a 1% relative lift, but the business only benefits meaningfully from a 10% lift, it makes more sense to design more impactful variants instead of chasing tiny differences.

Scenario	Baseline Rate	Relative Uplift	Variant Rate	Approx. Users per Group
Email signup form test	5.0%	5%	5.25%	About 75,500
Landing page CTA test	5.0%	10%	5.50%	About 31,400
Checkout redesign test	5.0%	20%	6.00%	About 8,100
Offer framing experiment	5.0%	30%	6.50%	About 3,900

The table above illustrates a realistic pattern: as the detectable uplift shrinks, the required sample increases dramatically. This is why test strategy matters just as much as statistics. Great experimentation programs do not just run more tests. They design better hypotheses, bigger treatment differences, cleaner measurements, and stronger segmentation plans.

How confidence level and power change the result

Suppose you keep the same baseline rate and target uplift, but you increase confidence from 95% to 99%. Your sample size rises because you are asking for stronger evidence before declaring a winner. Similarly, moving from 80% power to 90% power means you want a better chance of detecting the effect if it truly exists, which also increases the number of users required.

There is no universal “best” combination. Many commercial experimentation teams use 95% confidence with 80% power because it balances rigor with practical speed. More regulated contexts, high-stakes product decisions, or experiments with substantial revenue implications may justify stricter settings. The important thing is consistency and transparency in how those thresholds are chosen.

Alpha / Power Setting	Interpretation	Relative Strictness	Typical Impact on Sample Size
0.10 alpha / 80% power	Faster, less strict planning setup	Lower	Smallest of these examples
0.05 alpha / 80% power	Common business default	Moderate	Higher than 0.10 alpha
0.05 alpha / 90% power	More protection against false negatives	High	Meaningfully larger
0.01 alpha / 90% power	Very stringent standard	Very high	Largest among these examples

The formula behind a standard proportion-based calculator

For binary outcomes such as conversion or no conversion, a common approach is the two-sample z-test for proportions. The calculator estimates required sample size from the control rate, expected treatment rate, z-scores associated with alpha and power, and the pooled variance term. In simplified notation, you can think of it as the needed sample growing when variance is high, significance requirements are strict, power is high, or the target difference between groups is small.

The effect size in conversion experiments is often represented as an absolute difference in rates. If control conversion is p1 and treatment conversion is p2, then the absolute difference is |p2 – p1|. Even when teams talk about “10% uplift,” the actual test is detecting the corresponding absolute change in conversion rate.

How to use this calculator correctly

1. Estimate your current baseline conversion rate from recent, representative data.
2. Choose the smallest uplift that would matter commercially. This becomes your MDE.
3. Select a significance level and power appropriate for the decision risk.
4. Set the expected traffic split. If possible, use 50/50 for efficiency.
5. Enter your weekly eligible traffic to estimate test duration.
6. Launch the experiment only if you can realistically reach the required sample without major seasonality or campaign shifts distorting the result.

Common mistakes when planning A/B tests

• Stopping early: Looking at the test every day and stopping when a variant appears ahead can inflate false positives.
• Using unrealistic MDE assumptions: Planning for a huge uplift just to get a smaller sample size can make the test practically useless.
• Ignoring traffic quality: Raw sessions are not the same as eligible, stable, comparable users.
• Changing the experiment midstream: Altering targeting, UX, or conversion definitions after launch weakens interpretation.
• Forgetting about business cycles: A test should often run across full weekly cycles so weekday and weekend behavior are represented.
• Not accounting for multiple comparisons: If you run many variants or inspect many metrics, false positive risk can rise.

Real-world interpretation of test duration

Sample size is not just a statistical requirement. It is also an operational constraint. If your calculator says you need 30,000 users per variant and you only get 10,000 eligible visitors per week at a 50/50 split, then each group receives about 5,000 users per week and your test may need around six weeks. That timeframe may be too long if product changes, campaigns, pricing, or seasonality are likely to shift user behavior during the experiment.

When duration is too long, teams usually have four options: increase traffic, reduce the number of concurrent variants, target a larger effect size by designing a stronger change, or postpone the test until conditions are more stable. The right answer depends on business context, not just the formula.

Why authoritative methodology matters

Experimentation decisions affect product roadmaps, budget allocation, ad spend, and customer experience. That is why it is wise to reference high-quality statistical and research sources. For broader evidence and methodology standards, review guidance from NIST.gov, survey and estimation materials from the U.S. Census Bureau, and statistical education resources from universities such as Penn State University. These sources help reinforce core ideas behind significance testing, estimation, and experimental design.

When this calculator is appropriate

This calculator is ideal for classic web and product A/B tests where the key metric is binary, such as purchase conversion, signup completion, click-through to the next step, or trial activation. It is especially useful for planning tests on landing pages, pricing pages, checkout funnels, onboarding sequences, email capture forms, and ad-to-page experiences.

For more complex metrics, such as average revenue per user, retention over long windows, or heavily skewed continuous outcomes, a different sample size method may be more appropriate. Likewise, sequential testing frameworks, Bayesian models, and adaptive experiments use different decision rules than a fixed-horizon z-test, even though the planning concepts are related.

Bottom line

An A/B test sample size calculator is not merely a convenience feature. It is a decision-quality safeguard. It forces clarity about what effect matters, how much uncertainty is acceptable, and whether the organization has enough traffic to run a valid experiment. Teams that plan sample size in advance are less likely to misread random noise as business insight. They also build healthier experimentation cultures because stakeholders understand that reliable learning requires enough data, not just fast dashboards.

If you use the calculator thoughtfully, combine it with sound experiment design, and avoid peeking or post-hoc metric changes, you will make stronger product and marketing decisions. In experimentation, rigor is not the enemy of speed. It is what keeps speed from becoming expensive guesswork.