A/B Test Sample Size Calculator

Sample size in A/B testing is the number of visitors (or users) needed in each test variation to achieve statistically reliable results. Running a test with too few visitors leads to inconclusive or misleading results, while too many visitors wastes time that could be spent running additional tests.

The required sample size depends on four key factors: your baseline conversion rate, the minimum effect you want to detect (MDE), your desired statistical significance (typically 95%), and statistical power (typically 80%). A 5% baseline conversion with 10% MDE requires roughly 31,000 visitors per variation at standard settings.

This calculator uses the two-proportion z-test formula, the standard method for comparing conversion rates between two independent groups. It accounts for the variance in both control and treatment groups to determine the sample needed to detect your specified effect.

Start with business impact. Ask yourself: what's the smallest improvement that would matter? If a 5% lift in conversion translates to $10,000/month in revenue, that's worth detecting. If it only means $100/month, you probably need to detect larger effects to justify the testing time.

Consider your traffic constraints. Lower MDEs require exponentially more traffic. Detecting a 5% relative improvement needs about 4x the sample size of detecting a 10% improvement. If you only have 1,000 daily visitors, targeting 20-30% MDEs is more realistic than 5%.

The 10-20% rule of thumb. Most practitioners use 10-20% relative MDE as a reasonable balance. This catches meaningful improvements without requiring months of testing. For high-traffic sites, 5-10% MDE becomes feasible. For low-traffic sites, 20-30% may be necessary.

Peeking at results early. The single most common mistake. Looking at results before reaching your calculated sample size and stopping when you see “significance” inflates your false positive rate from 5% to as high as 30%. Wait for your full sample or use sequential testing methods.

Running tests for calendar time, not sample size.“Let's run this for two weeks” ignores whether you've reached adequate sample. Use this calculator to determine how many visitors you need, then calculate how long that takes at your traffic levels. The test ends when you hit sample, not a date.

Testing too many variations. Each additional variation multiplies your required sample. A 4-variation test needs 4x the traffic of an A/B test, and requires correction for multiple comparisons. Start with simple A/B tests; only add variations when you have the traffic to support them.

Ignoring novelty effects. New designs often win initially because they're different, not better. Run tests for at least one full business cycle (typically 1-2 weeks minimum) to ensure you're not just measuring the excitement of change.

Focus on high-impact pages. Test your highest-traffic pages first: homepage, pricing page, main landing pages. These give you the fastest path to significant results. Save low-traffic pages for after you've optimized the high-traffic ones.

Test bigger changes. Small tweaks like button colors need huge samples to detect. Test substantial changes - different value propositions, page layouts, pricing structures - that could produce 20-50% lifts. These require smaller samples and teach you more about your customers.

Use Bayesian methods for faster decisions.Traditional frequentist statistics (what this calculator uses) require fixed sample sizes. Bayesian methods let you monitor results continuously and stop when you have sufficient confidence. Tools like Optimizely use this approach for low-traffic scenarios.

Consider qualitative research instead.If a test would take 3+ months, user interviews and usability testing might give you faster insights. Talk to 10 customers about their experience, watch session recordings, and make informed decisions without waiting for statistical significance.