What is A/B testing?

A/B testing (also called split testing) is an experimentation method where two versions of a webpage, email, ad, or product feature are shown to randomly assigned user groups, with the goal of measuring which version performs better against a defined metric (clicks, conversions, signups, revenue).

According to a 2024 Optimizely benchmark report, top-performing SaaS teams run 5-10 A/B tests per quarter and achieve cumulative 20-40% conversion lifts over a 12-month period. A/B testing is the foundational tool of modern conversion rate optimization (CRO) and growth-hacking practices.

How A/B testing works

A standard A/B test process:

1. Hypothesis — "Changing the CTA from 'Sign Up' to 'Start Free Trial' will lift signup conversion"
2. Variant creation — build the new version (B) alongside the original (A)
3. Random assignment — split incoming users 50/50 (or other ratios)
4. Measurement — track the conversion metric for each variant
5. Statistical significance — wait until results reach 95%+ confidence
6. Decision — implement the winner, document the learning

The math behind A/B testing relies on statistical significance: the probability that the observed difference is real, not due to chance. Most platforms aim for 95% confidence, meaning there's only a 5% chance the result is noise.

According to a 2023 Convert.com analysis of 1,000+ A/B tests, only 25-30% of tests show statistically significant winners. The remaining 70-75% are inconclusive or show losses. The discipline is the iteration, not any single experiment.

Common A/B test types:

• Landing page tests — headlines, CTAs, hero images, form length
• Email tests — subject lines, send times, content layout
• Ad tests — copy, creative, audience targeting
• Pricing tests — price points, tier structures, billing frequency
• Onboarding tests — flow steps, copy, defaults
• Product feature tests — UX changes measured against engagement metrics

A/B testing requires sufficient traffic to reach significance. Pages with under 1,000 visitors per variant per week typically take too long to test reliably.

Examples of A/B testing in practice

Example 1: Booking.com's massive testing program

Booking.com runs over 1,000 A/B tests concurrently across its site. The company has been called the "world's largest A/B testing operation" and credits the practice with sustained 20%+ year-over-year conversion improvements.

Example 2: Obama 2008 presidential campaign

Dan Siroker (later founder of Optimizely) led A/B testing for the Obama 2008 campaign's email signup pages. Tests on the homepage video and CTA copy lifted signups 40%+, contributing $60M+ in additional fundraising.

Example 3: Solopreneur SaaS founder

A solo SaaS founder runs monthly A/B tests on the pricing page. Testing $19 vs $29 starter tier reveals $19 produces 2x signups but $29 produces 50% higher revenue per visitor. The founder lands on $29 as the optimal price.

When to use A/B testing

Use A/B testing when:

• You have enough traffic for significance (typically 1k+ visits per variant)
• You can articulate a clear hypothesis
• You have a defined success metric (conversion, click, signup)
• The change is meaningful enough to potentially move the metric
• You can build variants without major engineering cost
• You're prioritizing data-driven decisions over opinion

When NOT to A/B test

• Low traffic pages — Tests take months to reach significance
• Brand-defining changes — A/B testing brand assets can damage long-term equity
• Tiny changes with low expected impact — Test more meaningful variants
• Pre-PMF — Don't optimize a product that hasn't yet found fit

A/B testing vs related concepts

A/B is the simplest. Multivariate is more powerful but requires more traffic. Bandits adapt traffic allocation as results come in.

Common mistakes with A/B testing

• Stopping tests too early — Premature stopping inflates apparent significance.
• Testing too many variables at once — Without isolation, you can't attribute the lift.
• Ignoring statistical significance — A 3% lift at 60% confidence isn't real.
• Testing trivial changes — Button color tests rarely move metrics meaningfully.
• No documentation system — Lessons from failed tests get lost.

Frequently asked questions about A/B testing

What is the difference between A/B testing and multivariate testing? A/B testing compares two variants of one element (e.g. CTA copy). Multivariate testing compares combinations of multiple elements simultaneously (e.g. headline × image × CTA combinations). A/B is simpler and faster; multivariate gives richer insight but requires more traffic.

How long should an A/B test run? Until statistical significance is reached, typically 1-4 weeks for high-traffic pages. Run for at least one full business cycle (usually a week) to avoid day-of-week bias. Don't stop prematurely just because results "look" good.

How do I implement A/B testing? Pick a high-traffic page or flow. Identify a clear hypothesis. Build variant A (control) and variant B (test). Use a testing platform (Optimizely, VWO, Google Optimize successor). Wait for significance. Implement the winner. Document the learning.

What tools support A/B testing? Optimizely, VWO, AB Tasty, Convert.com for general web testing. Statsig and LaunchDarkly for product feature testing. Mailchimp, ConvertKit for email A/B tests. Meta Ads Manager and Google Ads for ad tests.

Can A/B testing be automated? Largely, yes. Platforms automate variant assignment, measurement, and significance calculation. Multi-armed bandit algorithms automatically shift traffic to winning variants over time. AI-driven personalization extends A/B testing into per-user variant selection.

Why do most A/B tests fail? Hypothesis quality varies; many tests are minor cosmetic changes that can't move metrics. Statistical noise dominates small samples. The discipline is in the iteration: even a 25% win rate compounds significantly over 50-100 tests per year.

How PostKit uses A/B testing

PostKit's product team runs A/B tests on landing page variants, pricing tier presentations, and onboarding flows. Tests are tracked in the team's growth log and prioritized via ICE scoring. Founder Tadeáš Raška has shared specific test results in build-in-public posts (e.g. testing "Start free trial" vs "Try it free" CTA copy). Future plans include in-product A/B tests for content generation flows.

Related glossary terms

• Multivariate testing — More complex testing methodology
• Conversion rate optimization (CRO) — Discipline that uses A/B testing
• Growth hacking — Strategy that depends on A/B testing
• Landing page — Common A/B test surface
• Actionable metrics — What A/B tests should measure

Sources

• Optimizely benchmark research
• Convert.com A/B testing analysis
• Booking.com testing case studies