A/B Testing: What to Test First

A/B testing has a reputation for being technical and complicated. The idea behind it is not. You show one group of visitors version A of a page, another group version B, and you keep whichever one gets more people to take the action you want. That is the whole concept.

The hard part is not running the test. It is choosing what to test. Most beginners test the wrong things in the wrong order, and end up with a tidy little experiment that teaches them nothing. This guide is about picking what to test first, so your effort actually moves the number.

If the underlying idea of raising your conversion rate is new, read conversion rate optimization for beginners first. This guide is the practical companion: what to actually change.

The short version#

Test the big things first. In order, that usually means the offer, then the headline and the promise, then the page structure and the call to action, and only much later the small cosmetic stuff like colors and fonts.

The reason is simple. A test can only reveal a difference that exists. If you test two button colors, the biggest possible win is the small gap between two button colors, which is usually tiny. If you test two genuinely different offers or two genuinely different headlines, the possible win is large, because those things actually change whether people say yes.

Beginners get this backwards. They start with the safe, easy, low-stakes change, because it is easy to make, and then wonder why the "winner" barely moved anything. Changing your button color is rarely your biggest problem. Start where the money is.

How an A/B test actually works#

              +------------------+
   Traffic -> | Split in half    |
              +------------------+
                 |            |
                 v            v
            Version A     Version B
          (the current) (one change)
                 |            |
                 v            v
           Conversion    Conversion
             rate A        rate B
                 |            |
                 +-----+------+
                       v
             Keep whichever won,
             if the gap is real

The rules that keep a test honest:

• Change one thing at a time. If A and B differ in five ways, a win tells you nothing about which change caused it. One variable per test.
• Split the traffic fairly and at the same time. Do not run A this week and B next week. Seasons, days, and traffic sources shift, and that shift can masquerade as a result.
• Wait for enough visitors. A percentage is not trustworthy on tiny numbers. If 4 people see A and 1 buys, that is not a "25% conversion rate," it is noise.

That last point is where most beginner tests fall apart. On low traffic, a difference between two versions can be pure luck. You need real volume before you believe the winner. If your traffic is thin, test the biggest possible changes, because only a large effect will show up clearly on small numbers.

The priority order: what to test first#

Here is the order that actually moves your conversion rate, from highest leverage to lowest.

1. The offer#

This is the biggest lever by a wide margin, and almost nobody tests it. What are you actually offering, at what price, with what guarantee? Two different offers can convert wildly differently even on the identical page. Before you touch design, ask whether people even want the thing at that price. If the offer is wrong, no headline or button saves it.

2. The headline and the core promise#

The headline is the first thing a visitor reads and it decides whether they keep reading at all. Testing two genuinely different promises, not two reworded versions of the same one, can produce big swings. A promise stated in the reader's own words usually beats a clever one that makes them think.

3. The angle: features vs benefits#

How you frame the offer matters as much as the offer itself. A page that lists what the product has often loses to one that spells out what it does for the reader. This is worth a dedicated test, and features vs benefits explains why the framing changes the result.

4. The call to action#

Test what the button says and where it sits. "Get the guide" against "Submit." The action stated plainly against a vague one. This can move the number more than people expect, because a confused visitor who cannot tell what happens next often does nothing.

5. Page structure#

Removing distractions, cutting the menu, shortening a form, reordering the page so the offer appears sooner. These structural changes affect how many people make it to the decision at all.

6. Cosmetic details (last)#

Colors, fonts, images, spacing. These are real, but they are the smallest lever, and testing them first is the classic beginner trap. Get here only after the big things are settled.

A simple example with numbers (hypothetical)#

These numbers are invented to show why order matters, not to promise a result.

You have a page converting at 2 percent. You decide to run one test. Compare two choices.

Test 1: two button colors (a small-lever test)

Version A (blue button):   2.0% conversion
Version B (green button):  2.1% conversion
Best possible gain:        +0.1 points -> barely anything

Test 2: two genuinely different headlines (a big-lever test)

Version A (vague promise):    2.0% conversion
Version B (specific promise): 3.2% conversion
Gain:                         +1.2 points -> a 60% lift in sales

Say you send 1,000 visitors to a $50 offer. The button test moves you from 20 sales to 21, an extra $50. The headline test moves you from 20 sales to 32, an extra $600. Same traffic, same effort to run the test. The only difference is that one of them tested something that could actually move.

That is the entire argument for testing the big things first. The gain is capped by how much the thing you tested can matter, so test the things that can matter a lot.

What you need#

• Two versions that differ in exactly one meaningful way.
• A way to split traffic and measure the conversion rate of each. Many page builders and email tools include this.
• Enough traffic that the result is not luck.
• The discipline to test one thing at a time.

What it costs#

• Required: usually nothing beyond the tools you already run traffic with.
• Optional: a dedicated A/B testing or split test tool if your builder does not include one.
• Nice to have: more traffic, because it lets you trust smaller differences sooner.

How long it takes#

A test runs until enough visitors have seen both versions for the result to be trustworthy. On healthy traffic that might be days. On thin traffic it can be weeks, which is exactly why beginners on low traffic should test only the biggest changes: a large effect shows up fast, a tiny one may never show up clearly at all.

What beginners usually get wrong#

• Testing colors and fonts first. The easiest change to make is rarely the one worth making.
• Testing two nearly identical versions. If A and B are basically the same, the winner is meaningless.
• Calling a winner too early. A lead on 15 visitors is noise. Wait for volume.
• Changing several things at once. Then you cannot tell what caused the result.
• Testing a page that should not get traffic at all. If the offer is wrong or the traffic is mismatched, testing the page is polishing the wrong thing.
• Scaling on a fake win. Pouring more traffic behind a "winner" that was really luck just spends money faster, which ties back to why more traffic isn't always more money.

How I would start#

1. Confirm the offer and the traffic match are actually right before testing anything on the page.
2. Test the headline and core promise first, using two genuinely different versions.
3. Change one thing, split traffic fairly, and wait for real volume.
4. Keep the winner only if the gap is large enough to trust, then move down the priority list.
5. Save colors and fonts for last, once the big levers are settled.

What I would not do#

I would not start with button colors, and I would not trust a result built on a handful of visitors. And I would not run a test at all on an offer nobody wants, because the best possible outcome is still a page that fails. To keep the whole optimization habit in perspective, read conversion rate optimization for beginners, and to understand the number every test is really moving, read conversion rate explained.