An A/B test sends two versions of an email to similar groups and compares the result. Done well, it teaches you what your audience likes. Done badly, it produces confident but wrong conclusions.
Change one thing at a time
Swap the headline, the picture and the button all at once and the result tells you nothing about which one worked. Choose a single variable for each test.
What to test first
- Subject line: question versus statement, short versus long.
- Preview text, which many senders forget.
- The main call to action: wording and position.
- Send time and day.
- Plain layout versus a designed layout.
Decide what success means
For subject lines, look at opens. For content and buttons, look at clicks and, even better, sales or bookings. A higher open rate that does not lead to action is not a win.
Give the test enough time and people
Very small lists produce noisy results. Let the test run until both versions have been seen by a reasonable number of people, and avoid declaring a winner after an hour.
Keep a testing log
Write down what you tested, the result and what you will do next. After a few months you will have a playbook of your own.
How big should your test be?
A test needs enough people to separate real differences from luck. There is no universal number, but the principle is simple: the smaller the difference you hope to detect, the more recipients you need. If your list is small, test bigger changes, run the test across several sends or combine similar emails before drawing conclusions.
How long to run a test
Most opens and clicks happen within the first day or two. Wait until the activity has settled before choosing a winner, and avoid ending tests early just because one version is ahead after an hour.
Test ideas with a hypothesis
Write a short hypothesis before each test so you learn something even when it fails.
- 'A subject line that names the benefit will get more opens than one that names the product.'
- 'A single button will get more clicks than three links.'
- 'Putting the offer in the first paragraph will increase sales.'
- 'Sending at 8pm will beat 10am for this audience.'
- 'A plain text style email will feel more personal and get more replies.'
Reading results without fooling yourself
- Compare the metric that matches your goal: opens for subject lines, clicks for content, revenue for offers.
- A small difference with a small audience can be luck. If the result is close, treat it as a tie and test again.
- Watch for outside factors such as a holiday or a news event that affected one send.
- Do not test five things and then claim to know why one version won.
Testing automated flows
Campaign tests give quick answers, but flows such as welcome sequences and abandoned cart emails run for months, so improving them has a lasting effect. Test one step at a time, for example the subject line of email one or the delay before the reminder.
Arabic and English tests
If your list is bilingual, test whether language preference affects results. Check that Arabic emails display correctly right to left, with natural wording rather than literal translation.
Common mistakes
- Changing several things at once.
- Declaring a winner too early.
- Ignoring deliverability problems that affect one group more than the other.
- Not writing down results, then repeating old tests.
A six-week testing calendar
- Week 1: subject line style.
- Week 2: preview text.
- Week 3: button wording and position.
- Week 4: send time.
- Week 5: long versus short content.
- Week 6: review all results and update your templates.
A worked example
A boutique wants more people to open its weekly offers email. It splits its list in two. Version A has the subject line 'New arrivals this week'. Version B has 'Three new pieces we think you will love'. Both are sent at the same time on the same day. After two days, version B has a higher open rate and more clicks. The team keeps the more specific, personal style for the next four emails and tests a new element, the button wording, in the following week.
Subject line patterns to test
- Benefit-led: 'Save time on your weekly shop'.
- Curiosity: 'The one change that doubled our bookings'.
- Urgency, only when genuine: 'Ends tonight'.
- Personal: including the first name or a recent purchase.
- Plain and direct: 'Your invoice for March'.
Significance in plain language
Statistical significance is a way to judge whether a difference is likely to be real or just chance. Many email tools show a confidence figure. A higher confidence means the result is more likely to hold if you repeat the test. If the tool says the result is not clear, accept it as a draw and move on to a different test.
Setup checklist
- Define the goal and the metric.
- Choose one variable.
- Split the list randomly into equal groups.
- Send both versions at the same time.
- Wait long enough for results to settle.
- Record the outcome and the next step.
Frequently asked questions
How often should I test?
Run at least one test per month, or one per important campaign.
Can I test on a tiny list?
You can, but treat results as clues. Make bigger changes and combine learnings across several sends.
Our email marketing service includes testing, design and reporting, in English and Arabic.
Want help putting this into practice?
Book a free audit and we will review your marketing and suggest the quickest wins.
