If your landing page has little traffic, start with a question people can answer through a real task. Can a suitable reader explain the offer? Can they find the information they need? Can they complete the next step? You do not need to declare an A/B winner to investigate those problems.
An A/B experiment asks a different question: how do assigned versions compare on a defined outcome under a suitable design? A few completed actions can produce a large-looking percentage difference without providing enough evidence for a confident decision.
This guide helps you choose between message checks, usability sessions and a conversion experiment. Use the Test Planner to record the question and limits. The goal is a useful next investigation, not a green score that claims your homepage is proven to sell.
Start with the uncertainty, not the testing software
A business owner saying “the page does not work” may mean several things. The offer may be unclear, the audience may be wrong, the form may fail or the service may simply be unsuitable for the visitors arriving. These are different problems.
Write one uncertainty in plain words. For example: “We do not know whether freelance designers understand that this is a monthly bookkeeping service.” Another question is: “We do not know whether mobile visitors can submit the enquiry.” Neither is answered merely by comparing two button click rates.
A good first test isolates a decision the team can act on. It does not need to solve the entire business model. Record what evidence would help choose a next step, then select the method that can actually produce it.
If the underlying task remains unclear, start with customer discovery interviews before showing a draft. Asking people to judge copy for a problem they have never faced may generate opinions without resolving your uncertainty.
Distinguish three testing jobs
| Question | Suitable starting method | What the result can describe |
|---|---|---|
| What do relevant readers understand? | A message or brief-exposure check | Responses to the tested draft under stated conditions |
| Can someone complete a realistic task? | An observed usability session | Task behavior, obstacles and expectations |
| Does an assigned version change a defined outcome? | A properly designed experiment | A comparison with uncertainty and design limits |
These methods can support one another. A comprehension problem can inform a revision that later enters a conversion experiment. A conversion difference can motivate research into why users behaved differently. The methods do not become interchangeable because each is called testing.
Keep the evidence labels in the report. A participant response is not a purchase. A completed task in a facilitated session is not a population conversion rate. A platform's estimated score is not an independently observed customer outcome.
Know what an A/B test actually requires
Google's A/B test definition describes random assignment to concurrent page variants and evaluation against a specific goal. This is different from showing version A this week and version B next week without accounting for other changes.
Concurrent randomization helps make groups comparable on average under the design. It does not repair broken event collection, an incorrect eligibility rule or a page that accidentally sends returning visitors between inconsistent variants. The implementation is part of the experiment.
Google also states that running these experiments with Analytics requires an integrated third-party experiment tool. An ordinary analytics report is not automatically an assignment system. A local draft-comparison workspace is not one either unless it actually supplies and validates those capabilities.
Use provider documentation to check the current product's behavior. Do not buy a tool merely because its sales page contains the phrase A/B testing. Ask whether the required assignment, outcome collection, limits and reporting fit your actual task.
Fix clear failures before treating them as a contest
A contact form that loses input or never reaches the inbox needs investigation and repair. A page promising a free example but opening an undisclosed payment step has a mismatch worth addressing. You do not need to preserve a broken experience until it loses an experiment.
Verify the failure with an appropriate test and save a record. Explain what was observed, where it occurred and how the repair was checked. That is a legitimate engineering or usability improvement without claiming a measured conversion lift.
Check mobile input, keyboard access, error recovery and confirmation wording. A successful-looking browser message should correspond to an actual received request when receipt is the promised outcome. A form-start event does not prove that handoff happened.
After the fix, keep the claim bounded: the reproduced defect no longer occurs under the tested conditions. Whether the change improves enquiries or sales is a separate question requiring business evidence.
Use a message check for a specific misunderstanding
Show the actual draft to people who plausibly need the offer. Ask what is offered, who it serves and what they expect to happen next. Record their words before explaining your intended meaning.
A brief-exposure exercise can investigate first impressions or recall. Lyssna's five-second testing documentation describes this established approach and follow-up questions. Five seconds is a configurable research condition, not proof that every person's attention expires at that point.
Choose a duration and format appropriate to the question and participant's access needs. If the task requires careful comparison of a professional service, a longer reading exercise may be more useful. Do not force an unsuitable timed visual method merely because it produces a tidy score.
The Reader Check helps record responses from readers you recruit yourself. It does not supply a representative panel. Our reader-feedback guide provides the detailed protocol; the decision here is when that method answers the right question.
Use a usability session for the actual journey
Ask a suitable person to attempt a realistic task, such as checking service coverage and finding how to request help. Avoid telling them the exact navigation path if finding that path is part of the question.
The GOV.UK moderated-usability guidance describes observing participants complete specific tasks. The useful evidence includes hesitation, wrong turns, unavailable information and what the person expected at each stage.
Keep ability separate from willingness. Someone who can find the price and decides the service is unsuitable has not necessarily failed the task. Someone who likes the design but cannot find a supported contact route has supplied a different finding.
Record the participant context, draft version, task and observed result. A handful of sessions can reveal a concrete problem without estimating how often it occurs across all visitors. Do not publish a representative completion rate from a convenience group.
Work through a small-count example without inventing a winner
Consider chosen demonstration counts, not an experiment conducted by Plain Reckoner. Version A records two completed actions from 100 eligible visits. Version B records three from 100. The observed rates are 2% and 3%.
The absolute difference is one percentage point. The relative difference compared with A is 50%, because one divided by two is one half. A headline saying “50% improvement” can therefore describe only one additional recorded action in this example.
That arithmetic does not establish that B will keep performing better. The comparison still needs a valid design, enough evidence for the question and an appropriate uncertainty analysis. A spreadsheet can calculate a difference without making the conclusion reliable.
Do not label the example statistically significant or use it to prescribe a universal sample size. The amount of evidence needed depends on the baseline, meaningful effect, design and decision standard. Those inputs need explicit review before launching a serious conversion experiment.
Define the outcome before reading the result
A contact-button click, successful message receipt, qualified enquiry and sale are different stages. Choose the primary outcome that reflects the question, then preserve useful intermediate measures for diagnosis.
Suppose a chosen example gives A ten suitable enquiries and B twelve, but B also generates many more irrelevant requests and much more handling work. The extra count does not automatically establish a better business outcome. Qualification and cost definitions matter.
Do not change the target after viewing results merely because another metric favors the version the team already likes. Record any legitimate change of question and treat it as a new decision rather than presenting it as the original test's success.
For a free tool, a completed useful result may be more relevant than an email capture. For a service, successful receipt may be the immediate outcome while later business value needs a longer window. Keep the units and time periods explicit.
Avoid the before-and-after trap
Publishing a rewrite on Monday and seeing more requests on Tuesday is an observation, not by itself a controlled experiment. Traffic sources, weekday patterns, demand, offers and operational changes may also differ.
A before-and-after record can still be useful when randomization is unavailable. Save the baseline, dates, page versions and concurrent changes. Describe the result as observational and keep plausible alternative explanations visible.
Do not turn a seasonal rise into proof of a headline effect. Do not call every fall a failed message. Inspect the actual journey and the scope of the data before choosing a causal story.
Our two-endpoint guide explains another limitation: two values do not reveal the path between them. Intermediate observations and measurement changes can matter as much as the final percentage difference.
Do not confuse adaptive routing with a fixed experiment
Unbounce's current testing documentation distinguishes its A/B mode from Smart Traffic, which uses machine learning to route visitors. Those descriptions identify different product behaviors; they are not independent proof of which method is best for your business.
If allocation changes based on visitor characteristics or observed outcomes, interpretation needs to account for that method. Do not analyze every platform's variant totals as though they came from the same fixed randomized design.
Check which mode is active, what is being optimized and how results are reported. A platform label such as champion should not replace a review of the outcome definition and evidence. Vendor interfaces can make a result look decisive without answering every business question.
Plain Reckoner's draft and reader tools help prepare and record an investigation. They do not automatically deploy randomized variants or certify the inference from another platform's adaptive system. Keep those product boundaries visible.
Decide when low traffic changes the next step
Low traffic is not one universal monthly number. The useful question is whether the relevant visitors and completed outcomes can support the intended comparison within a meaningful period. Rare outcomes and small worthwhile differences can demand more evidence than a quick dashboard check suggests.
If a suitable experiment would take too long, change the research question rather than lower the evidence standard without saying so. Investigate comprehension, suitability or a reproducible task obstacle first. These methods can identify improvements that deserve a later experiment.
Do not claim that qualitative work proves a conversion winner. Its value is explaining a specific problem and generating a more credible change. It can also reveal that the proposed change addresses the wrong issue.
Use the Test Planner to record why the method was chosen and what remains unknown. An honest inconclusive result is preferable to a green winner badge produced from a few events.
Turn a book idea into a testable change
A customer-centered writing framework can suggest that the page should explain the visitor's task, the business's help and a clear next action. Those ideas can guide the draft, but they do not supply actual readers' answers or prove an outcome.
For an original repair-service example, replace an unclear company-first introduction with a concrete service and coverage statement. Before claiming a commercial effect, ask suitable readers what they believe is offered and observe whether they can request the relevant help.
Use the Offer Builder and Homepage Planner to keep the explanation coherent. Work on the visual draft in the page editor, then save the version you actually test. A visually polished update should not inherit evidence collected on an earlier message.
For optional further reading, Donald Miller's <a href="https://link.amazon/B08rUqegA" rel="sponsored noopener noreferrer">Building a StoryBrand 2.0</a> discusses customer-centered messaging. This is an affiliate link: we may earn a commission if you buy. As an Amazon Associate I earn from qualifying purchases. Plain Reckoner is independent and is not endorsed by the author. The book is not required to use the tools or this protocol.
Apply the same rule to a SaaS or store page
An existing SaaS business may want more trials, more paying customers or a new audience for the same product. Start by naming which of these jobs the page should do. A rise in free sign-ups does not answer whether more people stayed, paid or needed support.
If you are testing a new use for an existing product, keep the new page and the old offer separate in your notes. Ask people from the new audience to describe the problem and try the intended task. Feedback from current customers may be helpful, but it does not automatically represent the new group you want to reach.
For a store, an add-to-cart action is a step toward a purchase, not a completed profitable order. Keep completed purchases, refunds and selling costs separate when you later check the business result. The people who arrive, the product and the observation period all matter.
These are different goals, but the same habit helps: write down the question before changing the page. Save the version people saw. Record what happened without upgrading a click into a sale. Our free business plan helps you choose a starting toolkit while keeping those unknowns visible.
Write a decision record before the next round
Keep one short record: question, audience, draft version, method, primary outcome, recruitment or allocation, observation period, known limits and next decision. Add what would cause the team to stop or seek a better design before collecting more data.
Do not repeatedly inspect a conventional experiment and stop the moment the preferred version appears ahead unless the method explicitly supports that decision rule. Use an appropriately reviewed analysis and stopping approach rather than inventing one from a live chart.
Separate safety or correctness stops from a success declaration. A broken form should be stopped or repaired even if no statistical conclusion exists. Ending a harmful implementation is not the same as proving an alternative wins commercially.
The useful result of low-traffic landing-page testing is often a narrower, better-supported next step. Find what readers understand, remove concrete obstacles and save the evidence. Run a conversion experiment when the question and conditions support it, not simply because the page builder offers a split-test button.