The wrong way to size a test
Most test budgets are set by feel. Someone suggests a number that sounds bearable, the campaign runs for a week, results are mixed, and the conversation ends in opinion rather than evidence. Then the same idea gets retested six months later because nobody ever settled it.
A test is a purchase of information. Size it by the information you need.
Step 1: Decide what the test must answer
Write the question in one sentence, and make it a decision, not a curiosity.
Good: "Does a free site-visit offer produce qualified enquiries at a lower cost than our current quote-request offer?"
Bad: "Let us try some new creative and see."
If you cannot state what you will do differently depending on the answer, do not run the test.
Step 2: Work out how many conversions you need
You do not need academic rigour, but you do need enough events that random variation cannot explain your conclusion. As a practical rule for SME budgets, aim for a few dozen conversions per variant before drawing conclusions, and treat anything under a dozen as noise.
If your decision is between two very similar options, you will need far more data to separate them — which is itself a signal that the difference does not matter much.
Step 3: Estimate cost per conversion
Use your existing account data. If your current cost per qualified enquiry is a known number, use it as the base and assume the new offer may perform somewhat worse initially while the platform learns.
Test budget equals conversions needed multiplied by expected cost per conversion, plus a buffer of roughly a quarter for the learning period.
Step 4: Set the rules before launch
Write these down before spending anything:
- Success threshold: what result would make you adopt the new approach.
- Kill threshold: what result at what spend makes you stop early.
- Decision date: when you decide regardless.
- Owner: who makes the call.
The kill threshold prevents the most expensive failure mode — a test that limps along for months because nobody wants to declare it dead.
Step 5: Isolate the variable
Test one thing. If you change the offer, the creative, and the landing page simultaneously and results improve, you have learned that the bundle works, which tells you nothing reusable.
Order of testing that usually produces the most learning per rupee:
- Offer
- Audience or intent level
- Landing page structure
- Creative execution
- Format and placement
Offer changes produce the largest swings, which is why they should come first when budget is scarce.
Step 6: Protect the control
Always keep the current best performer running alongside the test. Without a live control, you cannot separate the test's effect from a market shift, a seasonal change, or a competitor's campaign.
Split budget so both have enough volume. A test starved of budget to protect the control produces an unreadable result.
What to do with the answer
Clear win: scale gradually, not instantly. Doubling budget overnight often degrades performance while the platform re-learns. Step up in increments and watch cost per qualified lead at each level.
Clear loss: document what you learned about why, and do not retest the same idea in six months without changing something structural.
Ambiguous: treat as a loss for now. Ambiguity at your scale means the effect is too small to matter, even if it is real.
Budgeting for a testing programme, not a test
Single tests answer single questions. A testing programme compounds.
Reserve a fixed slice of monthly spend — ten to twenty percent is a common working range — and run one structured test at a time rather than several half-funded ones. Keep a simple log: hypothesis, spend, result, decision, date. After a year the log is more valuable than any individual result, because it maps what your market responds to.
The most common mistakes
- Testing during peak season, when unusual demand contaminates the result.
- Judging on cost per lead rather than cost per qualified lead.
- Stopping a test three days in because early numbers looked bad.
- Running tests without a control.
- Never writing down the outcome, so the same debate recurs annually.
The underlying principle
Testing is not about being scientific for its own sake. It is about converting money into decisions that stay decided. A properly sized test with pre-agreed thresholds settles a question permanently. An underfunded one buys an argument you will have again.
