SEO A/B Testing: How Split Tests Work, and What Smaller Sites Do

SEO A/B testing splits a set of similar pages into two groups, changes one group, and compares how the two groups do in search over the same weeks. It splits pages rather than visitors, because Google sees one version of each URL. It works well on a large site with hundreds of pages on one template. On a smaller site the same question gets answered differently: change one page, then judge it against the rest of the site you did not touch.

TL;DR:

• A conversion A/B test splits visitors. An SEO test splits pages, because "there is only one Googlebot."
• Split testing needs scale. SearchPilot names "at least hundreds of pages on the same template" and "at least 30,000 organic sessions per month" to the tested group.
• Google allows testing with rules: no cloaking, rel="canonical" on alternate URLs, 302 redirects rather than 301, and the test removed once it ends.
• Below that scale, test one change at a time and compare it with the untouched rest of the site over the same window, starting from Google's recrawl.

How is an SEO A/B test different from a normal A/B test?

A normal A/B test shows half your visitors version A and half version B, then compares what they do. That cannot work for search, because the visitor that decides your ranking is Google's crawler. SearchPilot puts it directly: "The 'user' we are testing for is Googlebot, not human users," and "there is only one Googlebot." Each URL can only have one version in Google's index.

So an SEO test splits pages instead. You take a set of pages built on the same template, such as product pages or city pages, and divide them into a control group and a variant group. The change goes onto the variant pages only. Then you compare the search traffic of the two groups over the same weeks.

How do you set up an SEO split test?

Four steps:

1. Pick one template. The pages have to be alike, or the groups will differ for reasons that have nothing to do with your change.
2. Build two matched groups. SearchPilot's criteria are that "Both buckets should have similar levels of traffic" and "should be statistically similar to each other", meaning they "should trend up and down at similar times."
3. Change the variant group only. A new title pattern, a new block of copy, added structured data. One change per test.
4. Compare the groups over the same period. Both groups face the same algorithm updates, seasons and news, so the difference between them is the part your change is responsible for. SearchPilot's model forecasts each group from history, "usually about 100 days of data."

What are Google's rules for SEO testing?

Google publishes them, and they come down to not misleading the crawler.

• No cloaking. "Don't show one set of URLs to Googlebot, and a different set to humans."
• Canonical, not noindex. On alternate URLs, use rel="canonical" pointing at the original, because "Using noindex rather than rel="canonical" in such a situation can sometimes have unexpected bad effects."
• Temporary redirects. If a test redirects, "use a 302 (temporary) redirect, not a 301 (permanent) redirect."
• Finish the test. Once it ends, "remove all elements of the test as soon as possible." Google warns that an experiment run "for an unnecessarily long time" may be read "as an attempt to deceive search engines."

Google also notes that small design changes, "such as the size, color, or placement of a button or image", "often have little or no impact on that page's search result snippet or ranking." SEO tests are for changes Google reads: titles, headings, copy, links and markup.

How big does a site need to be for split testing?

Bigger than most. SearchPilot, which sells split testing, says a test needs "at least hundreds of pages on the same template" and "at least 30,000 organic sessions per month to the group of pages you want to test on." Smaller sections can be tested, but "the changes in traffic need to be much higher to be able to reach statistical significance."

That rules out most sites. A SaaS site with a pricing page, a dozen feature pages and a blog does not have hundreds of pages on one template, and splitting twelve feature pages into two groups of six gives you two groups that move for their own reasons.

How do you test an SEO change on a smaller site?

Test it over time instead of across pages, and borrow the split test's best idea: a comparison that faced the same weeks.

1. Change one page, for one search. Write down what you changed and what you expect to happen before you ship it.
2. Start the clock when Google recrawls the page, not when you deploy. Until Google has read the new version, the old one is what ranks.
3. Compare against what you did not touch. Rankings move on pages nobody edited. Measure the change on your page, then take out what the rest of the site did over the same window. What is left is a fair reading of the change.
4. Do not edit the page again until the reading is in. A second change inside the window mixes into the first, and neither can be measured.

This is weaker evidence than a large split test. One page is noisy. But a change judged against the site's own movement, from the day Google read it, is far better than a before-and-after screenshot, which credits your edit with every update and season that happened in between. The guide to tracking SEO changes covers the waiting windows, and how to measure SEO impact covers the reading.

That is how QueryScope measures every change your coding agent records. It reads your own Search Console, starts each change's clock at Google's recrawl, takes out what the rest of the site did over the same weeks, and does not hand the page back for another change while the first is still settling. It measures search traffic and stops at the click.

Sources

• Google Search Central, Minimize A/B testing impact in Google Search ("Don't show one set of URLs to Googlebot, and a different set to humans"; "Using noindex rather than rel="canonical" in such a situation can sometimes have unexpected bad effects"; "use a 302 (temporary) redirect, not a 301 (permanent) redirect"; "remove all elements of the test as soon as possible"; experiments run "for an unnecessarily long time" may be read as "an attempt to deceive search engines"; small changes "such as the size, color, or placement of a button or image" "often have little or no impact on that page's search result snippet or ranking").
• SearchPilot, What is SEO A/B testing? ("The 'user' we are testing for is Googlebot, not human users"; "there is only one Googlebot"; "Both buckets should have similar levels of traffic"; buckets "should trend up and down at similar times"; "usually about 100 days of data"; "at least hundreds of pages on the same template"; "at least 30,000 organic sessions per month to the group of pages you want to test on"; on smaller sections "the changes in traffic need to be much higher to be able to reach statistical significance").