The rules you set may not be the rules Google follows
You added a few lines to your robots.txt file to keep certain pages out of Google. Your internal search results. Your cart and checkout pages. A staging folder your developer left behind. Months later, you search your own business and there they are, sitting in Google right next to your real service pages.
You did everything right. The rules are still there. So why is Google ignoring them?
There is a quirk in how robots.txt works that catches out even experienced developers. Google’s John Mueller confirmed it again in July 2026, answering a site owner whose internal search pages kept getting indexed despite a clear rule telling crawlers to stay out. The cause was not a Google bug. It was the way the file was written.
Here is what is happening, and how to check your own site in about five minutes.
What robots.txt is actually for
Your robots.txt file is a short text file that lives at the root of your website, at yoursite.com.au/robots.txt. It tells search engine crawlers which parts of your site they should not crawl. Think of it as a note on the front door for visiting robots: these rooms are off limits.
Local service businesses usually use it to block pages that add no value in search. Internal site search results. Cart and checkout pages. Thank-you pages. Login areas. Old staging paths. None of these should compete with your real pages for attention, so you tell Google not to crawl them.
The rules are grouped by which crawler they apply to. A line starting with User-agent names the robot, and the Disallow lines beneath it are the rooms that robot cannot enter. User-agent: * means “every robot”. User-agent: Googlebot means Google specifically.
The mistake: Google reads only one group of rules
Here is the part almost nobody expects. A crawler does not read every rule in your file. It finds the single most specific group that names it, obeys only that group, and ignores all the others. This is not Google being difficult. It is the official standard.
The Robots Exclusion Protocol, published as RFC 9309 in 2022, is clear that only the most specific matching group applies. Google’s own documentation says the same thing: only one group is valid for a particular crawler, user-agent-specific groups and the global group are not combined, and the rest are ignored.
So if your file has a User-agent: * group with all your protective rules, and a separate User-agent: Googlebot group with even one rule in it, Googlebot follows only its own group. Every rule you wrote under the wildcard is invisible to it. The rooms you thought were locked are wide open to the one crawler that matters most.
A real example: before and after
This is the kind of file that quietly breaks. Say your developer wanted to block Googlebot from a staging folder, so they added a group just for it:
User-agent: *
Disallow: /?s=
Disallow: /cart/
Disallow: /checkout/
User-agent: Googlebot
Disallow: /staging/It looks sensible. It is broken. Googlebot sees its own named group and obeys only that one line, Disallow: /staging/. It never reads the wildcard group above, so your internal search results (/?s=), cart and checkout pages are all fair game. That is exactly how search spam ends up indexed: a spammer hits your site’s search box with junk queries, Google crawls those search result pages, and they land in the index under your domain.
There are two clean ways to fix it. The simplest is to drop the separate Googlebot group and put every rule under one wildcard, so it applies to all crawlers at once:
User-agent: *
Disallow: /?s=
Disallow: /cart/
Disallow: /checkout/
Disallow: /staging/If you genuinely need a Googlebot-specific rule, you must repeat the shared rules inside its group, because Google will not borrow them from the wildcard:
User-agent: *
Disallow: /?s=
Disallow: /cart/
Disallow: /checkout/
User-agent: Googlebot
Disallow: /?s=
Disallow: /cart/
Disallow: /checkout/
Disallow: /staging/Both versions work. The first is easier to maintain, which means it is harder to break later.
How to check your own site in five minutes
You do not need to be technical to spot the warning sign. Open a browser and go to yoursite.com.au/robots.txt. Read down the file and look for more than one User-agent line. If you see a group for a named crawler like Googlebot as well as the User-agent: * group, check whether your protective rules are repeated inside the named group. If they are not, you have found the problem.
For a second check, type site:yoursite.com.au into Google and skim the results. If you see internal search pages, cart or checkout URLs, or anything with /?s= in the address, pages that should never appear, that is the symptom showing up in the wild.
What to do if you find it
Fixing the file is quick. Undoing the damage takes a little longer, because pages already in the index do not vanish the moment you correct the rules. You may also need to remove the junk pages through Google Search Console and, in some cases, use a noindex tag rather than robots.txt, since a blocked page can still be indexed if Google finds it linked elsewhere. That is a separate quirk worth understanding, but the group precedence trap is the one that silently undoes work you thought was already done.
If you would rather have someone check this properly, it is one of the first things we look at in a technical SEO review. We also cover how to build a site crawlers can read cleanly in our guide to structuring your website for search.
The takeaway
Most robots.txt problems are not dramatic. They are quiet. A rule that looks right, does nothing, and nobody notices until junk pages start competing with the pages you actually want found. If your file names a specific crawler anywhere, the rest of your rules may not apply to it at all.
So the question worth asking is not whether you have a robots.txt file. It is whether the one crawler that matters is actually reading the rules you wrote. When did you last check?
