You did the work to show up in AI search. The content, the structure, the effort to be the business ChatGPT recommends when someone asks for a plumber in your suburb. Now a setting you have never touched, in a dashboard you have probably never opened, could be quietly undoing it. If your website runs through Cloudflare, the file that tells AI bots what they can and cannot do may no longer be the file your developer wrote. It can change on its own, and you might not see it happen.
Here is the short version, so you know whether to keep reading. On 1 July 2026, Cloudflare launched a feature called Bot Preference Sync. It writes your robots.txt file for you, based on three toggles in the Cloudflare dashboard: Search, Agent and Training. From 15 September 2026, it is switched on by default for every new Cloudflare customer. If your site sits behind Cloudflare, this affects you. If it does not, you can stop reading and get your afternoon back.
What Cloudflare actually changed
Your robots.txt file is a plain text instruction sheet at the root of your website. It tells crawlers, including the AI bots behind ChatGPT, Google and Perplexity, which parts of your site they may read. For years, that file was static. Someone wrote it once, and it stayed put until someone changed it.
Bot Preference Sync ends that. Instead of a fixed file, Cloudflare now generates and updates your robots.txt to match the AI bot policy you set in its dashboard. Cloudflare describes the goal plainly: the preference you set is the preference you publish, so what you tell the world and what you enforce at the edge stay in step (Cloudflare, “Say it once: introducing Bot Preference Sync”, 21 August 2026, updated 15 September 2026). It is available on every plan, from free to enterprise, and it can be turned on or off at any time.
The feature is not the problem. It fixes a real one. Slobodan Manic, writing for Search Engine Journal, found that his own robots.txt had drifted from his actual preferences for months without him noticing (Search Engine Journal, “Cloudflare Will Write Your Robots.txt, And It Has A Point”, 18 September 2026). That is the exact gap Cloudflare is closing: files that say one thing while the server does another. Fair intent, sensible fix.
The catch is different. If a toggle you never set is now deciding what your robots.txt says, you need to know what it decided. And because the enforcement can live in the dashboard rather than the file, checking the file alone may not tell you the truth any more.
Why this matters if you want AI to cite your business
Most trade and service business owners in WA have never logged into a Cloudflare dashboard, and never will. But plenty of sites run through Cloudflare without the owner ever choosing it directly, added by a web host, a security plugin, or a CDN setup done years ago.
If yours is one of them, a security default could now be deciding whether ChatGPT can read your site and recommend you to a customer. Set the Search toggle to block, and you are telling the crawler that powers AI search results to stay out. You could have the best AI visibility work in your market and still be invisible, because an unrelated setting locked the door. This is no longer a “go check your robots.txt” story. It is a “check your CDN dashboard too, because the file might not be the whole story” story.
The five-minute check, using a real example
Say you run a plumbing business in Rockingham, and your site sits behind Cloudflare. Here is how to check where you stand, in two steps, without needing a developer.
Step one: read your own robots.txt. Type your domain into a browser followed by /robots.txt, so yourbusiness.com.au/robots.txt. Look for a line that reads # BEGIN Cloudflare Bot Preference Sync. If that marker is there, Sync is active on your site, and Cloudflare is managing this part of the file. If it is not there, Sync is not writing your file, and your robots.txt is still whatever your developer set.
Step two: if the marker is present, check the dashboard. This is the part people miss. When Sync is on, the file reflects the dashboard, so the dashboard is where the real decision lives. In Cloudflare, that setting sits under the AI bot policy for your domain, where you choose what to do with Search, Agent and Training traffic. Each has its own choice, and the wording matters.
Take a business that wants to be found in AI search but does not want autonomous agents acting on its site or its content used to train models. A conservative position looks like this: Allow Search, Block Agent, Disallow Training. Search stays open so the crawler behind AI results can read and cite you. Agent stays blocked unless you have a reason to let AI assistants act on your site. Training is disallowed so cooperating crawlers receive your no-training preference. The one that decides your AI visibility is Search. If you want ChatGPT to recommend your business, Search needs to be allowed, at minimum. Block it, and no amount of good content changes the outcome.
That is the whole check. Two steps, five minutes, and you know whether a setting you never chose is working for you or against you.
What to do with what you find
If the marker is absent, Bot Preference Sync is not writing this section of your robots.txt, but that does not prove every Cloudflare edge policy is inactive. Check the dashboard if you need certainty. If the marker is present and Search is allowed, you are in good shape, and it is worth a note in your records so the next person who touches the site knows Sync is live. If the marker is present and Search is blocked, that is the finding: a setting is standing between your business and AI search, and it is a deliberate change, not a fault.
Changing a live security setting is not something to do on a hunch. The safe move is to confirm what the toggle is doing, decide what you actually want it to do, and make the change knowing why. If you are not sure, that is a conversation with whoever manages your hosting or your site, not a switch to flip blind.
This sits alongside two things worth understanding: which AI bot tokens actually matter in your robots.txt, and how the same “the file you think you have is not the one being enforced” trap shows up in the way robots.txt group precedence is handled. Get all three right, and you close the most common ways a business quietly locks itself out of AI search.
The point of doing the work to be found by AI is that it actually pays off. Do not let a toggle you never set decide otherwise. If AI citation is the goal, here is how the pieces fit together. So the question worth five minutes of your afternoon is simple: right now, does your own site let the AI bots in, or has something already decided that for you?
Sources
- Cloudflare (Jin-Hee Lee), “Say it once: introducing Bot Preference Sync”, 21 August 2026, editorial update 15 September 2026, https://blog.cloudflare.com/bot-preference-sync/, retrieved 2026-09-18.
- Slobodan Manic, Search Engine Journal, “Cloudflare Will Write Your Robots.txt, And It Has A Point”, 18 September 2026, https://www.searchenginejournal.com/cloudflare-will-write-your-robots-txt-and-it-has-a-point/589262/.
