Cloudflare

My robots.txt has two authors

The robots.txt in my repo is 17 lines. On 2026-11-06 the one ibrah5em.me served was 147. The extra 130, all above mine, were Cloudflare’s: a legal preamble about content signals, then a managed block:

# BEGIN Cloudflare Managed content

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

# Training crawlers
User-agent: Amazonbot
Disallow: /
...
User-agent: ClaudeBot
Disallow: /
...
User-agent: GPTBot
Disallow: /
...
# END Cloudflare Managed Content

My file came after it, and says something a bit different.

I’d seen it before. I checked the repo’s main, it matched, and I forgot about it while I rebuilt the site.


My file: retrieval yes, training no

My file welcomes the bots that fetch a page to answer a question and cite it: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User. Everyone gets the same signal:

Content-Signal: search=yes, ai-input=yes, ai-train=no

Nothing in my file is blocked; ai-train=no only asks.

The managed block says ai-train=no too, but it never says ai-input=yes, it adds use=reference, and it blocks the training crawlers by name. So the served file has two User-agent: * groups with two different Content-Signal lines. I don’t know which one a crawler honours.

Cloudflare130 linesBot PreferenceSync offmine, 17User-agent: *search=yesai-train=nouse=referencetraining bots: DisallowUser-agent: *search=yes, ai-train=noai-input=yes

The block that comes and goes

On 2026-08-17 Cloudflare had prepended it. By 2026-09-29 it was gone: the live file was just the repo’s, and I diffed it again on 2026-10-02 to be sure. I wrote in my decision notes that the repo file was the whole policy.

My plan from that time had another thing wrong. It said ai-input=yes was “one toggle” in AI Crawl Control. That control doesn’t exist. The signal went into my own file instead.

On 2026-11-06 the block was back.

The switch is Bot Preference Sync, in the Signals tab of AI Crawl Control. Cloudflare’s own dashboard label says it “prepends your robots.txt with your configured Cloudflare settings”. I turned it off that day, and the live file went back to the repo’s 17 lines.


Diffing the served file against the repo

diff <(curl -s https://ibrah5em.me/robots.txt) public/robots.txt

That’s how this turned up: 130 lines on the left that git has never seen.

Now I run that diff by hand.