This piece has been added as a companion to our AIO service.
The first golden rule in AI optimization is to let AI bots reach your site. Everything else only works if you have given permission first. If you use Cloudflare, Cloudflare tries by default to stop AI bots reaching your site. (Mostly it does not manage it) But when that access block is in place, one thing becomes certain: that you do not want to share your site’s content with LLMs.
So Who Should Allow Them?
If you carry out research, produce content and earn your income from the content you produce, then not giving permission to AI systems is your own decision. Your priority is search engines, and search engine bots should index your site and bring it up in searches. You may not want to let AI bots come to your site and take content you produced with enormous effort and use it with none. In that case, both using LLM.txt to block them and leaving Cloudflare’s bot blocking on its default makes far more sense for you.
What Companies Should Do
The more systems that use the content on your company website, the more that tends to raise the value attached to your products or services. For that reason, blocking your content with LLM.txt or banning it with Cloudflare does not make sense. If your content is structured correctly, having AI systems consume it will usually work in your company’s favor.
Step 1: First, go into your Cloudflare account and click the website you want to allow.
Step 2: Click the “Security” link on the left.
Step 3: Go to the “Bot traffic” section.
Step 4: Click the edit pencil.
Step 5: Tick the box Do not block (allow crawlers)AI training bots will be allowed to scrape content.
By saving, you have allowed AI bots to reach your site and record it.
What should LLM.txt say?
User-agent: *
Allow: /If you do not want your company’s content used in a particular AI system, your LLM.txt code should look like this:
# Block a specific bot (e.g. AIBot)
User-agent: AIBot
Disallow: /
# Allow all the remaining bots
User-agent: *
Allow: /The Names of the Known AI Bots
| Company | Bot name (user agent) |
| OpenAI (ChatGPT) | GPTBot, ChatGPT-User, OAI-Search |
| Google (Gemini/Vertex) | Google-Extended |
| Anthropic (Claude) | ClaudeBot, Claude-Web |
| Meta (Llama) | Meta-ExternalAgent, FacebookBot |
| Apple (Apple Intelligence) | Applebot-Extended |
| Perplexity AI | PerplexityBot |
| Common Crawl (General training data) | CCBot |
| ByteDance (TikTok/Lark AI) | Bytespider |
| Amazon | Amazonbot |
| Microsoft (Copilot) | BingPreview, MSNBOT |
| Cohere | cohere-ai |
| You.com | YouBot |
| Diffbot | Diffbot |
| ImageSift (Image models) | ImagesiftBot |
| PetalSearch (Huawei) | PetalBot |
Our View at Türk SEM
LLMs are growing, and they can gather data from your site whether you allow it or not. Even if you earn income by producing content, it is still possible to draw good visitor numbers from the citation sections. For that reason, trying to appear within them rather than blocking them may be more sustainable for your organization. After all, even if you block them, the information is unlikely to be your product. Producing information is generally an academic discipline. If you are not the one producing the information and are only sharing it, AI systems will reach the content you blocked through other sources anyway.
Rowing against the current can be laborious and may not be sustainable.
