Joomla and AI search: llms.txt and AI crawlers

ChatGPT, Claude, Perplexity and Google's AI answers can only quote your Joomla site if they can fetch it. Three things decide that: what your robots.txt allows, whether a firewall or CDN turns the crawlers away, and whether the text is in the HTML at all. An llms.txt file comes after that, as a table of contents for AI tools. You can write that file by hand, or let our own GEO AI plugin build it from your Joomla articles. Both are covered below.

Know which crawler does what

Most AI companies run more than one bot, and each one has its own user agent. Blocking one does not block the others.

• OpenAI: GPTBot collects training data, OAI-SearchBot builds the index for ChatGPT search, ChatGPT-User fetches a page when a user asks about it.
• Anthropic: ClaudeBot for training, Claude-SearchBot for search, Claude-User for fetches on request.
• Perplexity: PerplexityBot for the index, Perplexity-User for fetches on request.
• Google: Google-Extended is not a separate crawler but a robots.txt token. It decides whether Gemini may use your pages. It has no effect on Google Search, and AI Overviews are built from the normal Googlebot index.
• Common Crawl: CCBot. Its public dataset is used to train many language models.

A sensible middle ground for many sites: block the training bots if you object to that use, and allow the search and user bots, because those are the ones that cite and link to you. Note that the user-triggered fetchers may not apply robots.txt the way regular crawlers do; check each vendor's documentation.

Write the rules in robots.txt

Joomla ships the file as robots.txt.dist in the web root. Rename it to robots.txt if you have not done that yet, keep the default Joomla lines and add groups per bot:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /administrator/
Disallow: /api/
...

A bot follows the most specific group that names it and ignores the * group, so repeat any Disallow lines that should apply to it as well. The details of what robots.txt can and cannot do are in robots.txt in Joomla.

Check that nothing else blocks them

A clean robots.txt is useless if the request never reaches Joomla. Common causes:

• CDN settings. Cloudflare has a setting that blocks AI crawlers, and on newer accounts it can be on by default. Look under the security and bot settings of your zone.
• Security extensions. Bad-bot lists in firewall extensions or in a host's WAF sometimes include AI user agents.
• Rate limits. Crawlers fetch quickly and can trip a limit meant for brute-force attempts.

The access log tells you what really happens. On the server:

grep -E "GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|PerplexityBot" access.log | awk '{print $9}' | sort | uniq -c

That counts the status codes per request. Mostly 200 is good. Rows of 403 or 429 mean something in front of Joomla says no.

Make sure the text is in the HTML

Most AI crawlers do not run JavaScript. Joomla itself renders articles on the server, so plain articles, tabs and accordions are fine. The risk is in extensions that load content afterwards: some page builders, review widgets, AJAX-loaded tabs, product data fetched by a script. Test it the way a bot sees it:

curl -s -A "GPTBot" https://example.com/your-article | grep -c "a sentence from the article"

If the count is 0, the text is not in what the crawler receives.

Add an llms.txt file

llms.txt is a proposal from September 2024 for a Markdown file at /llms.txt that tells language models what a site is about and where the important pages are. It is not a standard yet. No major search engine has confirmed using it for ranking, and Google has said it does not use it. It costs little to add, and some AI tools and agents do look for it.

The format is simple: a # heading with the site name, a > line with a one-sentence summary, and ## sections with links, each followed by a short description:

# Example Company

> We repair and service espresso machines in Utrecht.

## Services
- [Repairs](https://example.com/repairs): what we fix, how long it takes
- [Maintenance plans](https://example.com/maintenance): yearly service for offices

## Optional
- [About us](https://example.com/about): history and team

A section called Optional marks links a tool may skip when it is short on space. Good descriptions already exist in most Joomla sites: the Meta Description of each article, which you should keep filled anyway (see page titles and meta descriptions).

Putting it in Joomla

Upload the file to the web root, next to index.php. Joomla's .htaccess only rewrites requests for files that do not exist, so Apache serves /llms.txt directly; on nginx the usual try_files $uri rule does the same. If accented characters come out garbled, add AddCharset utf-8 .txt to .htaccess.

The weak spot of a hand-written file is upkeep. Every new or unpublished article means editing it again, and a file that lists pages returning 404 does more harm than none. That upkeep is why we built a plugin for it.

Quick checklist

• robots.txt names the AI bots you want to allow or block.
• The access log shows 200 responses for those bots, not 403 or 429.
• The article text is present in the HTML without JavaScript.
• /llms.txt loads, lists only live pages and has a description per link.
• Someone or something keeps llms.txt current: you after every article, or a plugin such as GEO AI.