Robots.txt in Joomla: what to block, and what blocking cannot do

Every Joomla site has a small text file in its root that search engines read before anything else: robots.txt. It tells crawlers which folders they may skip. Most site owners never open it, and the ones who do often add lines that hurt more than they help. Here is what Joomla puts in it, what you can safely change, and what the file cannot do.

What Joomla ships

A fresh Joomla 4, 5 or 6 install comes with a robots.txt that blocks the folders containing PHP code and system files:

User-agent: *
Disallow: /administrator/
Disallow: /api/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

Notice what is missing: /images/, /media/ and /templates/. Older Joomla 2.5 and early 3.x versions blocked those too. Google renders pages the way a browser does, so it needs your CSS, JavaScript and images. If it cannot load them, it sees an unstyled page and may decide the site is not mobile friendly. If your site started life on Joomla 2.5 or 3, open the file and check that these three lines are gone.

Why updates leave your file alone

The Joomla package also contains robots.txt.dist. Updates only overwrite that copy, never your robots.txt. That is good news for your changes, and bad news if your file dates from 2012: it will stay outdated until you fix it by hand. Compare the two files after a major upgrade.

Three lines worth adding

• Your sitemap. If you use a sitemap extension, add a line with its full address, for example Sitemap: https://www.example.com/sitemap.xml. Use the exact URL your extension reports; many do not live at /sitemap.xml. Crawlers other than Google pick it up from here without you submitting it anywhere.
• Component-only views. URLs with tmpl=component show a page without the template, often used for pop-ups and modals. Disallow: /*tmpl=component keeps crawlers from indexing these bare copies. Google supports the * wildcard; some smaller crawlers ignore it, which is harmless.
• Search results. Internal search result pages are thin content and can multiply endlessly. If your search menu item lives at /search, add Disallow: /search?. Check the actual path of your menu item first.

Robots.txt does not remove pages from Google

This is the misunderstanding that causes the most trouble. A Disallow line stops crawling, not indexing. If other sites link to a blocked page, Google can still list it, with the note that no description is available. Worse: because Google is not allowed to fetch the page, it never sees a noindex instruction on it.

To keep a page out of search results, let it be crawled and tell Google not to index it. Joomla has this built in:

1. For one page: edit the menu item, open the Metadata tab and set Robots to noindex, follow.
2. For one article without its own menu item: the article edit screen has the same Robots field on its Publishing tab.
3. For the whole site: System > Global Configuration > Site, under Metadata, set Robots to noindex, nofollow. Useful on a staging copy, see below.

Once the page has dropped out of the index you may block it in robots.txt as well, but there is rarely a reason to.

Staging sites and subfolders

A staging copy at staging.example.com needs more than a robots.txt with Disallow: /. As explained above, that does not keep URLs out of the index once someone links to them. Protect staging with a password (most hosts offer directory protection in the control panel) and set the global Robots option to noindex, nofollow. Then make sure that setting is reset when you copy the site to production. Forgetting this is a classic cause of a new site that never appears in Google.

If Joomla runs in a subfolder, such as example.com/shop/, its robots.txt is ignored. Crawlers only read the file at the root of the domain. Move the rules to example.com/robots.txt and prefix every path with /shop.

Check your work

Open https://www.example.com/robots.txt in a browser to confirm the server delivers the file you edited, not a cached or host-generated one. Then use the robots.txt report in Google Search Console to see the version Google last fetched and any lines it could not parse. Finally, run URL Inspection on an important page and look at Page resources: anything listed as blocked by robots.txt is something Google could not load when rendering your page.

Robots.txt decides where crawlers go. What they show once they arrive comes from your page titles and meta descriptions.