Steps

A wiki open on the Internet is visited by search robots that follow every URL they find, including the ones that export a page, run a query or open an editor, and that costs the instance far more than serving a page: see Performance. A robots.txt keeps them to what is worth indexing.

1. Write the rules below into a robots.txt. The patterns use * rather than /bin/ so that they also match a path-based subwiki such as /xwiki/wiki/myalias/view/Space/Page.

   User-agent: *
   # Prevent bots from executing all actions except "view" since:
   # 1) we don't want bots to execute stuff in the wiki!
   # 2) we don't want bots to consume CPU and memory
   #    (for example to perform exports)
   # Attached images are allowed below, before /download/ is disallowed.
   Allow: /xwiki/*/download/*.png$
   Allow: /xwiki/*/download/*.jpg$
   Allow: /xwiki/*/download/*.jpeg$
   Allow: /xwiki/*/download/*.gif$
   Disallow: /xwiki/*/viewattachrev/
   Disallow: /xwiki/*/viewrev/
   Disallow: /xwiki/*/pdf/
   Disallow: /xwiki/*/tex/
   Disallow: /xwiki/*/edit/
   Disallow: /xwiki/*/create/
   Disallow: /xwiki/*/inline/
   Disallow: /xwiki/*/preview/
   Disallow: /xwiki/*/save/
   Disallow: /xwiki/*/saveandcontinue/
   Disallow: /xwiki/*/rollback/
   Disallow: /xwiki/*/deleteversions/
   Disallow: /xwiki/*/cancel/
   Disallow: /xwiki/*/delete/
   Disallow: /xwiki/*/deletespace/
   Disallow: /xwiki/*/undelete/
   Disallow: /xwiki/*/reset/
   Disallow: /xwiki/*/register/
   Disallow: /xwiki/*/propupdate/
   Disallow: /xwiki/*/propadd/
   Disallow: /xwiki/*/propdisable/
   Disallow: /xwiki/*/propenable/
   Disallow: /xwiki/*/propdelete/
   Disallow: /xwiki/*/objectadd/
   Disallow: /xwiki/*/commentadd/
   Disallow: /xwiki/*/commentsave/
   Disallow: /xwiki/*/objectsync/
   Disallow: /xwiki/*/objectremove/
   Disallow: /xwiki/*/attach/
   Disallow: /xwiki/*/upload/
   Disallow: /xwiki/*/download/
   Disallow: /xwiki/*/temp/
   Disallow: /xwiki/*/downloadrev/
   Disallow: /xwiki/*/dot/
   Disallow: /xwiki/*/svg/
   Disallow: /xwiki/*/delattachment/
   Disallow: /xwiki/*/login/
   Disallow: /xwiki/*/loginsubmit/
   Disallow: /xwiki/*/loginerror/
   Disallow: /xwiki/*/logout/
   Disallow: /xwiki/*/charting/
   Disallow: /xwiki/*/lock/
   Disallow: /xwiki/*/redirect/
   Disallow: /xwiki/*/admin/
   Disallow: /xwiki/*/export/
   Disallow: /xwiki/*/import/
   Disallow: /xwiki/*/get/
   Disallow: /xwiki/*/distribution/
   Disallow: /xwiki/*/imagecaptcha/
   Disallow: /xwiki/*/unknown/
   # Don't index sample content.
   Disallow: /xwiki/*/view/Sandbox/
   # The Admin space is protected by rights anyway; this is a safety net.
   Disallow: /xwiki/*/view/Admin/
   # Statistics pages are not useful and are CPU intensive.
   Disallow: /xwiki/*/view/Stats/
   # Panels are UI, not content.
   Disallow: /xwiki/*/view/Panels/
   # The search page.
   Disallow: /xwiki/*/Main/Search
   # The REST API.
   Disallow: /xwiki/rest/
   # UI elements that send crawlers into infinite loops (XWIKI-16915).
   Disallow: /xwiki/*?*xpage=*

2. Serve the file from the root of the host, which is the only place a crawler reads it: the document root of the web server in front of XWiki, or the ROOT web application of the Servlet Container when it answers directly. A robots.txt inside /xwiki/ is never read.
3. Give each host its own file. A wiki farm answering on several domains, and a subwiki on a domain of its own, each serve their own robots.txt.
4. Request the file and check that the rules are the ones the crawler will see.

   curl https://wiki.example.com/robots.txt

FAQ

Why is JavaScript and CSS not blocked?

Crawlers render a page before indexing it, so a wiki whose scripts and stylesheets are blocked is indexed as a broken page. The skin, jsx, ssx, webjars and resources paths are therefore left crawlable.

Why is /download/ both allowed and disallowed?

The four Allow lines are more specific than the Disallow that follows them, and a crawler applies the longest matching rule, so images attached to pages are indexed while other attachments are not.

Can whole spaces be kept out of the index?

Yes. Add Disallow: /xwiki/*/view/MySpace/ for a space that carries no content worth finding, and an Allow line above it for any single page inside it that is.

Does this keep private pages out of search results?

No. A robots.txt is a request a crawler chooses to honour, not an access control. Pages that must not be read by anyone are protected by Rights.

Related

• Performance
• HTTP Reverse Proxy
• Rights