how to apply robots.txt?

[gsmdib@…]

This is probably not solely related to trac, but still, could you please advise how to make robots.txt available for a trac instance? Thanks.

[Noah Kantrowitz]

How are you running the Trac? If via Apache or Lighty, use their normal methods for serving files. If you are using tracd, you can look at the ​RobotsTxt plugin.

[gsmdib@…]

Via apache. Could you please advise how to implement the 'normal method'? TIA.

[sid]

Doing a simple web search will provide a lot of documentation how to do this. ​example search query for 'apache robots.txt

This is not a Trac issue.

[anonymous]

When hosting trac with apache and mod_python you have to explicitly unset any Python handlers for the robots.txt file:

<Location "/robots.txt">
    SetHandler None
</Location>

[anonymous]

Also sone that you have to put that entry (Location "/robots.txt"…) *after* the Python handler, otherwise it won't help.

[anonymous]

I put the robots.txt file in the htdocs/ directory. Trac 0.10.4 can find it there.

[s.trac@…]

If like me you are having difficulties because trac resides in the root, i.e. trac.foo.com, and trac.foo.com/robots.txt gives "environment not found", then this may help if you are using apache:

RewriteEngine On
RewriteRule ^/robots.txt /htdocs/robots.txt [PT]
Alias /htdocs /where/ever/trac/is/htdocs

YMMV.

[Sebastian Krysmanski <sebastian@…>]

Just a note: I've created a Python script that automatically creates a robots.txt file for multiple Trac environments. You can find it ​here. Although it's programed to be used in conjunction with the ​Mass Trac Provider Project you could easily adjust it to your own needs.

[anonymous]

The above samples are overly complicated, just use this:

    Alias /robots.txt /var/www/trac-robots.txt

or wherever you want to put your robots.txt file. The sample attached earlier is good, but you can also use this to simply block all robots from accessing everything in the context you are putting the alias (VirtualHost, server, etc):

User-agent: *
Disallow: /

See also #6124.

[gregory@…]

For me a combination of

Alias /robots.txt /export/home/trac/base/htdocs/robots.txt
<LocationMatch "/robots.txt">
  SetHandler None
</LocationMatch>

worked.