Provide Etag and/or Last-Modified HTTP headers to ease cache control

[Giuseppe Bilotta <giuseppe.bilotta@…>]

Trac serves pages without any Etag or Last-Modified header, which makes cache management rather troublesome, and increases the server load. This is particularly important for RSS feeds, which might be checked relatively often.

(In one of the comments for #663 it was mentioned that conditional GET was planned for 0.9, but one of Trac setups I'm checking on is based on 0.10 and the functionality seems to be missing.)

I think most pages could be served with a Last-Modified header containing the date of the last change (for wiki and ticket pages) or the date of the most recent thing changed (for RSS feeds).

[mark@…]

I second this one.

It would also help if these headers wouldn't be send (which is with 0.10 the case):

Cache-Control: must-revalidate
Expires: Fri, 01 Jan 1999 00:00:00 GMT

[betatim]

it should be trivial todo for the wiki

after line 135 in source:/trunk/trac/wiki/web_ui.py​ add

req.check_modified(page.time)

this only sends a etag for pages without a query part, except for ?format options, not quite sure how to handle those. maybe

req.check_modified(page.time, format)

is a better idea. sorry for not diffing.

[Christian Boos]

Seems like the check_modified is gone from the wiki web_ui, need to check this.

[Christian Boos]

Speaking of Trac resources in general (Wiki pages, tickets, milestones), it's quite difficult to know what the actual content will be at any point in time, as we may have some dynamic content in it (think [[TicketQuery]] macro, or simply ticket references which can be striked out or not depending on the status of the ticket).

We try to do that for changesets, as they have a supposedly more stable content, but even in that case we sometimes present an outdated content (commit, view the changeset referencing a ticket, close the ticket, go back to the changeset → the ticket link is not striked out). I think we can't afford to serve a possibly outdated content for Wiki pages and other resources, so I think that the current Cache-Control: must-revalidate is just fine (it seems we don't send the Expires anymore).

Now for RSS feeds, this might be different, as the content is anyway supposed to be a "teaser" for the actual content and there it would probably be fine if we would send a Last-Modified: header or an Etag: based on the version number.

[Mitar]

I think there should definitely be some support for caching. At least for anonymous users. I wouldn't mind if an anonymous user gets a bit obsolete macro output. For example, I would like that for anonymous users HTTP cache is possible for 30 minutes. Yes, this will mean that macro outputs will be maybe obsolete, but so what. Also browsing files could definitely be cached. For example all those diffs and so on. This content does not change. And if you have web crawlers going around, you can get a lot of this traffic.

For crawlers, I have been also quite successful with this robots.txt file:

User-agent: *
Disallow: /newticket
Disallow: /search
Disallow: /query
Disallow: /wiki/AttachmentsIndex
Disallow: /wiki/CamelCase
Disallow: /wiki/InterMapTxt
Disallow: /wiki/InterTrac
Disallow: /wiki/InterWiki
Disallow: /wiki/PageTemplates
Disallow: /wiki/RecentChanges
Disallow: /wiki/SandBox
Disallow: /wiki/TitleIndex
Disallow: /wiki/Trac
Disallow: /wiki/Wiki
Disallow: /wiki/*?*version=
Disallow: /wiki/*?*action=
Disallow: /ticket/*?*version=
Disallow: /ticket/*?*action=
Disallow: /ticket/*?*cnum_hist=
Disallow: /ticket/*?*cversion=
Disallow: /browser/*?*rev=
Disallow: /changeset/*?*format=
Disallow: /report/*?*format=
Disallow: /query*?*format=
Disallow: /timeline*?*precision=
Allow: /