CSS and JS files should be versioned

[peter.westwood@…]

When we recently upgraded a trac install a number of users has strange behaviour on the Custom Query page.

It turned out that this was due to JS caching.

Ideally Trac should use version numbering on the CSS and JS uris so that the cache is busted on upgrades.

[Remy Blank]

Yes, we have discussed that a few times. Is it necessary to actually change the file name part of the URLs to static content, or is it enough to append a query string? In the latter case, we could append a query argument to every URL to static content, with a value containing the Trac version, or the timestamp of the last upgrade, or something like that.

This is material for 0.13.

[anonymous]

You just need to increment a query string argument and that should be enough.

In WordPress we use a scheme based on the date the file changes so as to work for people who have installs which follow development closely.

Adding the version number would still be great though :-)

[Remy Blank]

Replying to anonymous:

In WordPress we use a scheme based on the date the file changes so as to work for people who have installs which follow development closely.

I thought about this, too, but it might not work when using trac-admin $ENV deploy and having the web server serve static content, as Trac doesn't know where those files are.

[Carsten Klein <carsten.klein@…>]

Would it not suffice to include an ETAG with a given resource? This could be for example the MD5 hash of the resource data. Including this with the response will AFAIK reset the client cache for that url, replacing it by the new data?

See ​http://en.wikipedia.org/wiki/HTTP_ETag for more information on this.

I strongly vote against a …/trac.css?version=1…N approach…

[Carsten Klein <carsten.klein@…>]

see also Request#check_modified for this

[Christian Boos]

Replying to Carsten Klein <carsten.klein@…>:

Would it not suffice to include an ETAG with a given resource?

We're talking about a general solution here, not just when Trac is serving those resources through the Chrome component.

[Remy Blank]

Version URLs for CSS and JS files.

[Remy Blank]

9936-versioned-css-js-r10548.patch​ adds a query string to all CSS <link> and JavaScript <SCRIPT> URLs, of the form ?v={trac_version}. The interesting part is in chrome.py, the rest just transforms most <link> and <script> tags embedded in templates into the corresponding calls to add_stylesheet() and add_script(), respectively.

The interesting thing with using the Trac version in the query string is that:

• Users who install from a release package (source or binary) have a version 0.x.y, and the query string change with every release.
• Users who install from SVN have a version 0.x.ydev-r1234, and therefore the query string also changes with every update, even of a single revision.

Both Firefox and Chrome correctly cache the files, and re-get them when the version changes. Opera and IE8 display correctly, but I haven't been able to check their caching behavior. I'm pretty confident that it works correctly, though, as WordPress uses the same scheme.

Feedback and testing welcome.

[Christian Boos]

Replying to rblank:

… though, as WordPress uses the same scheme.

Wikipedia also.

Feedback and testing welcome.

Looks good! Applied on demo-0.13.

[osimons]

Hold your horses, please… Any solution that don't account for plugins is a no-starter as far as I'm concerned. Upgrading plugins can of course happen without upgrading Trac…

Using a value based on timestamp of each file resource seems like an easy and general solution to me, but of course having to stat each file each time it is added means quite a bit of overhead. It can be eased by caching timestamps for files as we go, but then we'd really need to think about caching invalidation strategy too so that we don't arrive at the point where the only way of updating static resources is to restart the process(es).

[Jun Omae]

Replying to rblank:

Feedback and testing welcome.

I need the solution using the timestamp of each file. Trac plugin developers maybe need it. I always say "Please force reload" to reporters each time I change my plugin.

I think that add_stylesheet should append v=VERSION when the prefix of filename parameter is common/.

trac/web/chrome.py

@@ -137 +137 @@
     """
     if filename.startswith('http://') or filename.startswith('https://'):
         href = filename
-    elif filename.startswith('common/') and 'htdocs_location' in req.chrome:
-        href = Href(req.chrome['htdocs_location'])(filename[7:], v=VERSION)
+    elif filename.startswith('common/'):
+        if 'htdocs_location' in req.chrome:
+            href = Href(req.chrome['htdocs_location'])(filename[7:], v=VERSION)
+        else:
+            href = req.href.chrome(filename, v=VERSION)
     else:
         href = req.href
         if not filename.startswith('/'):
             href = href.chrome
-        href = href(filename, v=VERSION)
+        href = href(filename)
     add_link(req, 'stylesheet', href, mimetype=mimetype, media=media)
 
 def add_script(req, filename, mimetype='text/javascript', charset='utf-8'):
@@ -159 +162 @@
 
     if filename.startswith('http://') or filename.startswith('https://'):
         href = filename
-    elif filename.startswith('common/') and 'htdocs_location' in req.chrome:
-        href = Href(req.chrome['htdocs_location'])(filename[7:], v=VERSION)
+    elif filename.startswith('common/'):
+        if 'htdocs_location' in req.chrome:
+            href = Href(req.chrome['htdocs_location'])(filename[7:], v=VERSION)
+        else:
+            href = req.href.chrome(filename, v=VERSION)
     else:
         href = req.href
         if not filename.startswith('/'):
             href = href.chrome
-        href = href(filename, v=VERSION)
+        href = href(filename)
     script = {'href': href, 'type': mimetype, 'charset': charset}
 
     req.chrome.setdefault('scripts', []).append(script)

(untested)

[Christian Boos]

Plugins may adopt the same strategy as Trac, and produce URLs with their own version number.

We can indeed rethink the way we handle all these add_script/add_… functions and make them instead methods of some class which can be given a base version. Plugins would use their own instance with their own version (current functions remain unchanged for backward compatibility).

The proposal concerning the timestamp was already addressed by Remy in comment:3, we don't know in Trac where those files are (they may even be located on another machine).

[Remy Blank]

Replying to osimons:

Hold your horses, please… Any solution that don't account for plugins is a no-starter as far as I'm concerned. Upgrading plugins can of course happen without upgrading Trac…

I know I'm guilty of asking for feedback, but somehow I would appreciate if it was a bit more constructive:

• The proposed solution, while not taking plugins into account, is already much better than the previous situation. It doesn't change any public interface, has close to zero overhead, and solves the issue at least for core, which is already a bit part. As such, it is a clear net gain. Calling it a no-starter feels… inappropriate.
• I'm of course open to improvement suggestions, in particular, how this could be extended to plugins. While the timestamp approach sounds nice and general, my feeling is that it will quickly become very complicated. Remember that finding the path to e.g. a .css requires going through the whole ITemplateProvider machinery, including looking into .egg files.

So at this point, I have a simple solution implemented that solves a good portion of the issue, and an idea to compare against that feels overly complicated. How about extending the simple solution by adding a version= argument to add_stylesheet() and add_script() that defaults to VERSION, and through which plugin authors could pass the version string of the plugin? Sure, it requires some work by plugin authors, and yes, it makes it more difficult to have the same code run on 0.11, 0.12 and 0.13. It's still better than the current state.

There's another unsolved issue, the images. Unfortunately, the URLs for those are often placed inside .css files, so it's not possible to dynamically add a query string there (except if we start generating CSS dynamically, I remember a discussion about that). Also, images tend to change much less frequently than CSS and JS, so I'm not too concerned if the solution doesn't take images into account.

(Mid-air collision with more feedback, I'll address those separately.)

[Remy Blank]

Replying to jomae:

I need the solution using the timestamp of each file.

Can you be more specific as to why you need that (in particular, the for each file part, as opposed to a per-plugin version)?

I think that add_stylesheet should append v=VERSION when the prefix of filename parameter is common/.

I'm not sure I understand the reason for the patch you posted. Do you mean that I add the ?v=... to too many items, i.e. that it shouldn't be added if the file isn't in /chrome?

[osimons]

Providing 'inheritable class-frameworks' that plugins can use is a sure way of guaranteeing that it will be handled incorrectly - and painful to support and revise both in Trac and in plugins. Grafting 'version' onto links manually just does not feel right, and we should not go that way.

Any installation that serves resources outside Trac, links to external resources, or serves Trac-deployed resources as static files, should also take care of their own content-delivery and make sure the files are cached and expired as they see fit. I don't see how that affects Trac itself?

Any file that is added and scoped for lookup inside Trac (ie. relative files) is a file that we can find and check the timestamp of. If myplugin.js is deployed and actually served outside Trac is of no importance - we stat the original that we have.

Going that way, using an Etag based on file stats would be just as useful and much more general - we Etag the files we serve from Trac when requested, and ignore the rest. Users that serve static files outside Trac are left to expire them as they see fit using own web server skills, content-delivery networks or whatever.

I'd say that using Etag is more useful than the proposed version idea.

[Remy Blank]

Replying to osimons:

Providing 'inheritable class-frameworks' that plugins can use is a sure way of guaranteeing that it will be handled incorrectly - and painful to support and revise both in Trac and in plugins. Grafting 'version' onto links manually just does not feel right, and we should not go that way.

"Doesn't feel right" is a pretty weak argument. Anything more concrete? I can't help noticing that some major sites are using this technique, so discarding it solely based on a feeling… doesn't feel right either :)

I'd say that using Etag is more useful than the proposed version idea.

That may well be the case. Then again, for it to be really useful, somebody has to implement it.

/me is waiting for a patch to compare against.

[osimons]

Replying to rblank:

I know I'm guilty of asking for feedback, but somehow I would appreciate if it was a bit more constructive:

Oh, so feedback needs to be constructive too now?! Heh, I do intend to be and sorry if it does not always come across that way.

• The proposed solution, while not taking plugins into account, is already much better than the previous situation. It doesn't change any public interface, has close to zero overhead, and solves the issue at least for core, which is already a bit part. As such, it is a clear net gain. Calling it a no-starter feels… inappropriate.

See, that is where we sort-of disagree - and to a large part it has to do with how we define 'core'. You define it as 'whatever ships as Trac', while I define it as the sub-set of services that all modules use whether they are shipped with Trac (like wiki, ticket etc) or provided as plugins. Having to tweak and add cruft to individual modules to compensate for missing or inefficient core APIs is what I want to avoid.

There's another unsolved issue, the images. Unfortunately, the URLs for those are often placed inside .css files, so it's not possible to dynamically add a query string there (except if we start generating CSS dynamically, I remember a discussion about that). Also, images tend to change much less frequently than CSS and JS, so I'm not too concerned if the solution doesn't take images into account.

Etag.

[Christian Boos]

Replying to osimons:

Any installation that serves resources outside Trac, links to external resources, or serves Trac-deployed resources as static files, should also take care of their own content-delivery and make sure the files are cached and expired as they see fit. I don't see how that affects Trac itself?

The problem we try to address here is to circumvent the overly aggressive caching done by web browsers which sometimes decide to ignore caching headers. For example, on t.e.o we serve the static resources via lighttpd, which properly sets the timestamps, ETag and what not. Yet this is not enough. There seem to be an emerging technique of appending fake parameters to such resource URLs as a way to force those browsers to do the right thing. Like it or not (my first reaction was not to find that particularly elegant either), but this seems to become common practice, and if this solves a real problem I don't see why we should not use it.

Any file that is added and scoped for lookup inside Trac (ie. relative files) is a file that we can find and check the timestamp of. If myplugin.js is deployed and actually served outside Trac is of no importance - we stat the original that we have.

stating files like that is likely to induce excessive overhead, or complexity if we want to do it in a smart way, as you originally mentioned. For no real benefit either, how often do you hack "live" the resources of a production system?

Going that way, using an Etag based on file stats would be just as useful and much more general - we Etag the files we serve from Trac when requested, and ignore the rest. Users that serve static files outside Trac are left to expire them as they see fit using own web server skills, content-delivery networks or whatever.

I'd say that using Etag is more useful than the proposed version idea.

We indeed don't add ETags in /chrome for the resource files Trac serves itself, this could be done. But this is a separate issue (i.e. at the risk of repeating myself, even if we would add proper ETags, the browsers are likely to over aggressively cache those files, Javascript in particular).

[osimons]

All that would be needed is to add Etag support to Request.send_file() + tune Request.check_modified(). That should take care of chrome, attachments and whatever else serves files via Trac. Or am I missing anything? I've just used Etags but never really implemented them myself.

As for agressive caching, that sounds like it is somewhat based on hearsay and occasional bad experiences. As with the edewall.org, I too serve all my static resources outside Trac and never have a problem with static resources not getting updated using Etag and modified information.

My experience is that browsers generally handle this OK (when done correctly server-side), but that may of course not be the same for proxy servers that organisations may employ - they are often much more agressive. However, using strong Etag validation as we could do for static files may well improve on any weak validation ('W/...) that are currently supported in Request.check_modified().

[Remy Blank]

Replying to osimons:

All that would be needed is to add Etag support to Request.send_file() + tune Request.check_modified(). That should take care of chrome, attachments and whatever else serves files via Trac. Or am I missing anything?

I was going to say that this wouldn't work with .egg plugins, but in fact it will, because static resources are extracted as files anyway. So this may indeed be sufficient. Care to provide a patch that we could try?

[Christian Boos]

Replying to osimons:

As for agressive caching, that sounds like it is somewhat based on hearsay and occasional bad experiences.

This is the heart of the problem. After seeing that WordPress and Wikiedia all adopted this strategy, I thought this was most probably a real issue. But to be sure I did some tests myself with demo-0.13 and the first browsers I tested immediately picked the changes I made to trac.js file served by lighttpd, which uses a strong ETag.

As with the edgewall.org, I too serve all my static resources outside Trac and never have a problem with static resources not getting updated using Etag and modified information.

My experiments seemed to confirm this… until I tested IE9 and Opera 11 ;-) (could someone else repeat the tests with IE8?) These last browsers don't care at all about the ETag and the Last-Modified headers, for the .js files or .css files, until you press <F5> (reload).

So while Chrome, Safari and FF (at least recent versions of those) seem to behave sensibly, this is a real problem for IE and Opera. Still a sizeable part of the Internet audience, I suppose.

[osimons]

Alternative patch using headers only.

[osimons]

Replying to rblank:

Care to provide a patch that we could try?

Sure, something like attachment:t9936-etag_files-r10542_012.diff​ was what I had in mind. It actually reduces LOC count by 4 :-) and seems to work in my testing at least. How it fares in the big world I don't know, but if anything I'm quite sure this strategy don't hurt and can only improve things.

[Christian Boos]

Replying to osimons:

Replying to rblank:

Care to provide a patch that we could try?

Sure, something like attachment:t9936-etag_files-r10542_012.diff​ was what I had in mind.

Looks good! Nitpick: I'd prefer weak=True instead of strong_validation=False though (immediately makes it obvious what 'W' stands for).

Though as I explained in comment:18 and confirmed in comment:21, this addresses a different issue than what this ticket was initially about.

[osimons]

Opera 11 works nicely for me, it returns 304 as it should for every subsequent file requests. Are you using non-standard cache settings? As for IE8 I have stopped trying to understand how its developer tools work, so I cannot for the world manage to figure out if images responds with 200 or 304…

I don't agree that my patch addresses a different issue, but it does of course provide a different solution for improving things with regards to caching.

[Christian Boos]

Replying to osimons:

Opera 11 works nicely for me, it returns 304 as it should for every subsequent file requests.

Ok, but does it also reload the resource when you modify it?

As for IE8 I have stopped trying to understand how its developer tools work

You don't need the developer tools: go to a Trac instance (e.g. WikiStart), then modify trac.js on the server in a visible way (e.g. add "Hello" before \u00B6 in addAnchor) and then go to a different Wiki page (or the same but by following a link, not by pressing reload): if you see "Hello", then the new trac.js was reloaded. If it only appears after an explicit reload, then you see the browser caching effect we're talking about.

I don't agree that my patch addresses a different issue

Sorry, it's not really a matter of opinion here. The issue is different: we're talking about browsers not respecting the ETag hints in case they are present, so adding ETags in places in which they were not already present is indeed something different.

[Christian Boos]

Some further thoughts on the topic, after reading ​http://code.google.com/speed/page-speed/docs/caching.html. For static resources, we could implement more aggressive caching if we would set the Expires or Cache-Control: max-age at a date far in the future, yet control the refresh when needed with the version parameter. In that page, they refer to this technique as "URL fingerprinting".

[Remy Blank]

Replying to osimons:

Sure, something like attachment:t9936-etag_files-r10542_012.diff​ was what I had in mind.

Looks good indeed, and should probably be applied (I do agree with Christian about weak vs. strong_validation, though). I even wonder why this wasn't done so in the first place. Then again, we did check If-Modified-Since, and this should already have been sufficient for .css and .js files, so this issue really is about browsers.

Another question: my patch changes most <script> tags into calls to add_script() (and <link> into add_stylesheet()). I assume this was historical, and the add_*() functions are the preferred method now, right?

[Christian Boos]

Replying to rblank:

Replying to osimons:

Sure, something like attachment:t9936-etag_files-r10542_012.diff​ was what I had in mind.

Looks good indeed, and should probably be applied

The caveat with ETag based on file timestamps for static resources is that it becomes ineffective and even detrimental to the performance in case multiple hosts are used to serve the static content, as the ETag will be different depending on which host happens to serve the static request (​Yahoo's performance tip about ETag). Probably not a common case in Trac deployments, but worth noting.

Another question: my patch changes most <script> tags into calls to add_script() (and <link> into add_stylesheet()). I assume this was historical, and the add_*() functions are the preferred method now, right?

Seems so, yes. However for some templates (e.g. source:trunk/trac/templates/diff_view.html​) which are used in several places, this means we will have to remember adding the same set of .css and .js each time (ok, here only diff.css, but you get the idea).

[Christian Boos]

Actually, it seems that the wide use of fingerprinting techniques is mostly because of this reason, making possible a "never expires" policy which drastically reduces the network load (​Yahoo's performance tip about Expires), rather than to circumvent buggy browser behavior (although this will also fix that problem, of course).

[osimons]

I see that fingerprinting is a recommended technique, but only when used as an argument in the filename or path for file. Many proxies and browsers are naturally restrictive about caching URLs with query strings. So contrary to intention, worst case could be that adding ?v=NNNN could actually increase both network load and request count by forcing a fresh copy each time.

[Remy Blank]

Replying to osimons:

I see that fingerprinting is a recommended technique, but only when used as an argument in the filename or path for file. Many proxies and browsers are naturally restrictive about caching URLs with query strings. So contrary to intention, worst case could be that adding ?v=NNNN could actually increase both network load and request count by forcing a fresh copy each time.

According to Google's paper, proxies wouldn't cache a Trac site anyway, as we don't send Cache-Control: public headers. But we can also find a way to move the version into the path, if required. Placing it in the query string just makes for a simple implementation.

[Christian Boos]

Replying to osimons:

I see that fingerprinting is a recommended technique, but only when used as an argument in the filename or path for file.

We could do that as well, though this would greatly complicate deployment.

Many proxies and browsers are naturally restrictive about caching URLs with query strings. So contrary to intention, worst case could be that adding ?v=NNNN could actually increase both network load and request count by forcing a fresh copy each time.

While it's probably true that this will disable caching by proxies ("​Don't include a query string in the URL for static resources"), I suppose it will already be a huge saving to rely only on browser caches. We should nevertheless verify this with all the major browsers.

I'm confident it's not a problem in practice, otherwise heavily optimized sites like Wikipedia wouldn't do it.

[Remy Blank]

There's one thing I still don't understand. We already process the If-Modified-Since header and return a 304 if it matches. What does the ETag bring (for static resources) that the If-Modified-Since header doesn't? And if the If-Modified-Since header isn't enough to avoid issues with CSS and JS caching, will ETag really help (in the patch, it will change at the same time as If-Modified-Since won't match anymore)?

[Christian Boos]

There might be browsers which don't react appropriately on Last-Modified: but could handle ETag: better… according to Google's ​page-speed doc:

Last-Modified is a "weak" caching header in that the browser applies a heuristic to determine whether to fetch the item from cache or not. (The heuristics are different among different browsers.)

Note that it seems that we currently process the If-Modified-Since: but only send a 304 back, not a Last-Modified: header. Maybe this is also a glitch, which will be fixed by osimons' patch.

But at the risk of restarting the whole thread, let me reiterate this is not the main issue, as the problem with browser unduly caching resources persists for at least IE and Opera even when all those headers are set properly (for example when it's lighttpd or Apache serving directly when htdocs_location is set).

[Remy Blank]

So… What's the currently-favored solution? I would really like to avoid having to explain to too many people how they should do Shift+Reload after upgrading to every new version.

[Christian Boos]

I'd say we should first focus on the initial issue (see comment:21 for the wrap-up) and leave alone for now or make a separate ticket for the possible changes (fixes or optimizations) for the ETag and other headers, which are of separate concern. If someone still needs to be convinced that it's not a separate issue, then please re-read comment:18 and comment:25 and try to convince me that it's the same issue.

As for the technical solution, I still think we need something along the lines of comment:12.

In trac/web/chrome.py:

class HeaderBuilder(object):
    def __init__(self, version=v):
        self.version = v
    def add_link(self, req, ...):
        ...
    def add_stylesheet(self, req, ...
...
tracheader = HeaderBuilder(version=trac.version)

# Backward compatibility API
add_link = tracheader.add_link
add_stylesheet = tracheader.add_stylesheet

In module code (e.g. trac/wiki/web_ui.py):

from trac.web.chrome import tracheader
...
    tracheader.add_stylesheet(req, 'common/css/diff.css')
    tracheader.add_script(req, 'common/js/diff.js')

For a plugin, it would be trivial to do something similar:

plugheader = HeaderBuilder(version=plug.VERSION)
...
    plugheader.add_stylesheet(req, 'plug.css')

[Remy Blank]

I would like to move forward with this, and implement URL fingerprinting, with the version in the path and not as a query argument (as suggested in the various resources listed above), together with a far-future expiry date. This will somewhat complicate serving static resources through the web server, but an AliasMatch directive (or similar) should solve this issue.

If we place the version at the beginning of the path (e.g. /chrome/0.13/common/js/jquery.js), this should even work with images referenced from Trac's .css files, as the latter use relative paths. The only issue would be referencing images provided by Trac from .css files provided by plugins. Any good ideas for this case?

[osimons]

I've got a few comments about this, and somewhat unsure about the consequences. I maintain much own code, and many plugins, and they all reuse Trac static resources in various ways. Not to mention the fact that I rewrite URLs for thousands of static locations to be served outside of Trac.

1. Embedding version in the path will be messy for anyone that deploys static resources. Particularly as some will have versioning, and some plugins may not, and for some locations like /site/ versioning will make no sense at all. Apache rewrite magic should still prevail - there are ways, it's just much more complicated.
2. The referencing between static files as you mention is another complex issue, particularly plugins to Trac resources which is most common need. Like in my Blog plugin having to get wiki.css styles, or code.css if needed, I reuse RSS icons and so on. Theme engines, extensions and general theme plugins will also have to handle more variation - extending both Trac and common plugins.
3. Every change in static resources won't necessarily be reflected in a version change. If you run Trac from a Git or Mercurial mirror (or svn export), the version may be reported as 0.13dev for close to ~2 years.
4. Just like I turn off web server signature, I'm not keen to advertise version of Trac and plugins that I run. Primarily for security, but also because it isn't something anyone with interest in my projects need to care about.
5. Applications such as news readers (RSS) stores the markup, and 'starred' or 'saved' items may be called upon to render itself again to user after versions have been upgraded. That won't work so well when none of the external resources can be found.

And more? Most likely. How could it be improved? Not sure, but I'd very much prefer it if the versioning was optional - like a trac.ini config value to enable it for Trac and plugins. When we don't control all the code, there may just be new holes and issues opened up that are beyond our ability to fix.

[chrome]
versioned_resources = false

[Remy Blank]

Replying to osimons:

I've got a few comments about this, and somewhat unsure about the consequences.

Yes, me too…

1. Embedding version in the path will be messy for anyone that deploys static resources. Particularly as some will have versioning, and some plugins may not, and for some locations like /site/ versioning will make no sense at all. Apache rewrite magic should still prevail - there are ways, it's just much more complicated.

Unfortunately, it's the only reliable way. Using the query string prevents caching, so it's definitely not a good solution.

2. The referencing between static files as you mention is another complex issue, particularly plugins to Trac resources which is most common need.

I know, that's how I became aware of the issue. I hadn't thought about theme engines, so it's even worse than I thought.

3. Every change in static resources won't necessarily be reflected in a version change. If you run Trac from a Git or Mercurial mirror (or svn export), the version may be reported as 0.13dev for close to ~2 years.

That's true. OTOH, people installing from a SVN checkout at least get a correct version.

We could also try to find another version-specific tag. For example, a single, global version number kept in the database, and incremented every time Trac or a plugin is updated. Or increment it with a trac-admin command, and mention it on TracUpgrade.

4. Just like I turn off web server signature, I'm not keen to advertise version of Trac and plugins that I run. Primarily for security, but also because it isn't something anyone with interest in my projects need to care about.

Good point. We could replace the version number with a hash of the version, but it wouldn't really be more secure, as you just would have to compare against the hashes of a few known versions. We could hash the version number with a secret key (specified in trac.ini, and auto-generated at install-time).

5. Applications such as news readers (RSS) stores the markup, and 'starred' or 'saved' items may be called upon to render itself again to user after versions have been upgraded. That won't work so well when none of the external resources can be found.

That's a bit an edge case, and it doesn't really work today, either. Though the resources are found, after upgrading they aren't the right ones.

But we could also simply ignore the version component when serving static resources. So you could ask for /chrome/1.2.3/common/js/jquery.js and you would still get the resource. The important part is that the links to the resources change.

And more? Most likely. How could it be improved? Not sure, but I'd very much prefer it if the versioning was optional

It's possible that trying to keep as much as possible of the current mechanism makes the problem harder than it is. Maybe we should re-think the serving of static resources from the beginning, and try to re-design it with versioning from the ground up.

Then again, I have the feeling that using a single, global version number, and having it auto-increment whenever the version of any component changes (checked at environment startup), could be doable. The links to static resources could then have the following form:

/chrome/123/common/css/trac.css
/chrome/123/tracfullblog/css/fullblog.css

This allows relative links from plugin resources to reference core resources without having to specify the version number. Trac should serve the same static resources for any version number. That is, all resources matching the following pattern should serve the same file:

/chrome/([0-9]+/)?common/css/trac.css

This would even be backward-compatible. I'm going to experiment with that idea and report back.

[Remy Blank]

Implemented fingerprinting for static resources.

[Remy Blank]

9936-static-fingerprint-r10741.patch​ implements fingerprinting on static resources:

• A hash of static resources is calculated by taking the SHA1 of the concatenation of the content of all available static resources, and taking the first 8 characters of the hex digest.
• Static resources are served at URLs matching the following pattern:

  /chrome/(![0-9a-f]+/)?(?P<prefix>[^/]+)/+(?<filename>.+)
  

  If the fingerprint is present and matches the current hash, an Expires: header is sent with a date one year in the future. Otherwise, no Expires: header is sent. Other than that, the fingerprint is ignored.
• Links to static resources are generated with the same pattern as above, with the hash determined at environment load time.
• If [chrome] htdocs_location is set, no fingerprint is added to links using it.

This has some nice properties:

• Fingerprinting is global and works for all resources below /chrome (core, plugins, site).
• The previous URLs to static resources remain active, so the mechanism is backward-compatible.
• CSS files from plugins can reference static resources from core, by using relative paths.
• If the content of any file in the static resources changes (core, plugin, site), or if a file is added or removed, the fingerprint changes. This may generate more reloads than strictly necessary, for example when disabling or activating a plugin, but such events should be relatively rare.
• For the new fingerprint to be calculated, Trac must be restarted. This is good practice anyway after an update (to core or a plugin), so it shouldn't be a limitation.
• Serving static resources from the web server is still reasonably simple, by using an AliasMatch directive or similar, and simply ignoring the fingerprint.

AFAICT, this implementation addresses all of the concerns in comment:38. Also, it would be very simple to add an option to disable fingerprinting altogether, if desired. I may have missed a few implications, but on the whole, I'm quite happy with it. Thoughts?

[Christian Boos]

Looks good indeed! My only concern so far is the time it will take to compute the static_hash… consider the ​TH:WikiExtrasPlugin for example and its collection of >3000 icons ;-)

Maybe we should only use stat information (file mtime and size), and forget about the pathological case of change of content without mtime and size modifications.

[Remy Blank]

Fingerprinting committed in [10769]. It can be controlled with the [trac] fingerprint_resources option:

• content: Calculate the fingerprint from the resource contents.
• meta: Calculate the fingerprint from the resource metadata (size and mtime).
• disable: Disable fingerprinting.

I have measured the time to calculate the fingerprint with the ​th:WikiExtrasPlugin installed, which is a bit of a worst case. On my machine, it takes 1.9 seconds when using the content, and 1.4 seconds when using the metadata. As this is only apparent during environment startup, I hesitate to even keep the "meta" option.

I will update the documentation with the new functionality, especially the part about serving static resources from the web server.