"Cache-Control: must-revalidate" means shared cache

[patrakov@…]

Trac sends a "Cache-Control: must-revalidate" header for its content. This is wrong, because it doesn't say that the response in some cases must be different for anonymous (error page) and authenticated (real content) users. Some ISPs here still force Squid 2.5-STABLE10 on the users as a transparent proxy, and it does cache error pages.

Thus, if an anonymous user behind such a proxy receives a "You are currently not logged in. You may want to do so now." message and clicks the "do so" link, it again receives the cached error message instead of the desired page after logging in. Shift+Reload solves this.

The correct header that indicates that the content is indeed different for different users is "Cache-Control: private". It does solve the squid problem. Please use it instead of "Cache-Control: must-revalidate".

P.S. Suggestion to do so via web server configuration is invalid, as lighttpd can only add, not replace, headers.

[Christian Boos]

Well, this seems to be really a bug in Squid. Your wording implies that newer versions don't do this caching and obey the 'must-revalidate' specification to the letter.

Specifically, in Cache-Control:

must-revalidate: … the cache MUST do an end-to-end revalidation every time, if, based solely on the origin server's Expires or max-age value, the cached response is stale.
​rfc:2616#section-14.9.4

and in Authorization:

If the response includes the "must-revalidate" cache-control directive, the cache MAY use that response in replying to a subsequent request. But if the response is stale, all caches MUST first revalidate it with the origin server, using the request-headers from the new request to allow the origin server to authenticate the new request
​rfc:2616#section-14.8

And yes, the response is stale, as we set the Expires: header to Fri, 01 Jan 1999 00:00:00 GMT.

Therefore I don't see how our usage of 'must-revalidate' is wrong.

The suggestion to use "Cache-control: private" is a bit strange, as not specifying it doesn't mean "public" either, otherwise I wonder what will happen in this setup with all the pages you will reach after authentication succeeds… will they end up being cached and shared as well?

[patrakov@…]

It may be a bug (or a common misconfiguration) in squid, but many ISPs here push it upon users (i.e.: it is beyond my control at least at work). It looks like there is some popular "preconfigured router" product that local ISPs use and that comes with such squid.

I didn't test newer squid versions. In fact, I have absolutely zero experience in installing and configuring squid.

In all tests below, I reverted my change to the Cache-Control header.

The "Expires:" header exists for the error page, but doesn't exist for the authenticated page. I.e., for the error page I get (both tests are on a port different from 80, so the ISP's squid is out of the way):

HTTP/1.1 403 Forbidden
Cache-Control: must-revalidate
Expires: Fri, 01 Jan 1999 00:00:00 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 3355
Date: Wed, 29 Sep 2010 05:34:10 GMT
Server: lighttpd/1.4.28

and for the real page I get:

HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Type: text/html;charset=utf-8
Content-Length: 7209
Date: Wed, 29 Sep 2010 05:34:21 GMT
Server: lighttpd/1.4.28

With Apache instead of lighttpd (again, on non-80 port), for the error page:

HTTP/1.1 403 Forbidden
Date: Wed, 29 Sep 2010 05:42:38 GMT
Server: Apache/2.2.16 (Debian)
Cache-Control: must-revalidate
Expires: Fri, 01 Jan 1999 00:00:00 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html;charset=utf-8

For the real content:

HTTP/1.1 200 Ok
Date: Wed, 29 Sep 2010 05:42:49 GMT
Server: Apache/2.2.16 (Debian)
Cache-Control: must-revalidate
Vary: Accept-Encoding
Content-Encoding: gzip
Keep-Alive: timeout=15, max=97
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html;charset=utf-8

For the "behind transparent proxy" case and Apache, here are the requests and responses (with hostname and path edited out) as seen from my computer. Here is for the initial load:

GET /trac/ HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; ru; rv:1.9.2.10) Gecko/20100915 Ubuntu/10.04 (lucid) Firefox/3.6.10
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: ru,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Cookie: trac_form_token=2a473f925dff553b59cfea7d; trac_session=aebbe21ece863cffb27d1d9d; (non-trac cookies)

HTTP/1.0 403 Forbidden
Date: Wed, 29 Sep 2010 05:47:06 GMT
Server: Apache/2.2.16 (Debian)
Cache-Control: must-revalidate
Expires: Fri, 01 Jan 1999 00:00:00 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Type: text/html;charset=utf-8
X-Cache: MISS from server.
Proxy-Connection: close

After attempting to log in:

GET /trac/ HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; ru; rv:1.9.2.10) Gecko/20100915 Ubuntu/10.04 (lucid) Firefox/3.6.10
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: ru,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://example.com/trac/login
Cookie: trac_form_token=2a473f925dff553b59cfea7d; trac_session=aebbe21ece863cffb27d1d9d; trac_auth=be3bede40d011214bf34839df640f268; (some non-trac cookies)

HTTP/1.0 403 Forbidden
Date: Wed, 29 Sep 2010 05:47:06 GMT
Server: Apache/2.2.16 (Debian)
Cache-Control: must-revalidate
Expires: Fri, 01 Jan 1999 00:00:00 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Type: text/html;charset=utf-8
Age: 5
X-Cache: HIT from server.
Proxy-Connection: close

From the server perspective, the initial load looks like this:

GET /trac/ HTTP/1.0
Host: example.com
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; ru; rv:1.9.2.10) Gecko/20100915 Ubuntu/10.04 (lucid) Firefox/3.6.10
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: ru,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7
Cookie: trac_form_token=2a473f925dff553b59cfea7d; trac_session=aebbe21ece863cffb27d1d9d; (non-trac cookies)
Via: 1.1 server.:80 (squid/2.5.STABLE10) Dr.Web patch 2
X-Forwarded-For: 10.80.1.55
Cache-Control: max-age=259200
Connection: keep-alive

HTTP/1.1 403 Forbidden
Date: Wed, 29 Sep 2010 05:47:06 GMT
Server: Apache/2.2.16 (Debian)
Cache-Control: must-revalidate
Expires: Fri, 01 Jan 1999 00:00:00 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Connection: close
Content-Type: text/html;charset=utf-8

The second request simply doesn't make it to the server. I.e., this version of squid doesn't understand "must-revalidate", at least for error pages.

As for your last question - I have asked a colleague who doesn't have a login in my trac to visit a password-protected wiki page after I did so (presumably, populating the squid cache). He got the login page, not my cached content, so my initial description of the problem looks wrong. Nevertheless, "Cache-Control: private" helps. From the server perspective, here is how this looks. My visit:

GET /trac/ticket/53 HTTP/1.0
Host: example.com
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; ru; rv:1.9.2.10) Gecko/20100915 Ubuntu/10.04 (lucid) Firefox/3.6.10
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: ru,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7
Cookie: trac_form_token=2a473f925dff553b59cfea7d; trac_auth=be3bede40d011214bf34839df640f268; (non-trac cookies)
Via: 1.1 server.:80 (squid/2.5.STABLE10) Dr.Web patch 2
X-Forwarded-For: 10.80.1.55
Cache-Control: max-age=259200
Connection: keep-alive

HTTP/1.1 200 Ok
Date: Wed, 29 Sep 2010 05:47:20 GMT
Server: Apache/2.2.16 (Debian)
Cache-Control: must-revalidate
Vary: Accept-Encoding
Content-Encoding: gzip
Connection: close
Content-Type: text/html;charset=utf-8

His visit:

GET /trac/ticket/53 HTTP/1.0
Host: example.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: ru,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Cookie: trac_form_token=79732a013289a562f449c77a; trac_session=6f251a098e1380145d9fc532; (non-trac cookies)
Via: 1.1 server.:80 (squid/2.5.STABLE10) Dr.Web patch 2
X-Forwarded-For: 10.80.1.30
Cache-Control: max-age=259200
Connection: keep-alive

HTTP/1.1 403 Forbidden
Date: Wed, 29 Sep 2010 05:47:42 GMT
Server: Apache/2.2.16 (Debian)
Cache-Control: must-revalidate
Expires: Fri, 01 Jan 1999 00:00:00 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Connection: close
Content-Type: text/html;charset=utf-8

[Thijs Triemstra <lists@…>]

More cache-control related stuff in #6367.

[Christian Boos]

Let's see if anyone else stumble upon this problem or if it's really a corner case.