@James: Both good points. We’ll probably be taking advantage of ETags for certain pages. The pages where the modified times have been important were our big performance bottlenecks, but now that those are taken care of, I’m going to look into integrating ETags in some places.

With respect to ETags, if you need to generate the entire page in order to calculate your ETag, you won’t see a huge benefit from their use (it would still cut down the bandwidth though). They help a lot when you have a way of calculating the ETag without generating the entire contents (e.g ViewVC uses revision numbers as ETags for a number of pages, which are cheap to check). That said, if the only identifier you have is a monotonically increasing date field, then using last-modified is probably okay. Just make sure there is no way for the date to go backwards …

@anon: ETAGS doesn’t magically solve caching issues and doesn’t give us the ability to save time server-side. ETAGS is certainly a good thing, but not a replacement for modified times (nor is modified times a replacement for ETAGS).

ETAGS are often generated based on the content of a document, which means performing all the queries needed and doing the templating needed to build that document. What happens is that when an ETAG is generated, it’s sent to the server the next time, and the server still has to do all the work to generate the content. The ETAG saves the transfer time, but none of the server time. In fact, it actually adds to it a little since we need to generate something based on the document (which in our case can be expense for diffs).

By making clever use of the If-Modified-Since header, we can shortcut all that. Given that ability, ETAGS aren’t that helpful to us, so long as we’re handling it in all places. We’re not yet, but we’re also not losing out on much since with the aggressive caching, the page transfer time is pretty small for the remaining pages.