40-60% of Yahoo's users have an empty cache experience

willwagner · on Sept 25, 2009

We noticed this after finding an error in our apache config such that some of our assets were set to be cached longer than we really wanted (30 days), much to our chagrin. It turned out that even though you tell the browser to keep something for a very long time, it most likely won't stay there.

Our general assumption was that it is due to the default cache size of browsers being tiny in comparison to the amount of data that users now consume, coupled with every webpage you visit putting stuff into the cache (rightly so).

I believe the default cache of firefox is still set at 50mb which seems very small.

Periodic · on Sept 25, 2009

The default size for Firefox 3.0.12 on Ubuntu is indeed 50mb.

A fresh reload of CNN.com: 649 KB A fresh reload of YouTube.com: 113 KB A fresh reload of CDW.com: 331 KB

That's still a lot of sites in a 50MB cache. though I imagine a single youtube video would go a long way towards filling it up.

Jakob · on Sept 25, 2009

Quoting the comments section:

    The best possible way to achieve cacheability of an object is to perform
    a server-side re-write of all linked content (images, scripts etc) and
    re-write the links to refer to a file name based off the MD5 hash of the
    file content.
    
    So, your link:
    http://us.i1.yimg.com/us.yimg.com/i/ww/beta/y3.gif
    is re-written:
    http://us.i1.yimg.com/never-expire/A31D5F12.gif
    Where A31D5F12 is the MD5 hash.

Does somebody use this convention? It seems logical but I haven't seen it wildly used.

wmf · on Sept 25, 2009

I first heard about this several years ago from a CDN; their proxy was automatically rewriting the links so the Web developer didn't have to do anything. I've never seen it used in the open source world.

redorb · on Sept 25, 2009

Does this mess up what the new 'semantic web markup' movement is pushing? ... (I.e. google's image search uses the image name as a signal to what the image might be, albeit a small signal)

lsb · on Sept 25, 2009

It shouldn't, you can just have something like static.example.com/picture-of-cat-playing-piano-0123456789abcdef0123456789abcdef.jpg

mahmud · on Sept 25, 2009

Don't rewrite the URL completely, just append your session-data, caching hash and other metadata to the URL path.

ubernostrum · on Sept 25, 2009

I've seen it used, yes. I don't personally use MD5, but I've used variations based on modification timestamps.

pohl · on Sept 25, 2009

So does this constitute an indirect measurement of the size of the cache-savvy porn-consuming population?

redorb · on Sept 25, 2009

Perhaps browsers could have a white list that the user could modify and have a select view sites avoid losing the cache even when clearing the cache from a menu option...

I think it might constitute :)

mahmud · on Sept 25, 2009

Firefox:

Tools->Start Private Browsing.

Comes handy when you're giving someone your laptop to surf and you don't want them polluting your history (it distracts me when firefox auto-completes something I don't recognize)