These are the old pages from the weblog as they were published at Cornell. Visit for up-to-date entries.

March 19, 2004

Feed Analysis: Requests vs. Bytes - Part II

There are two more questions I want an answer to: How much data is transferred per feed request (on average), and how much data is transferred per user. In both cases we categorize according to the number of return visits in the 60 day period.

The first graph shows a fairly regular beginning that matches intuition, less frequent visits results in a higher chance of a successful request, and as such the average number of bytes transferred per request is higher. This graph should behave more or less according to the results in the posting on success vs. nochange. However we see that starting at 300 visits a number of remarkable spikes occur. All of these spikes are triggered by users/agents that do not use a 'changed-since' HTTP request, and retrieve the full feed on every request.

If we look at the average data transferred per user we again see that the graph is completely dominated by a few high request rate users that retrieve the full feed every time (up to 1 MByte/day/user). Most of these have no agent string or use 'mozilla'. Except for two users: one is running Desktop Sidebar and the other is running SharpReader (sorry Luke). If I look at the other Desktop Sidebar users in the logs I see that all of their requests result in the full transfer of the feed, so it is a deficiency in the application. The single SharpReader user is completely out-of-whack with the rest of the SharpReader users as they all behave nicely, so I am not sure why this particular user gets the aggregator to behave so differently.

Posted by Werner Vogels at March 19, 2004 02:04 PM


hmm - very strange - conditional GET's have been part of SharpReader since the very first release. The only few reasons I can think of to explain this are all rather unlikely:
* a bug in one version of SharpReader that temporarily disabled conditional GET's (unlikely since that part of the code hasn't been touched in a long time, and I haven't had to go in there to fix anything either)
* some other application sending a SharpReader useragent, but why would anyone do that?
* a user's custom proxy stripping the http headers used for conditional gets

could you tell me the exact useragent that's causing this behavior?

Posted by: Luke Hutteman on March 21, 2004 01:07 AM