These are the old pages from the weblog as they were published at Cornell. Visit www.allthingsdistributed.com for up-to-date entries.

September 10, 2004

Once More: Polling does not Scale.

For those who are surprised by the bandwidth problems that the polling of RSS feeds is causing the MS folks, let me say it once more:

UNCONTROLLED POLLING OF RESOURCES DOES NOT SCALE!!

20 years of developing distributed systems have thought us this, but the RSS advocates appear deaf for these issues. The simple approach that works well with small feeds and limited number of subscribers does not work with large feeds and many subscribers.

Whatever duct tape you put on this problem (e.g. make the subscribers poll less frequently) will get ripped off again when the growth to the next order of magnitude will happen. And if atom/rss is indeed such a great thing, this growth is indeed something that will happen.

There are simple, temporary solutions that will help big (group) feeds, such as using the 'changed-since' information from the client to serve a feed with only items that have changed since that time. But this will not solve the real problem.

For these highly popular (big) feeds, there is a range of solutions available. Many of them revolve around having multiple sources serve the feed, whether that is a cache (yes make your feed cacheable, that will provide automatic throttling), a neighbor (through collaborative techniques), through service aggregators (such as pubsub.com or bloglines), etc.

I have said this before, and I hope it the ms experience will trigger people who are responsible for large sites to start working on alternative architectures.

Posted by Werner Vogels at September 10, 2004 03:24 PM
TrackBacks

If you must poll, at least poll well...
Excerpt: Microsoft's recent troubles with RSS files have focused the RSS/Atom communities squarely on fundamental problems with feed syndication as practiced today. While some of us have been warning about this problem for quite a while, it has taken Microsoft'...
Weblog: As I May Think...
Tracked: September 11, 2004 06:23 PM
RSS Reminder: Data ownership and Ways of Interactions in SOA
Excerpt: Recent posts about RSS (or more precisely uncontrolled polling) scalability issues, my Slashdot issue, and Jon's post about Next-generation infoware reminded me some process-awareness discussions on this blog (flying data, interaction patterns, data ow...
Weblog: Radovan Janecek: Nothing Impersonal
Tracked: September 14, 2004 07:42 PM
The state of Web services
Excerpt:
Weblog: XML Eye for the Object Guy
Tracked: September 29, 2004 09:49 PM
The state of Web services
Excerpt:
Weblog: XML Eye for the Object Guy
Tracked: September 29, 2004 10:16 PM
The state of Web services
Excerpt:
Weblog: XML Eye for the Object Guy
Tracked: September 29, 2004 10:17 PM
The state of Web services
Excerpt:
Weblog: XML Eye for the Object Guy
Tracked: September 29, 2004 10:23 PM
Comments

Since RSS/Atom were mostly developed by part-time volunteers, I wonder if they can afford to develop anything scalable. Of course, it goes without saying that anything scalable will be rejected by the "really simple" crowd. The question is how much leeway there is -- what level of expertise can we expect from an aggregator/publisher developer?

Posted by: Wes Felter on September 10, 2004 08:00 PM

> Since RSS/Atom were mostly developed by part-time volunteers,
So was the World Wide Web.

> I wonder if they can afford to develop anything scalable.
Volunteer != idiot.

> Of course, it goes without saying that anything scalable will be rejected by the "really simple" crowd.
Pro-active notifications systems are already in production and built by part-time volunteers. Look at what TrackBack 'ping' really can do. They are event notification over HTTP - notifications on the Web. And check out http://www.mod-pubsub.org/ (specifically http://www.mod-pubsub.org/kn_apps/knownews/?/what/syndic8.com/news/items if you have IE and scripting turned on)


> The question is how much leeway there is -- what level of expertise can we expect from an aggregator/publisher developer?
The developers are going to follow standards, and if those standards are built within a framework that lends itself to scalable approaches, then it will evolve to become scalable. Or the product will die.


Posted by: MikeD on September 11, 2004 02:11 AM

More on the issue today on CNet News:
http://news.com.com/Microsoft+flip-flop+may+signal+blog+clog/2100-1032_3-5368454.html

Posted by: David Oppenheimer on September 16, 2004 10:39 AM