March 2006 Archives

Dennis Howlett reminds me that the last paragraph of my Naked Answers post suggests that I think the only way innovation happens is in reaction to customer demand. Customer feedback is an important ingredient, but not of course not the only component. Henri Fords famous quote “If I had asked people what they wanted, they would have said, "Faster horses.”" signifies that to make a real difference revolutionary thinking is required. I live on the balance of these two components every day.

Scalability is frequently used as a magic incantation to indicate that something is badly designed or broken. Often you hear in a discussion “but that doesn’t scale” as the magical word to end an argument. This is often an indication that developers are running into situations where the architecture of their system limits their ability to grow their service. If scalability is used in a positive sense it is in general to indicate a desired property as in “our platform needs good scalability”.

What is it that we really mean by scalability? A service is said to be scalable if when we increase the resources in a system, it results in increased performance in a manner proportional to resources added. Increasing performance in general means serving more units of work, but it can also be to handle larger units of work, such as when datasets grow.

In distributed systems there are other reasons for adding resources to a system; for example to improve the reliability of the offered service. Introducing redundancy is an important first line of defense against failures. An always-on service is said to be scalable if adding resources to facilitate redundancy does not result in a loss of performance.

Why is scalability so hard? Because scalability cannot be an after-thought. It requires applications and platforms to be designed with scaling in mind, such that adding resources actually results in improving the performance or that if redundancy is introduced the system performance is not adversely affected. Many algorithms that perform reasonably well under low load and small datasets can explode in cost if either requests rates increase, the dataset grows or the number of nodes in the distributed system increases.

A second problem area is that growing a system through scale-out generally results in a system that has to come to terms with heterogeneity. Resources in the system increase in diversity as next generations of hardware come on line, as bigger or more powerful resources become more cost-effective or when some resources are placed further apart. Heterogeneity means that some nodes will be able to process faster or store more data than other nodes in a system and algorithms that rely on uniformity either break down under these conditions or underutilize the newer resources.

Is achieving good scalability possible? Absolutely, but only if we architect and engineer our systems to take scalability into account. For the systems we build we must carefully inspect along which axis we expect the system to grow, where redundancy is required, and how one should handle heterogeneity in this system, and make sure that architects are aware of which tools they can use for under which conditions, and what the common pitfalls are.

Today Shel Israel and Robert Scoble stopped by at Amazon to present their book Naked Conversations in our Fishbowl series. As you can read in Shel's observations and Robert’s they appear shocked that we used a critical voice to address their work. Welcome to life at Amazon, we set a very high bar for our own works and we expect anyone that comes to sell us an approach to actually be prepared to really defend their ideas. Just because blogs are cool and everybody is doing them does not automatically mean that we should institutionalize them at Amazon. We have a long history of promoting customers to use their voice about our products and our operations, so if you come to Amazon to tell us our business is going to really suffer if we do not blog, you better be prepared to defend your ideas with very strong arguments and hard evidence. We expect that from anyone, externally or internally, who wants to promote an idea within Amazon.

This was my approach with challenging Shel and Robert at our lunch meeting. I wanted them abandon their fuzzy group hug approach, and counter me with hard arguments why they were right and I was wrong. Instead they appeared shell-shocked that anyone actually had the guts to challenge the golden wonder boys of blogging and not accept their religion instantly. I have been a promoter of weblogging for a long time, so I didn't feel particularly bad to challenge these two authors to tell me why customers would get a better Amazon product if we would institutionalize blogging at a wider scale around Amazon. Beyond "a more human face" and "conversations with individuals from Amazon" there was no real response how blogging will make the product named Amazon.com better for our customers given all the techniques we already use from soliciting customer feedback to discussion forums to snooping weblogs and comments sites, etc,. In my mind they had no solid data-driven answers to these challenges, which I would have expected from two seasoned evangelists. I myself actually knew some of the answers to my questions, but I was surprised to see that these guys were not prepared enough to slap me around with solid answers.

What I am a bit disappointed by is that these two smart guys did not understand what was going on, even though I clearly said that I was challenging them to a response. And they make the suggestion in their responses (at least Shel does) that Amazon doesn't "get" blogging. They are wrong. Amazon is a long time pioneer in the space of involving their customers with our product. And we really listen to our customers; any Amazon employee who encounters an issue on a forum or weblog or at any other place is empowered to escalate those issues internally immediately until they get fixed. Customer feedback is essential for Amazon and we will use all effective means to get it.

Shel wrote:

I left with the personal sense that it will be a tropical day in Seattle before any blogging between companies ad customers is forthcoming from Amazon.

Well it will certainly not happen because Shell and Robert convinced us with solid evidence of the tremendous benefits. If it does happen at a wider scale than it happens now, it will be because our customers have given us feedback that they think blogging is an excellent approach to interact with Amazon. Amazon will continue to innovate with involving customers with our sites, some of those may be weblog or wiki related techniques, many of those will be completely new approaches as people have come to expect from Amazon. We will do this because our customers want us to, not because "everybody else is doing it".

Update: In various comments and follow-up posts the authors express that they feel the questions at the presentation were rude, and that my response to their description of the meeting is not accurate. I am very sorry they feel that way, and I am sure there were people in the room who agree with them. I hope they do not hold my (perceived) rudeness as the test against which to hold those Amazonians. That said, I have also heard enough views by now that do not support Shel & Robert’s recantation of the event, and there were several people who feel it was appropriate to question them hard and deep. Most people seem to agree that my line of questioning was somewhat unforgiving when I felt they didn’t come up with the right answers. I promise to be nicer to our next guests …

Update 2 I think I prefer Nick Carr's characterisation: refreshingly blunt.

Update 3 If you read the posting well, you will find that ROI does not appear in it. That is on purpose as the questions about innovation and about providing value need to be answered first before you can talk about at what cost.

Go check out the Amazon Simple Storage Service (S3). This is Amazon.com’s Internet scale storage service available through a web service interface. S3 is another example of the Amazon Web Services mission: to expose all of the atomic-level pieces of the Amazon.com platform. Providing scalable, reliable, secure and fast storage is something that Amazon developers have already enjoyed for some time and now it is available to the developer community outside of Amazon.com.

S3 has a simple programmatic read/write interface through REST and SOAP services, and the data can be access through different distribution protocols. HTTP and BitTorrent are provided in this first release.

Congratulation to the S3 team!