All Things Distributed

In the recent years Mike Stonebraker has been advocating that the current commercial databases have become one-size-fits-all tools that are so general and heavy-weight that they do not excel in any particular area. Mike has written some papers on this topic of comparing general databases with specialized storage engines in areas of data-warehousing, text search, stream processing and scientific processing. In these papers he demonstrates that these specialized engines can run orders of magnitude faster than the commercial databases, using commodity hardware. I am not sure all the comparisons are fair, but he makes a compelling case. Of course Mike is a principal in Vertica which directly competes with what he calls “The Elephants”. Given that I have been close in the past to another example about how commercial goals can cloud academic judgment, am I rather cautious in evaluating these papers.

This week at VLDB Mike gave an invited talk on this topic in the Industrial Research track. His talk centered on that while he had previously proven that specialized approaches could run circles around the Elephants, he now could also demonstrate that OLTP, which is the bread-and-butter of the database industry, could be greatly outperformed by new architectures. In the paper and in his talk he put out a challenge to the research community, and to graduate students in particular, to take a particular interesting application area and build specialized solutions that provide at least a 50X improvement over the current products.

I like this challenge, given that 50X is likely to be able to make impact, where 2-4X in general can be easily compensated for by the next generation hardware. But something bugs me about the challenge and also about some of the demonstrations in the papers; 50X is still focused on scaling-up, just as many of the current database systems do, instead of scaling out, which is what the world really needs. The evidence in the paper is indeed about single box performance. This continuing N=1 thinking will never yield systems that can break through the current scalability limitations of enterprise software, regardless whether it runs 50 times faster or not.

If I could rewrite the challenge I would go for asking for “demonstrating performance at scale”; That one can achieve the rock solid performance and reliability guarantees by just incrementally increasing the components in the system, without any limitations. And every scaling axis needs to be satisfied; request rates, request complexity, data sets size, etc.

Only focusing on 50X just gives you faster Elephants, not the revolutionary new breeds of animals that can serve us better.

Posted by Werner Vogels at 6:32 PM | Permalink | Comments (0) | TrackBacks (0)

When I was building one of my first teams at Amazon, one that had to work on some really advanced distributed systems technology I put up a job description on this weblog. I was certainly pleased with the responses. Last year at a conference I heard from some of my former academic colleagues that they were using this description to educate their students abput where they were lacking in knowledge or experience. “Werner’s requirements” were used to explain to them that if they wanted to work on one of the really interesting distributed systems of this world, they better recognize the importance of [insert some random topic]

There are only a few engineers at Amazon who work directly for me and I currently have a job opening for such a position. It has most of the requirements from the previous description, but in this role there is more emphasis on analysis and modeling. If you are interested and you feel you qualify you can apply online at the Amazon careers site for job # 025213 or send email to the address on the right column of this page.

This is a job with some really tough requirements but it is an important job as you will have direct influence into how systems are build at Amazon and that is something we take very serious.

Senior Research Engineer in the Office of the CTO

Amazon.com's website is the most well-known front end to one of the world's largest and busiest service-oriented architectures. Its systems requirements are very challenging: maintain high-availability and guaranteed performance in an ultra-scalable fashion while being very cost efficient. From webpage rendering to order pipeline workflow, from data-warehouse to distributed caching, all require unique solutions. Many of these solutions require significant innovation: often these challenges have not even been addressed in research in a production setting at the scale of Amazon.

Continue reading "Job Opening for a Senior Research Engineer " »

Posted by Werner Vogels at 4:41 PM | Permalink | Comments (1) | TrackBacks (0)

Carbonado is an extensible, high performance persistence abstraction layer for Java applications, providing a relational view to the underlying persistence technology. Persistence can be provided by a JDBC accessible SQL relational database, or it can be a BDB. It can also be fully replicated between the two.

Even if the backing database is not SQL based, Carbonado still supports many of the core features found in any kind of relational database. It supports queries, joins, indexes, and it performs query optimization. When used in this way, Carbonado is not merely a layer to a relational database, it is the relational database. SQL is not a requirement for implementing relational databases.

The Amazon engineers who collaborated on developing Carbonado over time received feedback that there was a lot of interest in the developer community outside of Amazon for this technology. We decided to release this software through an open source process for other developers to use, improve and extend.

Congratulation to Brian and his close collaborators for developing excellent technology and putting in the work to make it publicly available and to Don, Ryan and Stephanie for navigating all the legal and other obstacles to make this a reality.

Posted by Werner Vogels at 6:38 PM | Permalink | Comments (0) | TrackBacks (0)

Last week I gave a keynote at the ACM Principles of Distributed Computing Conference (PODC) on the topic of technology transfer. My choice of topic was triggered by recent presentations by a number of other research luminaries, who had remarked that the distributed computing research community had failed to make its mark; lots of good ideas, little impact.

I subscribe to a longer term point of view when it comes to the transformation of technology into successful products. Richard Gabriel created a model that lays out the time it takes and means by which innovations become successful consumable products[1]. It is certainly fits the market success of a number of the Xerox Parc innovations, the spreadsheet, or even the Web. To illustrate, hyperlinks and markup languages were developed in the mid sixties, the tcp/ip based networks came to life in the seventies, and it wasn’t until the mid nineties before the combination of these three turned into the basis for a mass consumer product. Gabriel’s presentation on “Models of Software Acceptance: How Winners Win” has more examples and a better connection to the “Crossing the Chasm [2]” style of thinking.

I believe this applies to much of the deep, fundamental, distributed systems material as well. Felipe Cabrera has said, for example, that when Vista ships next year with the support of fine grained transactions in programming languages, it will have been more than 20 years after the concepts where developed in the Quicksilver project at IBM.

But things are accelerating. Amazon.com and other places use high-quality, massively distributed systems to their advantage and require the use of recent research technology to exploit this massive scale. We see an increased adoption of advanced distributed systems beyond established techniques such as edge caching, lazy processing, fusions & aggregation, etc.

Adoption of more recent research technology for use in products is not a walk in the park. Engineers have to be very determined to overcome the many roadblocks that come with early adoption.

Unrealistic assumptions

Research is focused on the details of the technology itself, and not very focused on the application context of the technology. Often, to be able to make progress in research, you need to restrict the environment it can be applied to. For example, many academics will confess to have made the assumption that failures of component are not correlated. This absolutely unrealistic assumption will come back to haunt you in real life, where failures frequently are correlated, as they are often triggered by external or environmental events.

When selecting research technology, it is often a major exercise to discover what exact assumptions the researcher made. Then, the even more difficult exercize is figuring out whether you can live with those assumptions, whether the assumptions were relevant at all, or whether the may impede the adoption of the technology. And in the latter case, whether there is something we can do to bring the research to more realistic standards.

Uncertainty

Many of the insurmountable assumptions deal with reasoning away uncertainty. By turning life into a state machine in which no surprises can be found, one has the perfect world in which everything is clean and organized. There is a limit to how much you can trick life into being predictable and how much control you think you will have to keep life in-check. At small scale you may succeed, but when your systems grow in size and complexity you will lose control. As such, building scalable systems is all about letting go of control. (Turing’s Type I organizations)

In Control Theory, for a long time, researchers were convinced that practitioners did not want to use their research because there was too much complex math in it. It turned out however that the research was largely irrelevant in practice because it didn’t model a realistic world. The moment researchers started to produce work that explicitly took uncertainty into account, their work was rapidly adopted by engineers and architects. Ironically the math has only grown more complex…

In distributed systems we see a similar pattern arising; research which realistically models uncertainty is more readily useful for adoption. Randomization and self-organizing systems are crucial techniques for scaling systems in the real world.

A perfect world

The last topic I want to mention is the use of academic publications as a source of technology selection. Academics often battle out subtle competing views in their research papers. But if there are at least 10 competing approaches to implementing consensus in distributed systems, an engineer needs to make a judgment call on which approach would be best to solve his problems. If the academics can’t even make their mind up on what appears to be the right way, how can their customers be expected to do this for them.

Papers are often written in an extremely positive manner: “This is, once again, the best improvement to life since sliced bread”. There is hardly any self-criticism. There certainly are no details about the things that didn’t work, and why they didn’t work. And let’s not even start talking about the use of statistics in system research papers. Do we really only care about averages? That is, of course, assuming the experiments were realistic in the first place.

You need to re-execute a number of the most promising research achievements in realistic settings to help your selection. There is no way around it. Which means that these research achievements will only be considered if an engineer really needs these results because it will be very time consuming.

Occam’s razor

This is an occasion where we actually use Occam’s Razor in its original sense; if two approaches produce the same result, select the one with the fewest assumptions. We have seen frequently that this selection criteria will lead you to the technology that has the greatest likelihood of being adopted.

entia non sunt multiplicanda praeter necessitatem

[1] Gabriel, Richard, "Money through Innovation Reconsidered" in Patterns of Software: Tales from the Software Community, Oxford University Press, USA; Reprint edition (May 1, 1998) (download book pdf).

[2] Moore, Geoffrey A., “Crossing the Chasm: Marketing and Selling High Tech Products to Mainstream Customers” HarperBusiness; Rev edition (July 1999)

Posted by Werner Vogels at 11:38 AM | Permalink | Comments (1) | TrackBacks (0)

Building systems that can guarantee performance and availability while scaling up to handle exponential growth in datasets and user requests is still very much a Dark Art. It is an art we master quite well by now at Amazon but it took a lot of growing pains to get to this level of sophistication.

At least someone at Technorati has a sense of humor about their pains…

tags: Technorati, Distributed Systems, Scalability,

Posted by Werner Vogels at 9:57 PM | Permalink | Comments (1) | TrackBacks (0)

In the A Word On Scalability posting I tried to write down a more precise definition of scalability than is commeonly used. There were good comments about the definition at the posting as well as in a discussion at The ServerSide.

To recap in a less precise manner I stated that

• A service is said to be scalable if when we increase the resources in a system, it results in increased performance in a manner proportional to resources added
• An always-on service is said to be scalable if adding resources to facilitate redundancy does not result in a loss of performance.
• A scalable service needs to be able to handle heterogeneity of resources.

There were quite a few comments about the use of performance in the definition. This is how I reason about performance in this context: I am assuming that each service has an SLA contract that defines what the expectations of your clients/customers are (SLA = Service Level Agreement). What exactly is in that SLA depends on the kind of service business you are in; quite a few of the services that contribute to an Amazon.com website have an SLA that is latency driven. This latency will have a certain distribution and you pick a number of points on the distribution as representatives for measuring your SLA. For example at Amazon we also track the latency at the 99.9% mark to make sure all of all customers are getting an experience at SLA or better.

This SLA needs to be maintained if you grow your business. Growing can mean increasing the number of requests, increasing the number of items you serve, increasing the amount of work you do for each request, etc. But no matter along which axis you grow, you will need to make sure you can always meet your SLA. Growth along some axis can be served by scaling up to faster CPUs and larger memories, but if you keep growing there is an end to what you can buy and you will need to scale out. Given that scaling up is often not cost effective, you might as well start by working on scaling out, as you will have to go that path eventually.

I have not seen many SLAs that are purely throughput driven. It is often a combination of the amount of work that needs to be done, the distribution in which it will arrive and when that work needs to be finished, that will lead to a throughput driven SLA. Latency does play a role here as it is often a driver for what throughput is necessary to achieve the output distribution. If you have a request arrival distribution that is non-uniform you can play various games with buffering and capping the throughput at lower than you peak load as long as you are willing to accept longer latencies.  Often it is the latency distribution that you try to achieve that drives you throughput requirements.

There were some other points made with respect to what should be part of a scalability definition, among others by Gideon Low @ the serverside thread (I tried to link to his individual response but seem to fail) who make some good points.

• Operationally efficient – It takes less human resources to manage the system as the number of hardware resources scales up.
• Resilient – Increasing the number of resources will also increase the probability of failure of one of those resources, but the impact of such a failure should be reduced as the number of resource grows.

These two points combined with a discussion about cost/capacity/efficiency should be part of a definition of a scalable service. I’ll be thinking a bit about what the right wording should be and will post a proposal later.

Posted by Werner Vogels at 2:50 PM | Permalink | Comments (3) | TrackBacks (1)

Scalability is frequently used as a magic incantation to indicate that something is badly designed or broken. Often you hear in a discussion “but that doesn’t scale” as the magical word to end an argument. This is often an indication that developers are running into situations where the architecture of their system limits their ability to grow their service. If scalability is used in a positive sense it is in general to indicate a desired property as in “our platform needs good scalability”.

What is it that we really mean by scalability? A service is said to be scalable if when we increase the resources in a system, it results in increased performance in a manner proportional to resources added. Increasing performance in general means serving more units of work, but it can also be to handle larger units of work, such as when datasets grow.

In distributed systems there are other reasons for adding resources to a system; for example to improve the reliability of the offered service. Introducing redundancy is an important first line of defense against failures. An always-on service is said to be scalable if adding resources to facilitate redundancy does not result in a loss of performance.

Why is scalability so hard? Because scalability cannot be an after-thought. It requires applications and platforms to be designed with scaling in mind, such that adding resources actually results in improving the performance or that if redundancy is introduced the system performance is not adversely affected. Many algorithms that perform reasonably well under low load and small datasets can explode in cost if either requests rates increase, the dataset grows or the number of nodes in the distributed system increases.

A second problem area is that growing a system through scale-out generally results in a system that has to come to terms with heterogeneity. Resources in the system increase in diversity as next generations of hardware come on line, as bigger or more powerful resources become more cost-effective or when some resources are placed further apart. Heterogeneity means that some nodes will be able to process faster or store more data than other nodes in a system and algorithms that rely on uniformity either break down under these conditions or underutilize the newer resources.

Is achieving good scalability possible? Absolutely, but only if we architect and engineer our systems to take scalability into account. For the systems we build we must carefully inspect along which axis we expect the system to grow, where redundancy is required, and how one should handle heterogeneity in this system, and make sure that architects are aware of which tools they can use for under which conditions, and what the common pitfalls are.

Posted by Werner Vogels at 5:21 PM | Permalink | Comments (7) | TrackBacks (3)