All Things Distributed

The laptop that appeared on the cover of business week as part of the story “Building the Perfect Laptop” is the Thinkpad X300. It arrived at my doorstep this afternoon. It is everything it promised to be and more; superlight, rugged, SSD, full ports, wifi, lan & cell networks, dvd, replaceable batteries and 13.3" screen with 1440x900 graphics.

And all of this weighs in at 1420 grams.

It is amazing how light it is for a full featured laptop. See the unboxing pictures on flickr.

My opening slide for tomorrow's keynote at the MySQL Conference has this feel of speed and excitement to it that represents the current progress towards Cloud Computing. Persistent Storage for EC2 will be an important part of the presentation, but I'll mainly focus on general non-functional lesson from building large-scale services.

I would like to introduce to you the newest feature of Amazon EC2: Persistent local storage. This has been very high on the request list of EC2 customers and I believe that combined with the Availability Zones and Elastic IP Address features released earlier this month this makes EC2 the ideal environment for building highly scalable and reliable applications.

Significant innovation has gone into this feature: Instead of restricting developers to the use of a particular (distributed) file-system we once again decided to look at what is the most fundamental building block and how we could offer that in the most scalable and reliable manner.

Persistent storage for Amazon EC2 will be offered in the form of storage volumes which you can mount into your EC2 instance as a raw block storage device. It basically looks like an unformatted hard disk. Once you have the volume mounted for the first time you can format it with any file system you want or if you have advanced applications such as high-end database engines, you could use it directly.

Developers can create any number of volumes they want, in size ranging from 1 GB to 1TB. This volume will be created within a specified Availability Zone and will be accessible by your EC2 instances running in that Availability Zone. As to be expected with a volume abstraction only one instance can have the volume mounted at any given time. Volumes can migrate and be reattached to other instances if necessary for failure handling or application migration reasons.

The consistency of data written to this device is similar to that of other local and network-attached devices; it is under control of the developer when and how to force flush data to disk if you want to bypass the traditional lazy-writer functionality in the operating systems file-cache. Because of the session oriented model for access to the volume you do not need to worry about eventual consistency issues.

Snapshots

If we would have stopped here that would have already been quite a solid service for developers to use. We realized we needed to do more to make sure that developers could build truly geo-scalable applications. For that we introduced snapshot functionality: you ask the EC2 to make a snapshot of your volume and store it into Amazon S3. You can use this for long term backup purposes, for use in rollback strategies, but also for (world-wide) volume re-creation purposes.

When you create a volume you can ask it to be created from a particular snapshot. And because this snapshot is stored in S3, which is accessible in all Availability Zones, your new volume can be created in any zone, not just the one where the snapshot originated from.

The snapshot is extremely powerful technology and allows for building highly fault-tolerant applications operating world-wide. Combine these snapshots with Availability Zones and Elastic IPs and you have all the tools to manage and migrate even the most complex of applications.

And the great thing is it that it is all done with using standard technologies such that you can use this with any kind of application, middleware or any infrastructure software, whether it is legacy or brand new.

Early access

This new functionality is already being used privately by a handful of customers, and will be publically available later this year. We are talking about this service at this early stage because we believe this will help many of our EC2 customers with setting their development priorities for this year.

You can find more information at the AWS developer’s blog.

update:Thorsten from RightScale, who has been using the service, writes about his experiences

Today Amazon Web Services launched two new features in Amazon EC2 that are essential tools in building highly resilient applications: Elastic IP addresses and Availability Zones.

In summary:

• Elastic IP addresses are associated with a customer account and allow the customer to do its own dynamic mapping of IP address to instance. Using this dynamic mapping applications can remain reachable even in the presence of failures. This is an area where for example DNS reconfiguration is too slow a technique.
• Availability Zones allow the customer to specify in which location to launch a new EC2 instance. The world is divided up into Regions and a Region can hold multiple Availability Zones. These Zones are distinct locations within a region that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same region. By launching instances in separate Availability Zones, applications can be protected from failure of a single location.

With these two features EC2 customers now have tools to build applications that can tolerate a wide range of failure scenarios.

For more details visit the EC2 detail page and the Forum announcement

Update: excellent articles by the guys at RightScale: Using Elastic IP in switch-over scenarios, using Availability Zones to set up a fault-tolerant site and combining Elastic IP and Availability Zones.

While the past months have been relatively quiet there is now a period coming up with public events that will take me across a few continents. The period already started two weeks with a Distinguished Lecture at the School of Computer Science of CMU. I had a wonderful day meeting many academics to discuss the relevance of particular research subjects for companies such as Amazon. I believe that Peter Lee, the Head of Computer Science, is the only Head/Chair/Dean of a Computer Science department who maintains a weblog. It is worth reading for a different perspective.

Last week there was a very different event: A fireside chat at the close of the Under the Radar conference. During the day many startups had 5 minute slots to present their unique proposition to VC Judges and the audience. I found that there were two frequently asked questions: 1) Are you using Amazon Web Services and if not, why not? 2) How is this different from what Amazon is doing….?

In the coming weeks there will be a number of events I will attend, but two stand out: The Next Web Conference in Amsterdam and the MySQL Conference in Santa Clara. The latter will be the first major public appearance of Mårten Mickos and Jonathan Schwartz since the acquisition of MySQL by Sun Microsystems. I am looking forward to hearing their vision for the products. I have the honor to follow them with a presentation on lessons learned from building scalable state management tools for Amazon.com.

The Next Web Conference 2008 is one I am really looking forward to. The fact that it is in my old hometown may have something to do with that. But mostly it is because Patrick and crew are working really hard, and succeeding very well in my eyes, to make this an Event instead of a Conference. There are many different activities with some crossovers into art, comedy and entertainment. The location is great: The Wester Gasfabriek (Gas factory) was built around the start of the previous century to produce gas out of coal. Although the factory and surroundings had been prime examples of brownfields, they have since 2000 been transformed into a magnificent park. The restored raw industrial atmosphere of the buildings makes it a very special location for large events such as The Next Web 2008.

Update: If you want a 35% discount on attending The Next Web, head over to Boris' blog for a discount code.

A few days ago, on March 14, Amazon S3 quietly celebrated its 2nd birthday. I think congratulations are in order and I certainly wish the service “many happy returns of this day”. That S3 is growing up fast is obvious from the number of objects the customers trust us with.

On Thursday I’ll be on stage at the Under the Radar conference for a fireside chat with Robert Scoble. The Under the Radar folks have asked for input into what questions Robert should ask me. The chat will be focused on cloud computing and related topics, so if you have questions, post them on their blog.

Today I finished processing my first set of High Dynamic Range images. HDR is where you take 3 shots of an object using automatic exposure bracketing, in this case at [-2, 0, +2]. Each of the images will have details of areas (highlight, shadows) that the others do not have. You then combine the 3 images into an HDR image using software such as PhotoMatix Pro. The resulting combined image has such depth that it cannot be displayed and you have to use a technique called tone mapping to bring it back into the 16 bit space.

These are only my first attempt and they are greatly handicapped by the fact that I didn’t use a tripod, which means that there are movement artifacts between the images. The shots are from a weekend in Tallinn, Estonia where I visited my daughter who is an exchange student at the Estonia Academic for Music and Theater for the spring, from her London school. It was still winter weather so the sun was bright but low on the horizon giving shadows that were ideal for HDR processing.

The complete result can be found in the flickr set.

I was going to pick up posting again and what better way to do that than to point to today's announcement by Usenix to open up the access to all of their conference proceedings. Compared to the IEEE Computer Society and the ACM, who still hide the published material behind the walls of their digital libraries, Usenix already had a very liberal one-year-members-only policy. But now their publications will be available for everyone, instantly.

I am a Usenix, ACM & IEEE member and will remain one, so it doesn't make much of a difference to me, but I believe all professional organizations would serve their members better by making their publications freely available to public.

For those of you who always wanted an Amazon robot
you can now buy this special Amazon version of the
Danboard character from the Yotsuba & ! series.