The backend servers that power the world of Internet Services have become increasingly diverse. With today's announcement that Microsoft Windows Server is available on Amazon EC2 we can now run the majority of popular software systems in the cloud. Windows Server ranked very high on the list of requests by customers so we are happy that we will be able to provide this.
One particular area that customers have been asking for Amazon EC2 with Windows Server was for Windows Media transcoding and streaming. There is a range of excellent codecs available for Windows Media and there is a large amount of legacy content in those formats. In past weeks I met with a number of folks from the entertainment industry and often their first question was: when can we run on windows?
There are many different reasons why customers have requested Windows Server; for example many customers want to run ASP.NET websites using Internet Information Server and use Microsoft SQL Server as their database. Amazon EC2 running Windows Server enables this scenario for building scalable websites. In addition, several customers would like to maintain a global single Windows-based desktop environment using Microsoft Remote Desktop, and Amazon EC2 is a scalable and dependable platform on which to do so.
Amazon EC2 with Windows Server is still currently in private beta testing, but will be available for general use before the end of the year. Keep an eye on the AWS Weblog for information about Amazon Web Services at the Microsoft Professional Developer Conference.
The last week for submitting the applications for the AWS Startup Challenge has started. Looking at the proposals that are being submitted it looks like this will be another very inspiring challenge. These proposals are reviewed by a panel and five finalists will be selected. The finalists will come to Seattle to compete for $50K in cash, $50K in AWS credits, 2 years of Premium Support and more. All finalists will receive Rightscale Premium for 6 months and there will be a number of promotional events that includes all the finalists.
Last year there were 900 applications which made for very intense proposal reading sessions. Eventual Ooyala won the challenge and got to smash the server to bits. The videos of last year's finalists are still online.
If you have a brilliant idea/business that we should be evaluating you have until October 10 to let us know.
For many the "Cloud" in Cloud Computing signifies the notion of location independence; that somewhere in the internet services are provided and that to access them you do not need any specific knowledge of where they are located. Many applications have already been built using cloud services and they indeed achieve this location transparency; their customers do not have to worry about where and how the application is being served.
However for developers to do their job properly the cloud cannot be fully transparent. As much as we would like to make it easy and simple for everyone, building high-performance and highly reliable applications in the cloud requires that the developers have more control. For example a reality is that failures can happen; servers can crash and networks can become disconnected. Even if these are only temporary glitches and are transient errors, the developer of applications in the cloud really wants to make sure his or her application can continue to serve customers even in the face of these rare glitches. A similar issue is that of network latency; as much as we would like to see the cloud to be transparent, the transport of network packets is still limited to the speed of light (at best) and customers of cloud applications may experience a different performance depending on where they are located in relation to where the applications are running. We have seen that for many applications that works just fine, but there are developers who would like more control over how their customers are being served and for example would like to give all their customers low latency access, regardless of their location.
At Amazon we have been building applications on these cloud principles for several years now and we are very much aware of the tools that developers need to build applications that are required to meet very high standards with respect to scalability, reliability, performance and cost-effectiveness. We are also listening very closely to the feedback AWS customers are giving us to make sure we expose the right tools for them to do their job. We launched Amazon S3 in Europe to ensure that developers could build applications that could serve data out of a European storage cloud. We launched Regions and Availability Zones (combined with Elastic IPs) for Amazon EC2 such that developers would have better control over where their applications would be running to ensure high-availability. We are now ready to expand the cloud even further and bring the cloud storage to its customers' doorstep.
Today we are announcing that we are expanding the cloud by adding a new service that will give developers and businesses the ability to serve data to their customers world-wide, using low-latency and high data transfer rates. Using a global network of edge locations this new service can deliver popular data stored in Amazon S3 to customers around the globe through local access.
We have developed this content delivery service using the robust AWS principles we know work well for our customers:
- Cost-effective: no commitments and no minimum usage requirements. You only pay for what you use in a manner similar to the other Amazon Web Services.
- Simple to use: one API call gets you going. You store the data you want to distribute in an Amazon S3 bucket and you use this API call to register this bucket with the content distribution service. The registration will provide you with a new domain name that you can use in url's to access the data through this service with HTTP. When your customer accesses your content through your new url the data it refers to will be delivered through a network of edge servers.
- Works well with other services: The service integrates seamlessly with Amazon S3 and the data/content served through the service can be accessed using the standard HTTP access techniques.
- Reliable: Amazon S3 will give you durable storage of your data, and the network of edge locations on three continents used by the new service will deliver your content to your customers from the most appropriate location.
This is an important first step in expanding the cloud to give developers even more control over how their applications and their data are served by the cloud. The service is currently in private beta but we expect to have the service widely available before the end of the year. You can get a few more details and sign up to get notified when the service is becoming on this AWS page Also check Jeff Bar's posting on the AWS weblog.
Today marks the launch of Amazon EBS (Elastic Block Store), the long awaited persistent storage service for EC2. Details can be found on the EC2 detail page, the press release and Jeff Barr's posting over on the AWS evangelists blog. Also the folks at Rightscale have two detailed postings: why Amazon EBS matters and Amazon EBS explained.
With the launch of the Elastic Block Store we complete an important milestone in offering a complete suite of storage solutions as part of the Amazon Infrastructure Services. Back in the days when we made the architectural decision to virtualize the internal Amazon infrastructure one of the first steps we took was a deep analysis of the way that storage was used by the internal Amazon services. We had to make sure that the infrastructure storage solutions we were going to develop would be highly effective for developers by addressing the most common patterns first. That analysis led us to three top patterns:
- Key-Value storage. The majority of the Amazon storage patterns were based on primary key access leading to single value or object. This pattern led to the development of Amazon S3.
- Simple Structured Data storage. A second large category of storage patterns were satisfied by access to simple query interface into structured datasets. Fast indexing allows high-speed lookups over large dataset. This pattern led to the development of Amazon SimpleDB. A common pattern we see is that secondary keys to objects stored in Amazon S3 are stored in SimpleDB, where lookups result in sets of S3 (primary) keys.
- Block storage. The remaining bucket holds a variety of storage patterns ranging special file systems such as ZFS to applications managing their own block storage (e.g. cache servers) to relational databases. This category is served by Amazon EBS which provides the fundamental building block for implementing a variety of storage patterns.
I have written before about the basic features of Amazon EBS:
- Amazon EBS will be offered in the form of storage volumes which you can mount into your EC2 instance as a raw block storage device. It basically looks like an unformatted hard disk. Once you have the volume mounted for the first time you can format it with any file system you want or if you have advanced applications such as high-end database engines, you could use it directly.
- Developers can create multiple volumes, in size ranging from 1 GB to 1TB. This volume will be created within a specified Availability Zone and will be accessible by your EC2 instances running in that Availability Zone. As to be expected with a volume abstraction only one instance can have the volume mounted at any given time. Volumes can migrate and be reattached to other instances if necessary for failure handling or application migration reasons.
- The consistency of data written to this device is similar to that of other local and network-attached devices; it is under control of the developer when and how to force flush data to disk if you want to bypass the traditional lazy-writer functionality in the operating systems file-cache. Because of the session oriented model for access to the volume you do not need to worry about eventual consistency issues.
However Amazon EBS isn't just a massive volume storage array within an Availability Zone, it provides a unique feature that allows for the creation of novel storage management scenarios: the ability to create snapshots and store those snapshots into Amazon S3. These snapshots can then be used as the starting point for creating new volumes within any availability zone.
We see developers use this feature for long term backup purposes, for use in rollback strategies, for (world-wide) volume re-creation purposes. Snapshots also play an important role in building fault-tolerance scenarios when combined with managing applications using Elastic IP addresses and Availability Zones.
Congratulations to the EBS team for delivering a great service that will help a lot of EC2 customers managing their storage efficiently.
What better way to pick up posting again than with following a meme. Nick Carr in Albums Going Steady describes the challenge to list "a favorite album for every year of your life." I actually do not have the problem described by Nick and others to really start with my birth year. The challenge has two restrictions: only one album per year and there can be no repeats of artists. I have added for myself the restriction that I should actually own the album, which restricts the set to choose from significantly and also makes for some peculiar choices.
Here is my list
1958: Jerry Lee Lewis, Great Balls of Fire
1959: Ray Charles, What I'd Say
1960: Miles Davis, Sketches of Spain
1961: Robert Johnson, King of the Delta Blues Singers
1962: Booker T & MG, Green Onions
1963: James Brown, Live at the Apollo
1964: John Coltrane, Love Supreme
1965: Bob Dylan, Highway 61 Revisted
1966: Cream, Fresh Cream
1967: The Doors, The Doors
1968: Johnny Cash , At Folsom Prison
1969: Rolling Stones, Let it Bleed
1970: The Who, Live at Leads
1971: Marvin Gaye, What's going on
1972: Deep Purple, Made in Japan
1973: Pink Floyd, Dark Side of the Moon
1974: Genesis, The Lamb Lies Down on Broadway
1975: Led Zeppelin, Physical Graffiti
1976: Eagles, Hotel California
1977: The Stranglers, Rattus Norvegicus
1978: Herman Brood and his Wild Romance, Shpritsz
1979: The Clash, London Calling
1980: AC/DC, Black in Black
1981: The Police, Ghost in the Machine
1982: Steel Pulse, True Democracy
1983: U2, Under a Blood Red Sky
1984: Talking Heads, Stop Making Sense
1985: John Cougar Mellencamp, Scarecrow
1986: Run DMC, Raising Hell
1987: Guns N' Roses, Appetite for Destruction
1988: Public Enemy, It Takes A Nation of Millions to Hold Us Back
1989: Eric Clapton, Journeyman
1990: Angelo Badalamenti, Twin Peaks Soundtrack
1991: Nirvana, Nervermind
1992: Rage Against the Machine, Rage Against the Machine
1993: Live, Throwing Copper
1994: Neil Young , Sleeps with Angels
1995: Garbage, Garbage
1996: James Cotton, Deep in the Blues
1997: Erykah Badu, Baduizm
1998: DMX, Flesh of my Flesh, Blood of my Blood
1999: Red Hot Chili Peppers, Californication
2000: Eminem, The Marshal Mathers LP
2001: The Strokes, Is This It
2002: Richard Locker, Jewish Cello Master Pieces
2003: Linkin Park, Meteora
2004: Green Day, American Idiot
2005: Fiona Apple , Extraordinary Machine
2006: Matisyahu, Youth
2007: Foo Fighters, Echoes, Silence, Patience & Grace
The hardest part was leaving Albums out; Too many masterpieces in the 70's for example. But also some other era were difficult: I really wanted Linton Kwesi Johnson in there but every time he had formidable competition. Madness got beaten by AC/DC, Beastie Boys by Run DMC, Nirvana kept Metallica out, The Stranglers win it from Jonny Rotten every time,. Honorable mentions for Traffic, Apocalyptica, Counting Crows and Nine Inch Nails; they almost made it.
The laptop that appeared on the cover of business week as part of the story “Building the Perfect Laptop” is the Thinkpad X300. It arrived at my doorstep this afternoon. It is everything it promised to be and more; superlight, rugged, SSD, full ports, wifi, lan & cell networks, dvd, replaceable batteries and 13.3" screen with 1440x900 graphics.
And all of this weighs in at 1420 grams.
It is amazing how light it is for a full featured laptop. See the unboxing pictures on flickr.
My opening slide for tomorrow's keynote at the MySQL Conference has this feel of speed and excitement to it that represents the current progress towards Cloud Computing. Persistent Storage for EC2 will be an important part of the presentation, but I'll mainly focus on general non-functional lesson from building large-scale services.
I would like to introduce to you the newest feature of Amazon EC2: Persistent local storage. This has been very high on the request list of EC2 customers and I believe that combined with the Availability Zones and Elastic IP Address features released earlier this month this makes EC2 the ideal environment for building highly scalable and reliable applications.
Significant innovation has gone into this feature: Instead of restricting developers to the use of a particular (distributed) file-system we once again decided to look at what is the most fundamental building block and how we could offer that in the most scalable and reliable manner.
Persistent storage for Amazon EC2 will be offered in the form of storage volumes which you can mount into your EC2 instance as a raw block storage device. It basically looks like an unformatted hard disk. Once you have the volume mounted for the first time you can format it with any file system you want or if you have advanced applications such as high-end database engines, you could use it directly.
Developers can create any number of volumes they want, in size ranging from 1 GB to 1TB. This volume will be created within a specified Availability Zone and will be accessible by your EC2 instances running in that Availability Zone. As to be expected with a volume abstraction only one instance can have the volume mounted at any given time. Volumes can migrate and be reattached to other instances if necessary for failure handling or application migration reasons.
The consistency of data written to this device is similar to that of other local and network-attached devices; it is under control of the developer when and how to force flush data to disk if you want to bypass the traditional lazy-writer functionality in the operating systems file-cache. Because of the session oriented model for access to the volume you do not need to worry about eventual consistency issues.
Snapshots
If we would have stopped here that would have already been quite a solid service for developers to use. We realized we needed to do more to make sure that developers could build truly geo-scalable applications. For that we introduced snapshot functionality: you ask the EC2 to make a snapshot of your volume and store it into Amazon S3. You can use this for long term backup purposes, for use in rollback strategies, but also for (world-wide) volume re-creation purposes.
When you create a volume you can ask it to be created from a particular snapshot. And because this snapshot is stored in S3, which is accessible in all Availability Zones, your new volume can be created in any zone, not just the one where the snapshot originated from.
The snapshot is extremely powerful technology and allows for building highly fault-tolerant applications operating world-wide. Combine these snapshots with Availability Zones and Elastic IPs and you have all the tools to manage and migrate even the most complex of applications.
And the great thing is it that it is all done with using standard technologies such that you can use this with any kind of application, middleware or any infrastructure software, whether it is legacy or brand new.
Early access
This new functionality is already being used privately by a handful of customers, and will be publically available later this year. We are talking about this service at this early stage because we believe this will help many of our EC2 customers with setting their development priorities for this year.
You can find more information at the AWS developer’s blog.
update:Thorsten from RightScale, who has been using the service, writes about his experiences
Today Amazon Web Services launched two new features in Amazon EC2 that are essential tools in building highly resilient applications: Elastic IP addresses and Availability Zones.
In summary:
- Elastic IP addresses are associated with a customer account and allow the customer to do its own dynamic mapping of IP address to instance. Using this dynamic mapping applications can remain reachable even in the presence of failures. This is an area where for example DNS reconfiguration is too slow a technique.
- Availability Zones allow the customer to specify in which location to launch a new EC2 instance. The world is divided up into Regions and a Region can hold multiple Availability Zones. These Zones are distinct locations within a region that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same region. By launching instances in separate Availability Zones, applications can be protected from failure of a single location.
With these two features EC2 customers now have tools to build applications that can tolerate a wide range of failure scenarios.
For more details visit the EC2 detail page and the Forum announcement
Update: excellent articles by the guys at RightScale: Using Elastic IP in switch-over scenarios, using Availability Zones to set up a fault-tolerant site and combining Elastic IP and Availability Zones.
Subscribe to this weblog's