As I discussed in my re:Invent keynote earlier this month, I am now happy to announce the immediate availability of Amazon RDS Cross Region Read Replicas, which is another important enhancement for our customers using or planning to use multiple AWS Regions to deploy their applications. Cross Region Read Replicas are available for MySQL 5.6 and enable you to maintain a nearly up-to-date copy of your master database in a different AWS Region. In case of a regional disaster, you can simply promote your read replica in a different region to a master and point your application to it to resume operations. Cross Region Read Replicas also enable you to serve read traffic for your global customer base from regions that are nearest to them.
About 5 years ago, I introduced you to AWS Availability Zones, which are distinct locations within a Region that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same region. Availability Zones have since become the foundational elements for AWS customers to create a new generation of highly available distributed applications in the cloud that are designed to be fault tolerant from the get go. We also made it easy for customers to leverage multiple Availability Zones to architect the various layers of their applications with a few clicks on the AWS Management Console with services such as Amazon Elastic Load Balancing, Amazon RDS and Amazon DynamoDB. In addition, Amazon S3 redundantly stores data in multiple facilities and is designed for 99.999999999% durability and 99.99% availability of objects over a given year. Our SLAs offer even more confidence to customers running applications across multiple Availability Zones. Amazon RDS offers a monthly uptime percentage SLA of 99.95% per Multi-AZ database instance. Amazon EC2 and EBS offer a monthly uptime percentage SLA of 99.95% for instances running across multiple Availability Zones.
As AWS expanded to 9 distinct AWS Regions and 25 Availability Zones across the world during the last few years, many of our customers started to leverage multiple AWS Regions to further enhance the reliability of their applications for disaster recovery. For example, when a disastrous earthquake hit Japan in March 2011, many customers in Japan came to AWS to take advantage of the multiple Availability Zones. In addition, they also backed up their data from the AWS Tokyo Region to AWS Singapore Region as an additional measure for business continuity. In a similar scenario here in the United States, Milind Borate, the CTO of Druva, an enterprise backup company using AWS told me that after hurricane Sandy, he got an enormous amount of interest from his customers in the North Eastern US region to replicate their data to other parts of the US for Disaster Recovery.
Up until AWS and the Cloud, reliable Disaster Recovery had largely remained cost prohibitive for most companies excepting for large enterprises. It traditionally involved the expense and headaches associated with procuring new co-location space, negotiating pricing with a new vendor, adding racks, setting up network links and encryption, taking backups, initiating a transfer and monitoring it until the operation complete. While the infrastructure costs for basic disaster recovery could have been very high, the associated system and database administration costs could be just as much or more. Despite incurring these costs, given the complexity, customers could have found themselves in a situation where the restoration process does not meet their recovery time objective and/or recovery point objective. AWS provides several easy to use and cost effective building blocks to make disaster recovery very accessible to customers. Using the S3 copy functionality, you can copy the objects/files that are used by your application from one AWS Region to another. You can use the EC2 AMI copy functionality to make your server images available in multiple AWS Regions. In the last 12 months, we launched EBS Snapshot Copy, RDS Snapshot Copy, DynamoDB Data Copy and Redshift Snapshot Copy, all of which help you to easily restore the full stack of your application environments in a different AWS Region for disaster recovery. Amazon RDS Cross Region Read Replica is another important enhancement for supporting these disaster recovery scenarios.
We have heard from Joel Callaway from Zoopla, a property listing and house prices website in UK that attracts over 20 million visits per month, that they are using the RDS Snapshot Copy feature to easily transfer hundreds of GB of their RDS databases from the US East Region to the EU West (Dublin) Region every week using a few simple API calls. Joel told us that prior to using this feature it used to take them several days and manual steps to set up a similar disaster recovery process. Joel also told us that he is looking forward to using Cross Region Read Replicas to further enhance their disaster recovery objectives.
AWS customers come from over 190 countries and a lot of them in turn have global customers. Cross Region Read Replicas also make it even easier for our global customers to scale database deployments to meet the performance demands of high-traffic, globally disperse applications. This feature enables our customers to better serve read-heavy traffic from an AWS Region closer to their end users to provide a faster response time. Medidata delivers cloud-based clinical trial solutions using AWS that enable physicians to look up patient records quickly and avoid prescribing treatments that might counteract the patient’s clinical trial regimen. Isaac Wong, VP of Platform Architecture with Medidata, told us that their clinical trial platform is global in scope and the ability to move data closer to the doctors and nurses participating in a trial anywhere in the world through Cross Region Read Replicas enables them to shorten read latencies and allows their health professionals to serve their patients better. Isaac also told us that using Cross Region Replication features of RDS, he is able to ensure that life critical services of their platform are not affected by regional disruption. These are great examples of how many of our customers are very easily and cost effectively able to implement disaster recovery solutions as well as design globally scalable web applications using AWS.
Note that building a reliable disaster recovery solution entails that every component of your application architecture, be it a web server, load balancer, application, cache or database server, is able to meet the recovery point and time objectives you have for your business. If you are going to take advantage of Cross Region Read Replicas of RDS, make sure to monitor the replication status through DB Event Notifications and the Replica Lag metric through CloudWatch to ensure that your read replica is always available and keeping up. Refer to the Cross Region Read Replica section of the Amazon RDS User Guide to learn more.
Today we are kicking off AWS re:Invent 2013. Over the course of the next three days, we will host more than 200 sessions, training bootcamps, and hands on labs taught by expert AWS staff as well as dozens of our customers.
This year’s conference kicks off with a keynote address by AWS Senior Vice President Andy Jassy, followed by my keynote on Thursday morning. Tune in to hear the latest from AWS and our customers.
If you’re not already here in Vegas with us, you can sign up to watch the keynotes on live stream here.
Outside of the keynotes, there are an incredible number of sessions offering a tailored experience whether you are a developer, startup, executive, partner, or other. You can see the full session catalog here. I’m impressed by the scale and technical depth of what’s offered to attendees.
After my keynote on Thursday I will host two fireside chat sessions with cloud innovators and industry influencers:
First, I’ll talk with three technical startup founders
In the second session I will talk with three startup influencers
I will follow those two sessions with Startup Launches, where five companies will either launch their business or a significant feature entirely built on AWS. It will be a busy, fun, and informative afternoon!
Look forward to seeing you around the conference.
Speed of development, scalability, and simplicity of management are among the critical needs of mobile developers. With the proliferation of mobile devices and users, and small agile teams that are tasked with building successful mobile apps that can grow from 100 users to 1 million users in a few days, scalability of the underlying infrastructure and simplicity of management are more important than ever. We created DynamoDB to make it easy to set up and scale databases so that developers can focus on building great apps without worrying about the muck of managing the database infrastructure. As I have mentioned previously, companies like Crittercism and Dropcam have already built exciting mobile businesses leveraging DynamoDB. Today, we are further simplifying mobile app development with our newest DynamoDB feature, Fine-Grained Access Control, which gives you the ability to directly and securely access mobile application data in DynamoDB.
One of the pieces of a mobile infrastructure that developers have to build and maintain is the fleet of proxy servers that authorize requests coming from millions of mobile devices. This proxy tier allows vetted requests to continue to DynamoDB and then filters responses so the user only receives permitted items and attributes. So, if I am building a mobile gaming app, I must run a proxy fleet that ensures “firstname.lastname@example.org” only retrieves his game state and nothing else. While Web Identity Federation, which we introduced a few months back, allowed using public identity providers such as Login with Amazon, Facebook, or Google for authentication, it still required a developer to build and deploy a proxy layer in front of DynamoDB for this type of authorization
With Fine-Grained Access Control, we solve this problem by enabling you to author access policies that include conditions that describe additional levels of filtering and control. This eliminates the need for the proxy layer, simplifies the application stack, and results in cost savings. Using access control this way involves a setup phase of authenticating the user (step 1) and obtaining IAM credentials (step 2). After these steps, the mobile app may directly perform permitted operations on DynamoDB (step 3).
With today’s launch, apps running on mobile devices can send workloads to a DynamoDB table, row, or even a column without going through an intervening proxy layer. For instance, the developer of a mobile app will use Fine-Grained Access Control to restrict the synchronization of user data (e.g. Game history) across the many devices the user has the app installed on. This capability allows apps running on mobile devices to modify only rows belonging to a specific user. Also, by consolidating users’ data in a DynamoDB table, you can obtain real-time insights over the user base, at large scale, without going through expensive joins and batch approaches such as scatter / gather.
Many of you know Thorsten von Eicken as the founder of Rightscale, the company that has helped numerous organizations find their way onto AWS. In what seems almost a previous life by now Thorsten was one of the top young professors in Distributed Systems and I had the great pleasure of working with him at Cornell in the early 90's. What set Thorsten aside from so many other system research academics was his desire to build practical, working systems, a path that I followed as well.
In the back to basics readings this week I am re-reading a paper from 1995 about the work that I did together with Thorsten on solving the problem of end-to-end low-latency communication on high-speed networks. The problem we were facing in those days was than many new high-speed network technologies, such as ATM, became available for standard workstations but that the operating systems were not able to deliver those capabilities to its applications. Throughput was often acceptable but individual message latency was as bad as over regular ethernet, a problem that Chandu Tekkath had described earlier in "Limits to Low-Latency Communication on High-Speed Networks"
The lack of low-latency made that distributed systems (e.g. database replication, fault tolerance protocols) could not benefit from these advances at the network level. The research to unlock these capabilities led to an architecture called U-Net. What set U-Net aside from other research was that it was first and foremost and engineering effort as we set out to build a system that actually had to function in production. Many of those engineering experiences found their way back into the paper. U-Net also heavily influenced what later became the Virtual Network Architecture industry standard.
The work on U-Net was continued by Matt Welsh who built among other things a version that could be used for fast-ethernet on regular PCs and one that could safely integrate into type-safe environments such as the JVM.
U-Net: A User-Level Network Interface for Parallel and Distributed Computing, Anindya Basu, Vineet Buch, Werner Vogels, Thorsten von Eicken. Proceedings of the 15th ACM Symposium on Operating Systems Principles (SOSP), Copper Mountain, Colorado, December 3-6, 1995
I am very excited to announce AWS Activate, a program designed to provide startups with the resources they need to build applications on AWS.
Startups will forever be a very important customer segment of AWS. They were among our first customers and along the way some amazing businesses have been built by these startups, many of which running for 100% on AWS. Startups operate in a world of high uncertainty and limited capital, so an elastic and on-demand infrastructure at low and variable cost aligns very naturally with their needs. By reducing the cost of failure and democratizing access to infrastructure, the cloud has enabled more startups to build, experiment, and scale.
When we launched AWS the original mission was
To enable businesses and developers to use web services to build scalable sophisticated applications
We’re continually amazed at the incredible sophisticated applications that these startups have built on top of our foundational services. That includes the startups that have become household names – Instagram, Spotify, Pinterest, Dropbox, Etsy, AirBnB, Shazam – as well as incredibly successful companies that you might not yet have heard of, such as Twilio, Viki, Redbus, Floorplanner and Tellybug, and many more. We’re proud to have helped all startups achieve their goals.
As I’ve traveled this past year for AWS Summits I’ve met startups in countries all over the world building apps for every imaginable use case. What’s exciting to me is infrastructure is no longer a bottleneck to innovation. The democratization of infrastructure means that an internet startup in Bangalore or Sao Paulo or Manila has access to the same compute power as Amazon.com; the same durability as Dropbox; the same scalability as Airbnb; the same global footprint as Netflix. The result is we’re beginning to see more and more startups grow up in more places.
We’re excited to be a part of this global momentum in the startup ecosystem. The challenge now is to support and assist an increasing number of startups across the world.
To that end, today we’re pleased to announce AWS Activate, a new program for startups. AWS Activate is designed to provide startups with the resources they need to build applications on AWS. It includes access to web-based AWS Training courses, to help startups become familiar and proficient with AWS services; an AWS Support period, to provide expert guidance when a startup might need it; and in some cases AWS Promotional Credit. AWS Activate also allows startups to leverage the unique and robust ecosystem that has grown around AWS, both in terms of the developer community and third-party software vendors. The new Startup Forum will give startups a place to find and share tips and lessons learned – there are already posts from customers like Coursera as well as best practice guidance from AWS Solutions Architects. In addition, AWS Activate will include discounts on software that many startups find useful. Included already are exclusive offers from Opscode (for automation), AlertLogic (for security), and SOASTA (for testing).
The best part is – it’s free to join. You can learn more and sign up at AWS Activate.
The anonymity routing network Tor is frequently in the news these days, which makes it a good case to read up on the fascinating technologies behind it. Tor stands for The Onion Router as its technology is based on the onion routing principles. These principles were first described by Goldschlag, et al., from the Naval Research Lab, in their 1996 paper on Hiding Routing Information. Almost immediately work started on addressing a number of omissions in the original work in what became known as the second-generation onion router. Tor is the implementation of such a second generation router and has a number of fascinating features. The paper describing Tor is also very interesting from a practitioners point of view as it deals with the system complexities of implementing the router at scale.
Hiding Routing Information, David M. Goldschlag, Michael G. Reed, and Paul F. Syverson, in the proceeding of the Workshop on Information Hiding, Cambridge, UK, May, 1996.
Tor: The Second-Generation Onion Router, Roger Dingledine, Nick Mathewson and Paul Syverson, in Proceedings of the 13th USENIX Security Symposium, August 2004
Traditionally records in a database were stored as such: the data in a row was stored together for easy and fast retrieval. Not everybody agreed that the "N-ary Storage Model" (NSM) was the best approach for all workloads but it stayed dominant until hardware constraints, especially on caches, forced the community to revisit some of the alternatives. Combined with the rise of data warehouse workloads, where there is often significant redundancy in the values stored in columns, and database models based on column oriented storage took off. The first practical modern implementation is probably C-Store by Stonebraker, et al. in 2005. There is a great tutorial by Harizopoulos, Abadi and Boncz from VLDB 2009 that takes you through the history, trade-off's and the state of the art. Many of the modern high-performance data warehouses such as Amazon Redshift are based on column stores.
But the groundwork for Column Oriented Databases was laid in 1985 when George Copeland and Setrag Koshafian questioned the NSM with their seminal paper on a "Decomposition Storage Model" (DSM). From the abstract:
There seems to be a general consensus among the database community that the n-ary approach is better This conclusion is usually based on a consideration of only one or two dimensions of a database system The purpose of this report is not to claim that decomposition is better Instead, we claim that the consensus opinion is not well founded and that neither is clearly better until a closer analysis is made along the many dimensions of a database system The purpose of this report is to move further in both scope and depth toward such an analysis We examine such dimensions as simplicity, generality, storage requirements, update performance and retrieval performance
A Decomposition Storage Model, George P. Copeland and Setrag N. Khoshafian, in the Proceedings of the 1985 SIGMOD International Conference on Management of Data
This spring I travelled through Europe for the AWS Global Summit series. In my many conversations with customers, and with the media, I encountered surprise and excitement about the extent that European enterprises have already been using the Amazon Web Services for some time. Whether it is large telecommunications manufactures like Nokia Siemens Networks running their real-time data analytics for network operators on AWS, or a luxury hotel chain like Kempinski moving their core IT functions to AWS such that they can get out of the IT business, or a major newspaper corporation like News International, who plan to have 75% of their infrastructure running on AWS within 3 years to improve their agility, European enterprises have been moving to the cloud for some time to become more agile and competitive.
Europe is a continent with much diversity and for each country there are great AWS customer examples to tell. Given that I am originally from the Netherlands I have, of course, a special interest in how Dutch companies are using our cloud services.
For many young Dutch businesses AWS is the platform of choice such that they can grow unconstrained, targeting a global customer base, in the most cost-effective manner possible. There is long list of success stories: WeTransfer, Floorplanner, Mobypicture, Albumprinter, Wercker, Cloud9, Silk, Layar, Karma, Wakoopa, Peecho, Foodzy Usabila and many, many more.
But it is not just Dutch entrepreneurs who build their business in the cloud, also traditional Dutch enterprises are moving to the cloud to improve their agility and cost-effectiveness. Here are some great examples from different industries each with unique use cases.
Royal Dutch Shell – is one of the world’s largest companies. In addition to its goal of reducing energy costs, Shell needed to be more agile in deploying IT services and planning for user demand. To reach those goals, Shell in 2010 began using AWS. Shell leverages sensors to find oil in wells formerly thought to have run dry or in places where previous exploration indicated there was no oil. These sensors create massive amounts of geological data. Shell's IT shop has to figure out how to drive costs down, effectively manage the giant files and make it profitable for the company to deploy these sensors. Shell leverages AWS for big data analytics to help achieve these goals. Shell's scientists, especially the geophysicists and drilling engineers, frequently use cloud computing to run models. They provision compute capacity themselves, run their models and then return the cloud compute capacity, getting charged only for what they used. Shell says that two hundred and 300 project teams could be up and running in a day versus the weeks it would take them prior to AWS.
Unilever – Unilever R&D program intended to accelerate the company’s scientific progress through improved access to global information. Due to the exponential growth of the biology and informatics fields, Unilever needs to maintain this new program within a highly-scalable environment that supports parallel computation and heavy data storage demands. It makes use of the Eagle Genomics platform running on AWS, resulting in that Unilever’s digital data program now processes genetic sequences twenty times faster—without incurring higher compute costs. In addition, its robust architecture supports ten times as many scientists, all working simultaneously. This genetics R&D is crucial for Unliver to develop new products faster; for example comparing a healthy mouth with one with gingivitis - by identifying the shared genes amongst these two can be very helpful in developing the next generation of toothpaste.
Essent – supplies customers in the Benelux region with gas, electricity, heat and energy services. Essent has moved to the cloud to take advantage of the low, pay-as you-go, cost model and also the flexibility and scalability the cloud provides. Essent currently hosts all of their public facing websites and customer self-service portals in the AWS cloud. By offloading the task of managing infrastructure to AWS Essent is able to spend more time on innovating on behalf of their customers to help them in their energy usage. The company has used AWS to build an IT innovation zone, based upon open source products, which is being used to launch new innovations for customers like E-Mobility and E-thermostat products with a very fast time-to-market.
Tom Tom – Founded in 1991, TomTom is a leading provider of navigation and location-based products and services. In 2012 Tom Tom launched a new Location Based Services (LBS) platform to give app developers easy access to its mapping content to be able to incorporate rich location based data into their applications. When Tom Tom launched the LBS platform they wanted the ability to reach millions of developers all around the world without having them invest a lot of capital upfront in hardware and building expensive data centers so turned to the cloud. Using cloud computing as the underlying technology to run the LBS platform Tom Tom is able to provide developers with on-demand content that will enable them to build location based applications for fleet management, route planning, traffic management or spatial analytics.
Ohpen – The Dutch banking regulator, De Nederlandsche Bank (DNB), has cleared the path for Dutch financial institutions to make use of AWS. Dutch firm Ohpen has already moved to take advantage of the ruling by choosing AWS to host its core banking platform in an on-demand, software-as-a-service environment. According to Chris Zadeh, the CEO of Ohpen, large Dutch banks are already moving their entire retail banking platforms to the cloud using Ohpen core banking technology running on top of AWS.
These are just some of the public enterprise references from the Netherlands, but from personal conversations I know that in almost industry vertical Dutch companies are moving ahead rapidly to ensure that they can keep up with the global competition. Dutch enterprises from Media & Advertising, Financial Services, Energy, Transportation and Shipping, Life Sciences and Healthcare are experiencing the transformative nature of Cloud Computing; how IT can be enabler of innovation and greatly improve agility with traditional organizations.
I have picked the Netherlands as the obvious example, but the Dutch are not an exception; I can tell similar stories for almost all European countries. Enterprises in Europe are rapidly embracing cloud computing to continue to compete globally, and at AWS we are proud to help them achieve their goals.
On September 26 there is an AWS Summit in the Beurs van Berlage in Amsterdam for those from the Benelux interested in hearing how customers are making use of AWS and hearing the details about the AWS business and technologies. You can find more details and registration information at http://aws.amazon.com/aws-summit-benelux-2013/amsterdam/
Over the past few years, two important trends that have been disrupting the database industry are mobile applications and big data. The explosive growth in mobile devices and mobile apps is generating a huge amount of data, which has fueled the demand for big data services and for high scale databases. Meanwhile, mobile app developers have shown that they care a lot about getting to market quickly, the ability to easily scale their app from 100 users to 1 million users on day 1, and the extreme low latency database performance that is crucial to ensure a great end-user experience. These factors have made DynamoDB a compelling database for mobile developers, who happen to be among the biggest adopters of this technology. For example, Crittercism, a mobile application performance management company, uses DynamoDB to monitor the performance and stability of mobile apps on over 600 million devices with billions of daily transactions.
“We picked DynamoDB because it supports the scale we require" said Robert Kwok, CTO of Crittercism. "Our business is growing rapidly, and DynamoDB allows us to immediately scale up to support sudden increases in our workload. Switching to DynamoDB also reduced our costs by an order of magnitude and eliminated engineering efforts that used to come with growing our database layer."
Similarly, Earth Networks uses DynamoDB to power its lightning alert system, which gives millions of users real time access to live streaming lightning data, while Dropcam uses DynamoDB to power its cloud based monitoring system which has become one of the biggest video streaming platforms in the world. These innovative developers chose DynamoDB because it allows them to scale seamlessly without compromising on performance or cost.
The blooming mobile industry has made location data ubiquitous and offers mobile developers opportunities to build a whole new breed of novel mobile applications. Many mobile apps have features that help customers find nearby points of interest (e.g. “Find the closest coffee shop”), access location-specific offers, and find friends in your vicinity. Today, we are launching a geospatial indexing library that helps our customers build location-aware features by executing geospatial queries on their datasets in DynamoDB. With this library, you can:
- Efficiently store points of interest (POIs) and run exploration spatial queries
- Calculate great circle distances and perform spherical math. For example, “find points of interest near me”.
We put together a sample application that shows the power of the library we are launching today. For more details on this library, which you can find on github, please take a look at Jeff Barr’s blog.