Amazon DynamoDB - From the Super Bowl to WeatherBug

• 1310 words

Amazon DynamoDB is the fastest growing new service in the history of AWS.

In the five months since it launched in January, DynamoDB, our fast and scalable NoSQL database service, has been setting AWS growth records. Customers have used DynamoDB to support Super Bowl advertising campaigns, drive Facebook applications, collect and analyze data from sensor networks, track gaming information, and more. Customers such as Electronic Arts, News International, SmugMug, Shazam, IMDb, Amazon Cloud Drive, and many others are using DynamoDB to power their applications. The number of items that customers are storing in DynamoDB is more than doubling every couple of months (an item is the basic unit of data stored in DynamoDB and is between 0-64KB). After only two months, customers had already provisioned enough capacity to perform hundreds of billions of requests per day. We set very aggressive goals for provisioned capacity growth this year and we’ve already passed our 2012 targets. In short, DynamoDB is taking off.

We wanted to understand what was driving the accelerated growth and we reached out to some of our customers to find out more.

Earth Networks recently launched a new lightning proximity feature for their popular WeatherBug app. Built on DynamoDB, this lightning feature enables millions of users to quickly access live streaming lightning data to warn of dangerous storms in the area through Earth Networks’ consumer and enterprise products. In their own words:

“With DynamoDB, we were able to develop and test a new, high scale consumer application in just a few weeks. If we had chosen to use a more traditional database approach, we would still be in the process of designing the system for horizontal scalability in addition to negotiating license agreements.”

“The lightning project would not have been economically feasible if not for the cost savings and ease of provisioning and use of DynamoDB. We expect the system to provide the performance we need at 1/20th of the cost of the traditional DB approach.”

“The unique geo-lookup approach that we used was simple to implement in the DynamoDB NoSQL environment, and will be massively scalable and reliable across an initial consumer user base of approximately 6M smartphones. We expect to grow the use of this system across more than 40M unique users per month (mobile, tablet, desktop, Web).”

- Lex Crosett, CIO, Earth Networks

Another customer, Halfbrick Studios, noted that the scalability of DynamoDB was a key factor in allowing them to handle the rapid growth of a successful game:

“Our first Facebook game, Fruit Ninja Frenzy, jumped from 1 million to 8 million active monthly users in only two weeks. While growth like that is great, it’s really tough to quickly scale a normal database system to handle that kind of rapid increase in load, and our system was being strained. DynamoDB came along at just the right time. We switched to storing our game data in DynamoDB, which alleviated our scaling problems while also freeing us from the burden of managing all the underlying hardware and software. We love that DynamoDB handles so much of the management for us, freeing us to focus on development.”

- Glen Arrowsmith, Systems Architect, Halfbrick Studios

Madwell, an advertising agency based out of New York, talked about the rapid pace of development, the scalability, and the cost of DynamoDB. They needed a database that could quickly scale to handle millions of fans all simultaneously using an interactive Facebook app. Here’s what they had to say:

“DynamoDB allowed us to deploy and scale our interactive cross-brand Facebook application to a collective audience of almost 1.5 million fans, without having to worry about partitioning, sharding, or maintenance for a single moment. With its RESTful API and available SDKs, we were able to setup and implement our environment in just a week’s time for our Facebook app. We love the reactive scalability, which lets us scale only the tables we need during our predicted traffic spikes and turn them down during quieter times, paying only for the exact resources we needed. The seamless integration with the rest of the AWS infrastructure, especially CloudWatch, made real-time monitoring a cinch.”

- David Eisenman, Principal, Madwell

One of the most interesting things about Madwell’s use of DynamoDB is that they are able to minimize costs by paying attention to the exact capacity requirements of each table. They can dynamically adjust each table’s capacity as their usage patterns change throughout the day (read-heavy to write-heavy…high scale to low scale…etc.). This kind of fine grain control helps increase the utilization level of the resources tied to DynamoDB tables.

The emergence of very high scale use cases with short development times is another interesting consequence of DynamoDB’s ability to rapidly achieve high scale with no additional effort. Creating a table that can serve 100,000 writes/second is no more work than creating a table that can serve 10 writes/second – either way, it’s a single API command or a few clicks in the management console. Since DynamoDB handles the administration overhead automatically (you don’t have to worry about software updates, sharding, replication, cluster re-sizing…), the large overhead cost of setting up a massive distributed database is largely gone. Customers no longer need to spend months planning for large scale database requirements, and instead can ask for scale only when they need it. Since DynamoDB makes it more cost-effective to set up and run a massive application, it also opens up the range of use cases that a high scale database can serve.

Shazam’s use of DynamoDB is a particularly extreme (and impressive) example of how DynamoDB’s fast and easy scalability can be quickly applied to high scale applications. Shazam’s mobile app was integrated with Super Bowl ads, allowing for a much more interactive advertising experience. They needed to be able to handle an enormous increase in traffic for the duration of the event and used DynamoDB as part of their architecture. It only took three days of working with DynamoDB to go from the design phase to a fully production-ready deployment that was ready for one of the largest scale advertising events of the year.

DynamoDB’s unique ease of use and cost-effectiveness has enabled various customers to run high scale applications for a wide variety of use cases. We see this with customers like Earth Networks, who uses DynamoDB in their WeatherBug app, Electronic Arts, who uses DynamoDB to make data from their analytics system available in real time for use on their platform and games, and SmugMug, who uses DynamoDB to store metadata for the images and videos stored by their millions of users. Similarly, we see many customers in the online publishing industry using DynamoDB to store the metadata related to their online publications and customer interactions, while using Amazon Elastic MapReduce (EMR) to analyze the data. EMR is a web service that uses a hosted Hadoop framework to allow customers to easily and cost-effectively analyze vast amounts of data. By taking advantage of DynamoDB’s seamless integration with EMR, many of our publishing customers are able to easily analyze various trends in real-time, such as content popularity, using simple SQL commands.

DynamoDB is changing the game for a lot of our customers. The value of a fully-managed, elastic, and cost-effective service is one of the lessons we learned after years of effort at Amazon, and it cannot be overstated. The quick development time, easy scalability, and low administration overhead have allowed for quick iterations and faster innovation. The ease of use also means that developers can build a proof-of-concept test in hours, instead of weeks, which makes it easier to find out if the service is a good fit for you. We have a free tier, so I encourage you to try it out.

If you have an interesting DynamoDB use case that you’d like to tell us about, please let me know in the comments below.