New Ways to Discover and Use Alexa Skills

| Comments ()

Introducing New Features That Make It Easier for Customers to Discover and Use Your Alexa Skills

Alexa, Amazon’s cloud-based voice service, powers voice experiences on millions of devices, including Amazon Echo and Echo Dot, Amazon Tap, Amazon Fire TV devices, and devices like Triby that use the Alexa Voice Service. One year ago, Amazon opened up Alexa to developers, enabling you to build Alexa skills with the Alexa Skills Kit and integrate Alexa into your own products with the Alexa Voice Service. Today, tens of thousands of developers are building skills for Alexa, and there are over 1,400 skills for Alexa – including Lyft and Honeywell, which were added today.

A New Experience for Discovering Skills

Today, we announced new ways for customers to discover and use the Alexa skills that developers have built, including a new voice-enablement feature and a completely redesigned Alexa app. Customers can now quickly search, discover and use skills. Starting today, customers can browse Alexa skills by categories such as “Smart Home” and “Lifestyle” in the Alexa app, apply additional search filters, and access their previously enabled skills via the “Your Skills” section.

Also available today, Alexa customers can use their voice to enable your skills: simply say “Alexa, enable NBC News” or “Alexa, enable 7 Minute Workout” and access them instantly. Customers can also find your skills with Amazon’s Skill Finder. To use Skill Finder, simply enable it via voice or in the Alexa app and say "Alexa, ask Skill Finder for the top skills."

One-Year Anniversary: ASK, AVS and The Alexa Fund

In addition to the new Alexa skill features, June 25th marked the one-year anniversary of our developer services. Last June, we introduced the Alexa Skills Kit (ASK), the Alexa Voice Service (AVS), and the Alexa Fund to help enable anyone to build the experience they wanted for Alexa. Some fun facts about the Alexa Skills Kit, Alexa Voice Service, and Alexa Fund include:

  • There are now over 1,400 Alexa skills and the catalog has grown by 50% in just over one month

  • Customers have made over 3 million requests using the top 10 most popular Alexa skills

  • Since January 2016, selection of Alexa smart home API skills has grown by more than 5x

  • There are now over 10,000 registered developers using the Alexa Voice Service to integrate Alexa into their products

  • There are tens of thousands of developers currently working on Alexa projects

  • The Alexa Fund has invested in 16 startups, with a focus on smart home and wearable products to date. Over the next year, The Alexa Fund will be expanding investments into startups that focus on robotics, developer tools, healthcare, accessibility and more

  • Some of the most popular Alexa skills are Jeopardy!, Daily Affirmation, Magic 8 Ball, Fitbit, and The Bartender

Build a Skill Today - Special Offers

Our skill templates and step-by-step guides are a valuable way to quickly learn the end-to-end process for building and publishing an Alexa skill. You can get started quickly using the flash cards skill template, fact skill template, trivia skill template, or how to skill template. Plus, if you publish a skill, you’ll receive an Alexa dev t-shirt. Quantities are limited. See Terms and Conditions

Additional Resources

For more information on getting started with devloping for Alexa, check out the following resources:

Alexa Developer Platform

Voice Design Education

In June 2015, Amazon Web Services announced that it would launch a new AWS infrastructure region in India. Today, I’m happy to announce that the Asia Pacific (Mumbai) Region is generally available for use by customers worldwide.

The opportunity to revolutionize

A region in India has been highly sought after by companies around the world who want to participate in one of the most significant economic opportunities in the world – India, a rising economy that holds tremendous promise for growth, a thriving technology hub with a rich eco-system of technology talent, and more.

Rapid economic growth in India is creating several business opportunities such as distributed ledger technology with blockchains that could drive efficiencies in the real estate market, Fin-Tech innovations such as P2P mobile apps that have the power to change the social economic lives of people through financial inclusion, applying the sharing economy from cabs to other modes of transportation such as two-wheelers and tractors, telemedicine in the remote reaches of the nation with smartphone apps, or enabling the agricultural sector with on-demand diagnostics to improve farm yield, to name just a few.

The platform to revolutionize

Market innovators and change agents need a comprehensive infrastructure platform that can reliably scale on-demand. Here are the benefits of a comprehensive platform, with customer examples:

  • A connected platform to sense the business environment

Examples of continuous sensing are found in the managed cloud platform built by Rachio on AWS IoT to enable the secure interaction of its connected devices with cloud applications/other devices. In addition, Change Healthcare (previously known as Emdeon) uses Amazon SNS to handle millions of confidential client transactions daily to process claims and pharmacy requests serving over 340K physicians and 60K pharmacies in full compliance with healthcare industry regulations.  

  • Seamless ingestion of large volumes of sensed data

AdiMap uses Amazon Kinesis to process real-time streaming online ad data and job feeds, and processes them for storage in petabyte-scale Amazon Redshift warehouses to glean business insights for jobs, ad spend, or financials for mobile apps.

  • Advanced problem solving that connects big data with machine learning

BuildFax illustrates a practical use case using Amazon Machine Learning to provide roof-age and job-cost estimations for insurers and builders, with property-specific values that don’t need to rely on broad, zip code-level estimates.

  • At-scale computing and visual analysis

DNAnexus deploys its customers’ genomic pipelines on Amazon EC2 for highly complex and sensitive DNA research activities. On a more playful note, for those that are inclined to look at our serverless compute architecture, I would love to reacquaint you with Dubsmash’s innovative use of AWS Lambda.

  • A workflow engine to drive business decisions

NASA’s Jet Propulsion Laboratory (JPL) used Amazon SWF as an integral part of several missions, including the MER and Carbon in the Arctic Reservoir Vulnerability Experiment (CARVE). NASA/JPL engineers used Amazon SWF and integrated the service with the Polyphony pipelines responsible for data processing of Mars images for tactical operations; expressing it with SWF requires a few simple lines of Java code together with AWS Flow Framework annotations.

Let’s build groundbreaking innovations together

I hope these short sketches illustrate our optimism in what the future holds. We sincerely believe that such capabilities permit creative expressions for unique solutions that are not only affordable but also scale reliably in order to drive meaningful benefits to the end-user or drive efficiency into business operations. For more details, see the case studies at All AWS Customer Stories.

We are excited to offer a complete portfolio of services from our foundational service stack for compute, storage, and networking to our more advanced solutions and applications. We look forward to broaden this portfolio to include additional services over the next several quarters. For more information about the services we offer in our Mumbai region, see the Region Table.

We believe in the Indian market and are investing for the long term. With the Mumbai Region, we look to better serve end-users in India. We believe that with the launch of the Mumbai Region, AWS will enable many more enterprise customers and startups in India to not just reduce the cost of their IT operations but embark on transformational innovations rapidly in critical new areas such as big data analysis, Internet of Things, and more.

We are at the cusp of a dramatic age of technology. Our Mumbai Region is enterprise grade and is open for business now. We want you to start using it today. You can learn more about our growing footprint at AWS Global Infrastructure.

Serverless Reference Architectures with AWS Lambda

| Comments ()

Building your applications with only managed components has become very popular, and AWS Lambda plays a crucial role in that. I see a tremendous interest in examples how to build such applications, and articles such as "The Serverless Start-Up - Down With Servers!" about teletext.io are read eagerly around the globe.

If you are looking for more examples there are the Lambda Serverless Reference Architectures that can serve as the blueprint for building your own serverless applications.

Mobile Backend Serverless Reference Architecture

The Mobile Backend reference architecture demonstrates how to use AWS Lambda along with other services to build a serverless backend for a mobile application. The specific example application provided in this repository enables users to upload photos and notes using Amazon Simple Storage Service (Amazon S3) and Amazon API Gateway respectively. The notes are stored in Amazon DynamoDB, and are processed asynchronously using DynamoDB streams and a Lambda function to add them to an Amazon CloudSearch domain. In addition to the source code for the Lambda functions, this repository also contains a prototype iOS application that provides examples for how to use the AWS Mobile SDK for iOS to interface with the backend resources defined in the architecture.

Real-time File Processing Serverless Reference Architecture

The Real-time File Processing reference architecture is a general-purpose, event-driven, parallel data processing architecture that uses AWS Lambda. This architecture is ideal for workloads that need more than one data derivative of an object. This simple architecture is described in the "Fanout S3 Event Notifications to Multiple Endpoints" blog post on the AWS Compute Blog. This sample application demonstrates a Markdown conversion application where Lambda is used to convert Markdown files to HTML and plain text.

Web Applications Serverless Reference Architecture

By combining AWS Lambda with other AWS services, developers can build powerful web applications that automatically scale up and down and run in a highly available configuration across multiple data centers—with zero administrative effort required for scalability, backups, or multi–data center redundancy.

This example looks at using AWS Lambda and Amazon API Gateway to build a dynamic voting application, which receives votes via SMS, aggregates the totals into Amazon DynamoDB, and uses Amazon Simple Storage Service (Amazon S3)to display the results in real time.

The architecture can be created with an AWS CloudFormation template.

The template does the following:

  • Creates an S3 bucket named to hold your web app.
  • Creates a DynamoDB table named VoteApp to store votes
  • Creates a DynamoDB table named VoteAppAggregates to aggregate vote totals
  • Creates a Lambda function that allows your application to receive votes
  • Creates a Lambda function that allows your application to aggregate votes
  • Creates an AWS Identity and Access Management (IAM) role and policy to allow Lambda functions to write to Amazon CloudWatch Logs and write and query the DynamoDB tables

IoT Backend Serverless Reference Architecture

The Internet of Things (IoT) Backend reference architecture demonstrates how to use AWS Lambda in conjunction with Amazon Kinesis, Amazon DynamoDB, Amazon Simple Storage Service (Amazon S3), and Amazon CloudWatch to build a serverless system for ingesting and processing sensor data. By leveraging these services, you can build cost-efficient applications that can meet the massive scale required for processing the data generated by huge deployments of connected devices.

This repository contains sample code for all the Lambda functions depicted in this diagram as well as a AWS CloudFormation template for creating the functions and related resources. There is also a simple webpage that you can run locally to publish sample events and query the data from DynamodDB.

Real-time Stream Processing Serverless Reference Architecture

You can use AWS Lambda and Amazon Kinesis to process real-time streaming data for application activity tracking, transaction order processing, click stream analysis, data cleansing, metrics generation, log filtering, indexing, social media analysis, and IoT device data telemetry and metering. The architecture can be created with an AWS CloudFormation template.

The template does the following:

  • Creates a Kinesis Stream
  • Creates a DynamoDB table named stackname-EventData
  • Creates Lambda Function 1 (stackname-DDBEventProcessor) which receives records from Kinesis and writes records to the DynamoDB table
  • Creates an IAM Role and Policy to allow the event processing Lambda function read from the Kinesis Stream and write to the DynamoDB table
  • Creates an IAM user with permission to put events in the Kinesis stream together with credentials for the user to use in an API client

10 Lessons from 10 Years of Amazon Web Services

| Comments ()

The epoch of AWS is the launch of Amazon S3 on March 14, 2006, now almost 10 years ago. Looking back over the past 10 years, there are hundreds of lessons that we’ve learned about building and operating services that need to be secure, reliable, scalable, with predictable performance at the lowest possible cost. Given that AWS is a pioneer in building and operating these services world-wide, these lessons have been of crucial importance to our business. As we’ve said many times before, “There is no compression algorithm for experience.” With over a million active customers per month, who in turn may serve hundreds of millions of their own customers, there is no lack of opportunities to gain more experience and perhaps no better environment for continuous improvement in the way we serve our customers.

I have picked a few of these lessons to share with you in the hope that they may be of use for you as well.

1. Build evolvable systems

Almost from day one, we knew that the software we were building would not be the software that would be running a year later. The expectation was that with each order or two of magnitude, we would need to revisit and revise the architecture to make sure we could address the issues of scale.

But we couldn’t adopt the old style approach of upgrading systems through a maintenance outage, as many businesses around the world are relying on our platform for 24/7 availability. We needed to build such an architecture that we could introduce new software components without taking the service down. Marvin Theimer, Amazon Distinguished Engineer, once jokingly said that the evolution of Amazon S3 could best be described as starting off as a single engine Cessna plane, but over time the plane was upgraded to a 737, then a group of 747s, all the way to the large fleet of Airbus 380s that it is now. All the while, we were refueling in midair and moving customers from plane to plane without them even realizing it.

2. Expect the unexpected

Failures are a given and everything will eventually fail over time: from routers to hard disks, from operating systems to memory units corrupting TCP packets, from transient errors to permanent failures. This is a given, whether you are using the highest quality hardware or lowest cost components.

This becomes an even more important lesson at scale: for example, as S3 processes trillions and trillions of storage transactions, anything that has even the slightest probability of error will become realistic. Many of those failure scenarios can be anticipated beforehand, but many more are unknown at design and build time.

We needed to build systems that embrace failure as a natural occurrence even if we did not know what the failure might be. Systems need to keep running even if the “house is on fire.” It is important to be able to manage pieces that are impacted without the need to take the overall system down. We’ve developed the fundamental skill of managing the “blast radius” of a failure occurrence such that the overall health of the system can be maintained.

3. Primitives not frameworks

Pretty quickly, we started to realize that the way customers would like to use our services was a work in progress. When customers left the constraining, old world of IT hardware and datacenters behind, they started to develop systems with new and interesting usage patterns that no one had ever seen before. As such, we needed to be ultra-agile to make sure we were catering to our customers’ needs.

One of the most important mechanisms we provided was to offer customers a collection of primitives and tools, where they could pick and choose their preferred way to engage with the AWS cloud, instead of only providing one framework that they are forced to use, which includes everything and the kitchen sink. This approach has enabled our customers to become so successful, that even later generations of AWS services make use of exactly the same primitive services our customers have become accustomed to.

It is also important to realize that it is hard to predict what certain priorities are for your customers until they have the service in their hands and actually start building with it. This is why we deliver new services often with a minimal feature set and allow our customers to help drive the roadmap for extending the service with new features.

4. Automation is key

Developing software services that need to be operated is radically different from building software that needs to be shipped to customers. Managing systems at scale requires a very different mindset to ensure that we meet the reliability, performance, and scalability expectations of our customers.

A key mechanism to achieve this is to automate the management as much as possible, removing error prone, manual operations. To do this, we needed to build management APIs that control the key functionality of our operations. AWS helps its customers do this too. By decomposing your applications into essential building blocks, each with a management API, you can apply automation rules to maintain reliable and predictable performance at scale. A good litmus test has been that if you need to SSH into a server or an instance, you still have more to automate.

5. APIs are forever

This was a lesson we had already learned from our experiences with Amazon retail, but it became even more important for AWS’s API-centric business. Once customers started building their applications and systems using our APIs, changing those APIs becomes impossible, as we would be impacting our customer’s business operations if we would do so. We knew that designing APIs was a very important task as we’d only have one chance to get it right.

6. Know your resource usage

When building a financial model for a service to identify the appropriate charging model, be sure to have good data about the cost of the service and its operations, especially for running a high volume – low margin business. AWS needed to be very conscious as a service provider about our costs so that we could afford to offer our services to customers and identify areas where we could drive operational efficiencies to cut costs further, and then offer those savings back to our customers in the form of lower prices.

An example in the early days where we did not know the resources required to serve certain usage patterns was with S3: We had assumed that the storage and bandwidth were the resources we should charge for; after running for a while, we realized that the number of requests was an equally important resource. If customers have many tiny files, then storage and bandwidth don’t amount to much even if they are making millions of requests. We had to adjust our model to account for the all the dimensions of resource usage so that AWS could be a sustainable business.

7. Build security in from the ground up

Protecting your customers should always be your number one priority, and it certainly has been for AWS… from both an operational perspective as well as tools and mechanisms; it will forever be our number one investment area.

One approach that we learned quickly is that to build secure services, it is necessary to integrate security at the very beginning of service design. The security team is not a group that does validation after something has been built. They must be partners on day one to make sure that security is fundamentally rock solid from the ground up. There is no compromise when it comes to security.

8. Encryption is a first-class citizen

Encryption is a key mechanism for customers to ensure that they are in full control over who has access to their data. Ten years ago, tools and services for encryption were hard to use and it wasn’t until a few years into our operations that we learned how to best integrate encryption into our services.

It started by providing server-side encryption in S3 for compliance use cases. If you would inspect any disks in our datacenters, none of the data would be accessible. But with the launch of Amazon CloudHSM (for hardware security models) and later Amazon Key Management Service, customers could use their own keys for encryption, which removed the need for AWS to manage their keys.

For some time now, support for encryption has been integrated at the design phase of each new service. For example, in Amazon Redshift, each of the data blocks is encrypted by default with a random key and the collection of these random keys is again encrypted with a master key. The master key can be provided by customers, ensuring that they are the only ones who can decrypt and have access to their critical business data or personal identifiable information.

Encryption continues to be a high priority for our business. We will continue to make it even easier for our customers to make use of encryption so they can better protect themselves and their customers.

9. The importance of the network

AWS has come to support many different workloads; from high-volume transaction processing to video transcoding at scale, from high-performance parallel computing to massive web site traffic. Each of those workloads have unique requirements when it comes to the network.

AWS has developed a unique skill to innovate datacenter layout and operations, such that we can have flexible network infrastructure that can be adapted to meet the needs of our customers’ workloads, whatever they may be. We have learned over time that we should not be afraid to develop our own hardware solutions to ensure our customers can achieve their goals. This enables us to meet our very specific requirements, such as the ability to isolate AWS customers from each other on the network to achieve the highest levels of security.

Another successful example of how AWS-designed networking hardware and software enabled us to further improve performance for our customers was in addressing the virtualization tax on network access from virtual machines. Because network access is a shared resource, customers previously could experience significant jitter on the network at times. Developing a NIC that supported single root IO virtualization allowed us to give each VM its own hardware virtualized NIC. This lowered latency more than 2x and delivered more than 10x improvement in latency variability on the network.

10. No gatekeepers

The AWS team has delivered many services and features over time to create a very broad and deep platform for our customers. But AWS is so much more than the services that we built in-house: a very rich ecosystem exists of services delivered by our partners that extends the platform into many new directions.

For example, we have partners like Stripe offering payment services to Twilio making telephony programmable on AWS. Many of our customers are also building platforms themselves on top of AWS to serve specific vertical needs: Philips is building their Healthsuite Digital Platform for managing healthcare data, Ohpen has built a platform for retail banking on AWS, Eagle Genomics has built a genomics processing platform, and many more. What’s essential is that there are no gatekeepers on the AWS platform that tell our partners what they can and cannot do. “No gatekeepers” liberates the innovative processes and opens the door for many unexpected inventions, which are sure to follow.

I am looking forward to seeing what we learn – and AWS customers accomplish – over the next 10 years. And remember, it is still Day One ...

In November, Amazon Web Services announced that it would launch a new AWS infrastructure region in South Korea. Today, I’m happy to announce that the Asia Pacific (Seoul) Region is now generally available for use by customers worldwide.

A region in South Korea has been highly requested by companies around the world who want to take full advantage of Korea’s world-leading Internet connectivity and provide their customers with quick, low-latency access to websites, mobile applications, games, SaaS applications, and more. We’ve also been hearing many requests from Korean companies, including large enterprises like Samsung and Mirae Asset. For example, Samsung Electronic Printing used AWS to deploy its Printing Apps Center in a way that didn’t require them to invest up-front capital and kept total costs quite low. Mirae Asset Global Investments improved its web service environment and reduced annual management costs by 50% by consolidating the management of all web services, including servers, network, database, and security. We believe that with the launch of the Seoul Region, AWS will enable many more enterprise customers in Korea to reduce the cost of their IT operations and innovate faster in critical new areas such as big data analysis, Internet of Things, and more.

Many of these enterprises are assisted by our extensive partner ecosystem in Korea. The rapidly expanding AWS Partner Network (APN) in Korea includes independent software vendors (ISVs) and systems integrators (SIs) who are building innovative solutions and services around the AWS cloud. ISV partners such as Ahnlab, IGAWorks, Hancom, TMAXSoft, and Dreamline are providing a variety of software, security, and connectivity solutions that can be used in conjunction with AWS. SIs such as Vsystems, Bespin Global, Megazone, and GS Neotek are helping enterprises to migrate to AWS, deploy mission-critical applications on AWS, or are providing a full range of monitoring, automation, and management services for customers’ AWS environments. More details on these partners and solutions can be found at https://aws.amazon.com/partners/.

The Seoul Region also gives Korean gaming companies the freedom to successfully enable global services. For example, Nexon is Korea’s premier game company, operating 150 games in 150 countries, including major PC games such as FIFA Online 3, MapleStory 2, and Sudden Attack. Nexon uses AWS global infrastructure to manage its IT infrastructure more effectively, and they are now using AWS for their domestic workloads as well. With the Seoul Region now available, Nexon plans to use AWS not just for mobile games but also for latency-sensitive PC online games. All of the top 10 gaming companies in Korea use AWS, and we look forward to continuing to support their global growth and continued success.

Finally, the Seoul Region brings the benefits of the cloud much closer to home for Korean startups. In 2015, we expanded the AWS Activate program in Korea to provide startups with the resources needed to get started on AWS, such as access to guidance and 1:1 time with AWS experts as well as web-based training, self-paced labs, customer support, third-party offers, and AWS promotional credits. Through local partnerships with leading venture capitalists (VCs), accelerators, and incubators such as SparkLabs, Primer, Mashup Angels, BonAngels, TheVentures, and Futureplay, 250+ startups in Korea participated in the AWS Activate program this year, and we are excited to see what they are able to achieve with an AWS region in Korea.

You can learn more about our growing global infrastructure footprint at http://aws.amazon.com/about-aws/globalinfrastructure.

London Calling! An AWS Region is coming to the UK!

| Comments ()

Yesterday, AWS evangelist Jeff Barr wrote that AWS will be opening a region in South Korea in early 2016 that will be our 5th region in Asia Pacific. Customers can choose between 11 regions around the world today and, in addition to Korea, we are adding regions in India, a second region in China, and Ohio in 2016.

Today, I am excited to add the United Kingdom to that list! The AWS UK region will be our third in the European Union (EU), and we're shooting to have it ready by the end of 2016 (or early 2017). This region will provide even lower latency and strong data sovereignty to local users.

More startups, small and medium businesses, large enterprises, universities, and government organizations all over the world are moving to the AWS Cloud faster than ever before. We are committed to meeting our customers’ increasing needs for capacity and for powerful AWS services that eliminate the heavy lifting of the underlying IT infrastructure -- allowing them to focus more of their precious resources on their core business.

Leading UK organizations were among the early adopters of the cloud when we first started AWS back in 2006 and we continue to help them drive increased agility, lower IT costs, and easily scale globally. Here are some examples of how our UK customers are using the AWS platform:

The new region, coupled with the existing AWS regions in Dublin and Frankfurt, will provide customers with quick, low-latency access to websites, mobile applications, games, SaaS applications, big data analysis, Internet of Things (IoT) applications, and more.

Expanding the Cloud: Introducing Amazon QuickSight

| Comments ()

We live in a world where massive volumes of data are being generated from websites, connected devices and mobile apps. In such a data intensive environment, making key business decisions such as running marketing and sales campaigns, logistic planning, financial analysis, and ad targeting require deriving insights from these data. However, the data infrastructure to collect, store, and process data is geared primarily towards developers and IT professionals (e.g., Amazon Redshift, Amazon DynamoDB, Amazon EMR) whereas insights need to be derived by not just technical professionals but also non-technical, business users.

In our quest to enable the best data storage options for customers, over the years we have built several innovative database solutions such as Amazon RDS, Amazon RDS for Aurora, Amazon DynamoDB, and Amazon Redshift. Not surprisingly, customers are using them to collect and store massive amounts of data. Yet, the process of deriving actionable insights out of this wide variety of data sources is not easy. Traditionally, companies had to invest in a lot of complex tools to discover their data sets, ETL tools to prepare for analysis, and separate tools for analyzing and providing visually interactive dashboards.

Today, I am excited to share with you a brand new service called Amazon QuickSight that aims to simplify the process of deriving insights from a wide variety of data sources quickly, easily and at a low cost. QuickSight is a very fast, cloud powered, business intelligence service for the 1/10th the cost of old-guard BI solutions.

Big data challenges

Over the last several years, AWS has delivered on a comprehensive set of services to help customers collect, store, and process their growing volume of data. Today, many thousands of companies—from large enterprises such as Johnson & Johnson, Samsung, and Philips to established technology companies such as Netflix and Adobe to innovative startups such as Airbnb, Yelp, and Foursquare—use Amazon Web Services for their big data needs.

Every day, large amount of data is generated from customer applications running on top of AWS infrastructure, collected and streamed using services like Amazon Kinesis, and stored in AWS relational data sources such as Amazon RDS, Amazon Aurora, and Amazon Redshift; NoSQL data sources such as Amazon DynamoDB; and file-based data sources such as Amazon S3. Customers also use a variety of different tools, including Amazon EMR for Hadoop, Amazon Machine Learning, AWS Data Pipeline, and AWS Lambda to process and analyze their data.

There’s an inherent gap between the data that is collected, stored, and processed and the key decisions that business users make on a daily basis. Put simply, data is not always readily available and accessible to organizational end users. Most business users continue to struggle answering key business questions such as “Who are my top customers and what are they buying?”, “How is my marketing campaign performing?”, and “Why is my most profitable region not growing?” While BI solutions have existed for decades, customers have told us that it takes an enormous amount of time, IT effort, and money to bridge this gap.

Traditional BI solutions typically require teams of data engineers to spend several months building complex data models and synthesizing the data before they can generate their first report. These solutions lack interactive data exploration and visualization capabilities, limiting most business users to canned reports and pre-selected queries.

On-premise BI tools also require companies to provision and maintain complex hardware infrastructure and invest in expensive software licenses, maintenance fees, and support fees that cost upwards of thousands of dollars per user per year. To scale to a larger number of users and support the growth in data volume spurred by social media, web, mobile, IoT, ad-tech, and ecommerce workloads, these tools require customers to invest in even more infrastructure to maintain a reasonable query performance. This cost and complexity to implement and scale BI makes it difficult for most companies to make BI ubiquitous across their organizations.

Enter Amazon QuickSight

QuickSight is a cloud powered BI service built from the ground up to address the big data challenges around speed, complexity, and cost. QuickSight puts data scattered across various different big data sources such as relational data sources, NoSQL data sources, and streaming data sets at the fingertips of your business users in an easy-to-use user interface and at one-tenth the cost of traditional BI solutions. Getting started with QuickSight is straightforward. Let me walk you through some of the core experiences of QuickSight that makes it so easy to set up, connect to your data sources, and build visualizations in minutes.

QuickSight is built on large number of innovative technologies to get a business user their first insights fast. Here are the few key innovations that power QuickSight:

SPICE: One of the key ingredients that make QuickSight so powerful is the Super-fast, Parallel, In-memory Calculation Engine (SPICE). SPICE is a new technology built from the ground up by the same team that has also built technologies such as DynamoDB, Amazon Redshift, and Amazon Aurora. SPICE enables QuickSight to scale to many terabytes of analytical data and deliver response time for most visualization queries in milliseconds. When you point QuickSight to a data source, data is automatically ingested into SPICE for optimal analytical query performance. SPICE uses a combination of columnar storage, in-memory technologies enabled through the latest hardware innovations, machine code generation, and data compression to allow users to run interactive queries on large datasets and get rapid responses. SPICE supports rich calculations that help customers derive valuable insights as they explore their data without having to worry about provisioning or managing infrastructure. SPICE automatically replicates data for high availability and performance. This allows us to enable organizations to scale to thousands of users who can all perform fast, interactive analysis across a wide variety of AWS data sources. In addition to powering QuickSight, we are also enabling our AWS BI partners to integrate with SPICE, so that customers who use our partner tools can visualize their data quickly with a user interface that they are already familiar with.

Auto discovery: One of the challenges with BI and analytics is discovering the data and curating it for analytics. This requires an IT department to build a data catalog and make it discoverable with an analytics engine and tools. When a user logs in to QuickSight, it automatically discovers the list of data sources that a customer has access to and analyzes them without database configuration, setup, and so on. For instance, customers can visualize their data on an Amazon Redshift cluster by picking a table and then get to a visualization in less than 3 clicks. To enable this, we have built a live metadata catalog service that builds a catalog of data sources (e.g., Amazon Redshift, RDS, S3, Amazon EMR, and DynamoDB) to which the customer has access.

AutoGraph: Picking the right visualization is not easy, and there is lot of science behind it. For instance, optimal visualization depends on various factors: the type of data field one has selected (e.g., “Is it time, number, or string?”), cardinality of the data (e.g., “Does this field have only 4 unique values or 1 million values?”), and number of data fields one is trying to visualize. While QuickSight supports multiple graph types (e.g., bar charts, line graphs, scatter plots, box plots, pie charts, and so on), one of the things we have tried to simplify is a capability that automatically picks the right visualization for selected data using a technology called AutoGraph. With this, users pick which data fields they want to visualize and QuickSight automatically selects the right visual type for them.

Suggestions: Often the sheer volume of data can be overwhelming; many users just want to explore their data to learn interesting characteristics. For example, the most common query for sales data in an Amazon Redshift cluster might be “How do overall sales grow over time across different categories?” With QuickSight, we have built an engine that provides suggestions for interesting analytics that users might be interested in when they pick a data source to analyze. The engine derives its suggestions by analyzing the metadata of the data source, its most accessed queries and several other parameters. We believe this provides a simple way for users to deriving valuable insights without too much work.

Collaboration and sharing of live analytics: Often users want to slice and dice their data and share their analysis in a secure manner. With QuickSight, users can build a “storyboard” that contains multiple analyses with appropriate annotations, and share it with others in their organization. Unlike traditional tools, they can share live analysis instead of static images so that recipients can also derive insights on the storyboard that was shared. For enterprises, we are also providing Active Directory integration so that customers can share insights using their existing credentials.

I have highlighted only some of the key innovations behind QuickSight in this post. For detailed information about this product, visit the AWS Blog, the QuickSight Detail Page and the FAQ page.

What our customers are saying about QuickSight

As I mentioned earlier, many innovations at Amazon and AWS, including QuickSight, are driven by customer feedback. We actively listen to your pain points and handle the undifferentiated heavy lifting across the various dimensions of infrastructure, data management, and analytics. This strategy of constantly listening to customer feedback, and iterating on our capabilities rapidly, has been a virtuous cycle that has consistently worked well for us. QuickSight also started with similar roots and during the final stages of launch, I am pleased to hear such positive feedback from customers. We have heard great excitement from our customers like Nasdaq, and Intuit.

Nasdaq enables their customers to plan, optimize, and execute their business vision with confidence, using proven technologies to provide transparency and insight for navigating today's global capital markets. Their technology powers more than 100 marketplaces, clearinghouses, and central securities depositories in 50 countries, and so generates a lot of data. Nate Simmons, Principal Architect of Nasdaq Inc., tells us that they are always interested in new tools to analyze the data we have stored in Amazon Redshift, Amazon S3, and other sources. For him, having super-fast performance as the data volumes and usage grows is critical to their users. Based on their preview of QuickSight, they found the SPICE in-memory calculation engine combined with an easy-to-use UI to be appealing for their use cases.

Similarly, Troy Otillio, Director of Public Cloud at Intuit, tells us that based on their initial preview of QuickSight, they think this service is going to challenge the status quo. He mentions that it appears to be intuitive for their business users, particular those in marketing who need an easy-to-user tool with super-fast performance.

Summing it all up

We are excited about the launch of Amazon QuickSight and its early feedback. We believe this is one of the critical parts of our big data offerings. If you are interested in trying the product during our preview, you can sign up for the preview today.

The Startup Experience at AWS re:Invent

| Comments ()

AWS re:Invent is just over one week away—as I prepare to head to Vegas, I’m pumped up about the chance to interact with AWS-powered startups from around the world. One of my favorite parts of the week is being able to host three startup-focused sessions Thursday afternoon:

The Startup Scene in 2016: a Visionary Panel [Thursday, 2:45PM]
In this session, I’ll moderate a diverse panel of technology experts who’ll discuss emerging trends all startups should be aware of, including how local governments, microeconomic trends, evolving accelerator programs, and the AWS cloud are influencing the global startup scene. This panel will include:

  • Tracy DiNunzio, Founder & CEO, Tradesy
  • Michael DeAngelo, Deputy CIO, State of Washington
  • Ben Whaley, Founder & Principal Consultant, WhaleTech LLC
  • Jason Seats, Managing Director (Austin), & Partner, Techstars

CTO-to-CTO Fireside Chat [Thursday, 4:15 PM]
This is one of my favorite sessions as I get a chance to sit down and get inside the minds of technical leaders behind some of the most innovative and disruptive startups in the world. I’ll have 1x1 chats with the following CTOs:

  • Laks Srini, CTO and Co-founder, Zenefits
  • Mackenzie Kosut, Head of Technical Operations, Oscar Health
  • Jason MacInnes, CTO, DraftKings
  • Gautam Golwala, CTO and Co-founder, Poshmark

4th Annual Startup Launches [Thursday, 5:30 PM]
To wrap up our startup track, in the 4th Annual Startup Launches event we’ll invite five AWS-powered startups to launch their companies on stage, immediately followed by a happy hour. I can’t share the lineup as some of these startups are in stealth mode, but I can promise you this will be an exciting event with each startup sharing a special offer, exclusive to those of you in attendance.

Other startup activities

Startup Insights from a Venture Capitalists Perspective [Thursday, 1:30 PM]
Immediately before I take the stage, you can join a group of venture capitalists as they share insights and observations about the global startup ecosystem: each panelist will share the most significant insight they’ve gained in the past 12 months and what they believe will be the most impactful development in the coming year.

The AWS Startup Pavilion [Tuesday – Thursday]
If you’re not able to join the startup sessions Thursday afternoon, I encourage you to swing by the AWS Startup Pavilion (within re:Invent Central, booth 1062) where you can meet the AWS startup team, mingle with other startups, chat 1:1 with an AWS architect, and learn about AWS Activate.

Startup Stop on the re:Invent Pub Crawl [Wednesday evening]
And to relax and unwind in the evening, you won’t want to miss the startup stop on the re:Invent pub crawl, at the Rockhouse within The Grand Canal Shoppes at The Venetian. This is the place to be for free food, drinks, and networking during the Wednesday night re:Invent pub crawl.

Look forward to seeing you in Vegas!

The AWS Pop-up Lofts are opening in London and Berlin

| Comments ()

Amazon Web Services (AWS) has been working closely with the startup community in London, and Europe, since we launched back in 2006. We have grown substantially in that time and today more than two thirds of the UK’s startups with valuations of over a billion dollars, including Skyscanner, JustEat, Powa, Fanduel and Shazam, are all leveraging our platform to deliver innovative services to customers around the world.

This week I will have the pleasure of meeting up with our startup customers to we celebrate the opening of the first of the AWS Pop-up Lofts to open outside of the US in one of the greatest cities in the World, London. The London Loft opening will be followed in quick succession by our fourth Pop-up Loft opening its doors in Berlin. Both London and Berlin are vibrant cities with a concentration of innovative startups building their businesses on AWS. The Loft’s will give them a physical place to not only learn about our services but will aim to help cultivate a community of AWS customers that can learn from each other.

Every time I’ve visited the Loft’s in both San Francisco and New York there has been a great buzz with people getting advice from our solution architects, getting training or attending talks and demos. By opening the London and Berlin Loft’s we’re hoping to cultivate that same community and expand on the base of loyal startups we have, such as Hailo, YPlan, SwiftKey, Mendley, GoSquared, Playmob and Yoyo Wallet, to help them to grow their companies globally and be successful.

You can expect to see some of the brightest and most creative minds in the industry being on hand in the Lofts to help and I’d encourage all local startups to make the most of the resources which will be at your fingertips, ranging from technology resources through access to our vast network of customers, partners, accelerators, incubators and venture capitalists who will all be in the loft to help you gain the insight you need and provide advice on how to secure funding, and gain the ‘softer skills’ needed to to grow your businesses.

The AWS Pop-up Loft, in London will be open from September 10 to October 29 between 10am and 6pm and later for evening events, Monday through Friday, in Moorgate. You can go online now at http://awsloft.london, to make one-on-one appointments with an AWS expert, register for boot camps and technical sessions, including:

  • Ask an Architect: an hour session which can be scheduled with a member of the AWS technical team. Bring your questions about AWS architecture, cost optimisation, services and features, or anything else AWS related. You can also drop in if you don’t have an appointment.
  • Technical Bootcamps: a one-day training sessions, taught by experienced AWS instructors and solutions architects. You will get hands-on experience using a live environment with the AWS Management Console. There is a ‘Getting started with AWS’ bootcamp on Chef bootcamp which will show customers how they can safeguard their infrastructure, manage complexity, and accelerate time to market.
  • Self-paced Hands-on Labs: beginners through advanced users can attend the labs which will help sharpen AWS technical skills at a personal pace and are available for free in the Loft during operating hours.

The London Loft will also feature an IoT Lab with a range of devices running on AWS services, many of which have been developed by our Solutions Architects. Visitors to the Loft will be able to participate in live demos and Q&A opportunities, as our technical team demonstrates what is possible with IoT on AWS.

You are all invited to join us for the grand opening party at the Loft in London on September 10 at 6PM. There will be food, drinks, DJ, and free swag. The event will be packed, so RSVP today if you want to come and mingle with hot startups, accelerators, incubators, VCs, and our AWS technical experts. Entrance is on a first come, first serve basis.

Look out for more details on the Berlin Loft, which will follow soon. I look forward to seeing you in new European Lofts in the coming weeks!

Today, we are releasing a plugin that allows customers to use the Titan graph engine with Amazon DynamoDB as the backend storage layer. It opens up the possibility to enjoy the value that graph databases bring to relationship-centric use cases, without worrying about managing the underlying storage.

The importance of relationships

Relationships are a fundamental aspect of both the physical and virtual worlds. Modern applications need to quickly navigate connections in the physical world of people, cities, and public transit stations as well as the virtual world of search terms, social posts, and genetic code, for example. Developers need efficient methods to store, traverse, and query these relationships. Social media apps navigate relationships between friends, photos, videos, pages, and followers. In supply chain management, connections between airports, warehouses, and retail aisles are critical for cost and time optimization. Similarly, relationships are essential in many other use cases such as financial modeling, risk analysis, genome research, search, gaming, and others. Traditionally, these connections have been stored in relational databases, with each object type requiring its own table. When using relational databases, traversing relationships requires expensive table JOIN operations, causing significantly increased latency as table size and query complexity grow.

Enter graph databases

Graph databases belong to the NoSQL family, and are optimized for storing and traversing relationships. A graph consists of vertices, edges, and associated properties. Each vertex contains a list of properties and edges, which represent the relationships to other vertices. This structure is optimized for fast relationship query and traversal, without requiring expensive table JOIN operations.

In this way, graphs can scale to billions of vertices and edges, while allowing efficient queries and traversal of any subset of the graph with consistent low latency that doesn’t grow proportionally to the overall graph size. This is an important benefit for many use cases that involve accessing and traversing small subsets of a large graph. A concrete example is generating a product recommendation based on purchase interests of a user’s friends, where the relevant social connections are a small subset of the total network. Another example is for tracking inventory in a vast logistics system, where only a subset of its locations is relevant for a specific item. For us at Amazon, the challenge of tracking inventory at massive scale is not just theoretical, but very real.

Graph databases at Amazon

Like many AWS innovations, the desire to build a solution for a scalable graph database came from Amazon’s retail business. Amazon runs one of the largest fulfillment networks in the world, and we need to optimize our systems to quickly and accurately track the movement of vast amounts of inventory. This requires a database that can quickly traverse the logistics history for a given item or order. Graph databases are ideal for the task, since they make it easy to store and retrieve each item’s logistics history.

Our criteria for choosing the right graph engine were:

  1. The ability to support a graph containing billions of vertices and edges.
  2. The ability to scale with the accelerating pace of new items added to the catalog, and new objects and locations in the company’s expanding fulfillment network.

After evaluating different technologies, we decided to use Titan, a distributed graph database engine optimized for creating and querying large graphs. Titan has a pluggable storage architecture, using existing NoSQL databases as underlying storage for the graph data. While the Titan-based solution worked well for our needs, the team quickly found itself having to devote an increasing amount of time to provisioning, managing, and scaling the database cluster behind Titan, instead of focusing on their original task of optimizing the fulfillment inventory tracking.

Thus, the idea was born for a robust, highly available, and scalable backend solution that wouldn’t require the burden of managing a massive storage layer. As I wrote in the past, I believe DynamoDB is a natural choice for such needs, providing developers flexibility and minimal operational overhead without compromising scale, availability, durability, or performance. Making use of Titan’s flexible architecture, we created a plugin that uses DynamoDB as the storage backend for Titan. The combination of Titan with DynamoDB is now powering Amazon’s fulfillment network, with a multi-terabyte dataset.

Sharing it with you

Today, we are happy to bring the result of this effort to customers by releasing the DynamoDB Storage Backend for Titan plugin on GitHub. The plugin provides a flexible data model for each Titan backend table, allowing developers to optimize for simplicity (single-item model) or scalability (multi-item model).

The single-item model uses a single DynamoDB item to store edges and properties of a vertex. In DynamoDB, the vertex ID is stored as the hash key of an item, vertex property and edge identifiers are attribute names, and the vertex property values and edge property values are stored in the respective attribute values. While the single-item data model is simpler, due to DynamoDB’s 400 KB item size limit, you should only use it for graphs with fairly low vertex degree and small number properties per vertex.

For graphs with higher vertex degrees, the multi-item model uses multiple DynamoDB items to store properties and edges of a single vertex. In the multiple-item data model, the vertex ID remains the DynamoDB hash key, but unlike the single-item model, each column becomes the range key in its own item. Each column value is stored in its own attribute. While requiring more writes to initially load the graph, the multiple-item model allows you to store large graphs without limiting vertex degree.

Amazon’s need for a hassle-free, scalable Titan solution is not unique. Many of our customers told us they have used Titan as a scalable graph solution, but setting up and managing the underlying storage are time-consuming chores. Several of them participated in a preview program for the plugin and are excited to offload their graph storage management to AWS. Brian Sweatt, Technical Advisor at AdAgility, explained:

“At AdAgility, we store data pertaining to advertisers and publishers, as well as transactional data about customers who view and interact with our offers. The relationships between these stakeholders lend themselves naturally to a graph database, and we plan to leverage our experience with Titan and Groovy for our next-generation ad targeting platform. Amazon's integration between Titan and DynamoDB will allow us to do that without spending time on setting up and managing the storage cluster, a no brainer for an agile, fast-growing startup.”

Another customer says that AWS makes it easier to analyze large graphs of data and relationships within the data. According to Tom Soderstrom, Chief Technology Officer at NASA’s Jet Propulsion Laboratory:

“We have begun to leverage graph databases extensively at JPL and running deep machine learning on these. The open sourced plugin for Titan over DynamoDB will help us expand our use cases to larger data sets, while enjoying the power of cloud computing in a fully managed NoSQL database. It is exciting to see AWS integrate DynamoDB with open sourced projects like Elasticsearch and Titan, while open sourcing the integrations.”

Bringing it all together

When building applications that are centered on relationships (such as social networks or master data management) or auxiliary relationship-focused use cases for existing applications (such as a recommendation engine for matching players in a game or fraud detection for a payment system), a graph database is an intuitive and effective way to achieve fast performance at scale, and should be on your database options shortlist. With this launch of the DynamoDB storage backend for Titan, you no longer need to worry about managing the storage layer for your Titan graphs, making it easy to manage even very large graphs like the ones we have here at Amazon. I am excited to hear how you are leveraging graph databases for your applications. Please share your thoughts in the comment section below.

For more information about the DynamoDB storage backend plug-in for Titan, see Jeff Barr’s blog and the Amazon DynamoDB Storage Backend for Titan topic in the Amazon DynamoDB Developer Guide