Case Study: Complementing DynamoDB with Rockset for Real-Time IoT Analytics at 1NCE

Growth of the Internet of Things (IoT) hasn’t matched the hype due to numerous pain points: limited, unreliable network coverage, high connectivity, and device maintenance costs, and the uncertainty created by diverse, constantly-evolving cellular standards (4G versus 5G, LTE-M versus NB-IoT, etc.)

1NCE was founded in 2017 as a pure-play IoT connectivity provider to jumpstart IoT deployments by solving every one of those pain points.

For a flat-rate price of 10 EUR per device for 10 years, our business customers gain access to a fast, reliable global network – delivered by Deutsche Telekom and its worldwide roaming partners – and strong device management and security features.

This makes it simple and easy to deploy smart devices, everything from AR/VR headsets and smart energy meters for the home to tracking devices in delivery trucks for fleet management, remote monitors in factories, and other industrial settings.

All of this has helped 1NCE grow quickly. After just five years, we provide connectivity to ten million devices in 100+ countries on behalf of more than 7,000 customers.

Since 1NCE is so young, we were able to carefully build our back-end technology platform to be fully digital and cloud-native. The platform is based on container and serverless microservices and is mainly hosted on AWS, which provides developers with plug-and-play IoT integration so they can easily onboard and manage their devices.

Trying to Fit a Square Peg into a Round Hole

As an AWS shop, we naturally use Amazon DynamoDB as our main operational database. It stores most of the 50 million operational events we gather daily, which totals four TB of data per month. This comes from our network as well as the real-time state of every one of our customers’ devices, including location, connectivity, security, and battery life. DynamoDB also tracks all of the events associated with new devices as they are remotely set up and configured.

DynamoDB is very good at storing monitoring and management data. But as a transaction-focused database, DynamoDB had definite limits when it came to analyzing that data, especially in real-time. The most we could do were quick, large-scale aggregations and simple calculations of time-stamped data. And even enabling that was a lot of work for our small technical team. Meanwhile, more and more of our customers were telling us they needed more than the high-level KPI reports we periodically sent them. Their IoT devices were increasingly mission-critical to their business, and so they needed real-time business observability over them.

Since we already relied so heavily on DynamoDB, we tried to make it work for real-time analytics. We looked into BI and dashboard solutions compatible with DynamoDB but found they were still not granular nor real-time enough. We next tried building Lambda functions and step-function logic to enable customers to query DynamoDB. However, this stretched DynamoDB’s indexes too thin between customer queries and our own data operational needs. Queries were taking multiple seconds, which was unacceptable, as our target was less than one second. Moreover, the queries were cumbersome to develop and maintain.

We eventually came to the conclusion that trying to turn DynamoDB into our analytical database would be like trying to fit a square peg into a round hole.

We next started looking at migrating to a relational database in the cloud using Amazon RDS. We could then choose a database that naturally supported more powerful queries. However, this route would require us to custom build and manage data pipelines to continuously update and transform data between DynamoDB and RDS.

Besides the work involved, we were hesitant to choose a database that was not based around SQL. Everyone on our team knows SQL. Moving to a NoSQL database would require lengthy training for our engineers and/or new hires.

The Right Tool for the Task

Then we found a nearly effortless solution in a real-time analytics database in the cloud called Rockset. Rockset is natively integrated with DynamoDB, so it was easy to set up real-time sync between the two without requiring our data engineers to build a custom data pipeline.

Because it works with SQL, Rockset also made it very easy for our engineers to create and manage any type of query, from simple searches to complex joins and nested queries.

In particular, the Query Lambdas feature in Rockset enabled us to quickly create permanent, easy-to-manage, and secure SQL queries. These can automatically query new data mere seconds after it has been written to DynamoDB, without the need to transform it first. The results are served up to visual dashboards on our management portal that our customers interact with, basically in real-time.

At 1NCE, many technology tools we use are either part of AWS or something we built ourselves. The only exception is Rockset. That says a lot about how much we like Rockset, how easily it integrates into our stack, how fast and flexibly it queries DynamoDB, and how much our customers depend on it.

To give customers rich, real-time insights into their operations – in other words, business observability – with the least amount of work and time, Rockset is the right tool for the task.

Embedded content: https://www.youtube.com/watch?v=BcyJshqinbI

Rockset is the real-time analytics database in the cloud for modern data teams. Get faster analytics on fresher data, at lower costs, by exploiting indexing over brute-force scanning.