Rockset: 1 Billion Events in a Day with 1-Second Data Latency

This post weighs the advantages and disadvantages of moving read-heavy analytics off a primary MongoDB database to PostgreSQL.

With Rockset, Danish fintech Matter has the flexibility to run analytical queries on semi-structured data in S3 and DynamoDB as part of their NLP architecture.

Now that we've identified and investigated the slow and problematic queries on MongoDB, we discuss the advantages and disadvantages to various strategies for improving the performance of our database.

We released Rockset Developer Tools, including a new CLI tool and a new VS Code extension, to make it easier to develop real-time data applications on Rockset.

We explore the continuum of real-time analytics, from live, interactive dashboards to online applications that automatically take action on real-time data.

Serverless architectures offer ease of use and cost advantages. We explore what serverless means for your data stack.

Companies need to embrace real-time analytics to compete and survive. Only those that have invested in a real-time data stack will thrive.

eGoGames improves user experience, detects fraud, and makes business decisions using Rockset for real-time analytics on gaming data in Amazon DynamoDB and S3.

A real-time A/B experiments system is the best showcase of the power of real-time analytics. This post explains how to build for the requirements of a massive-scale A/B experiments platform.

Latency is one of the most essential factors in the performance of any application at scale. We explore various methods of identifying slow queries on MongoDB and understand how to improve them.

Performance of your primary MongoDB cluster is crucial. We look at how using multiple MongoDB clusters can help with performance isolation.

Real-time analytics demands low-latency complex queries. We discuss various options for speeding up read performance, including indexing, replication and sharding in MongoDB.

Lessons on building real-time data architectures based on experiences growing Facebook users 30x, from 50 million to 1.5 billion.

An in-depth look at indexing MongoDB data in Rockset and how it compares to indexing in MongoDB itself.

StoryFire uses Rockset to index data from their transactional MongoDB database to achieve performance and scale for analytical queries on their social video platform.

Generate ETA predictions for a delivery service using real-time location and order data from Kafka and DynamoDB.

Implementing a real-time recommendations API for an event ticketing system by indexing MongoDB data in Rockset for fast SQL.

We explore how real-time indexing on MongoDB enables fast aggregation and join queries, and how Rockset is specifically designed to meet real-time indexing requirements.

A compilation of MongoDB performance tuning resources, covering topics such as sharding, indexing, schema design and performance isolation.

To optimize RocksDB for cloud environments, we modified RocksDB-Cloud to allow remote compactions, separating compaction compute from storage, for greater efficiency.

Sessions to look forward to for MongoDB.live 2020

Build a Python application to create and execute APIs on aggregations and joins using Rockset and MongoDB.

Learn how Rockset indexes data from MongoDB change data capture (CDC) streams and how it compares to indexing in Elasticsearch.

Rockset uses Converged Indexing to make different types of queries run fast. We look at how Rockset's Index Scan uses the search index to accelerate range scans.

While joins are primarily an SQL concept, they are equally important in the NoSQL world as well. We explore how to perform the equivalent of SQL joins when using Elasticsearch.

This post outlines a fleet management solution using IoT and data technologies, such as DynamoDB, AWS IoT Core, AWS Lambda, and Rockset.

We discuss when stream processing, with KSQL and Kafka Streams, and when a real-time database like Rockset are best used for analyzing Kafka data.

Just as Rockset has reimagined and dramatically simplified the traditional ETL pipeline on the data-loading side, we’re now proud to release a new product feature - Query Lambdas - that similarly rethinks the data application development workflow.

What are the key considerations when selecting an analytics stack for building data applications on Kafka event streams?

This is a tech preview of an integration that will allow you to index your MongoDB data in row, column and inverted indexes, and run millisecond-latency SQL queries in real-time.

We demonstrate how to expose real-time IoT data in Kafka through the Rockset REST API in this example.

In this IoT example, we examine how to enable complex analytic queries on real-time Kafka streams from connected car sensors.

Standard Cognition, an AI-powered computer vision company, uses Rockset to enable their developers to deliver data APIs and product improvements.

An overview of what makes RocksDB well-suited to power many of the world's high-performance distributed data systems.

Learn how to create a SQL API for real-time Kafka analytics on the Twitter Streaming API, using AWS Lambda and Rockset.

Using SQL to join DynamoDB and S3 data, operations teams can perform live, ad-hoc analysis across multiple cloud systems.

Data engineers are often tasked with moving and preparing data to facilitate analytics. This guest post examines several considerations for data engineers designing for real-time analytics.

We discuss how different data backends - Druid, Elasticsearch and Rockset - can be used alongside Kafka for analytics on event data streams.

This guest post lays out the benefits of using DynamoDB, including 5 real-life examples, along with recommendations for performing analytics on DynamoDB data.

Learn how our UX team continually improves common user workflows in Rockset to simplify development of data-driven applications.

Build a real-time Tableau dashboard for operational monitoring and analytics on streaming event data from Kafka.

Learn how we rolled our own tool for analysis and visualization of Kubernetes events, and try the open-source dashboard for yourself.

How to use Rockset's fast geospatial indexes with Airbnb data.

How Rockset uses Grafana dashboards for monitoring production systems, Kubernetes, and GitHub metrics, and how we built a Rockset-Grafana plugin.

An architecture for real-time decision-making and live dashboards on VR data in Kafka, coming from live-streamed events.

We create an example dashboard in Tableau on data in DynamoDB, using Rockset as the SQL intelligence layer.

How to avoid costly mistakes with DynamoDB partition keys, read/write capacity modes, and global secondary indexes.

We explore operational analytics and why providing insights in real time to large numbers of users is crucial for organizations.

Get an inside look at the some of the techniques we used to reduce the cost of ingesting data from DynamoDB.

Get an in-depth look at the Kafka Connect Plugin for Rockset and the process to get it listed in Confluent Hub.

What’s the fastest we can load data into RocksDB?

We built an app to estimate the risk of a car break-in based on historical incidents.

We review several approaches to building Tableau operational dashboards and reporting on DynamoDB data, using SQL engines like Redshift and Athena.

We cover different approaches to real-time analytics on DynamoDB, using DynamoDB Streams, Lambda, and ElastiCache.

All businesses today are a series of real-time events. But what separates the good from the great is how they capture and operationalize that data.

What are the characteristics of an Operational Analytics processing system, and how does it differ from OLTP, OLAP and other data systems?

We discuss how SQL query planning is implemented to support operational analytics requirements, like low latency and high concurrency, in Rockset.

We examine various options for running SQL on JSON in relational databases, like PostgreSQL and MySQL, and in Rockset.

This blog describes how we use RocksDB at Rockset and how we tuned it for optimal performance.

Run high performance queries for operational analytics on data from Redshift tables by continuously ingesting and indexing Redshift data through a Rockset-Redshift integration.

Learn how and why Rockset developed a new SQL development environment for messy, semi-structured data.

This post shows one of many challenges that we encountered while building a fully dynamically typed SQL database: how we manipulate values of unknown types in our query execution backend, while approaching the performance of using native types directly.

Connect a Tableau live dashboard to a real-time event stream of complex JSON in a few easy steps.

FULL Creative uses Rockset to build live dashboards and run complex SQL on contact center call data in DynamoDB.

Learn how Rockset delivers low-latency SQL for search and analytics using a combination of row, column, and search indexes.

We built a web app that collects clickstream data as free-form JSON and runs SQL queries on the live data in a completely serverless fashion. We also seek to answer age-old questions besetting developers: tabs or spaces, vim or emacs?

When the existential question of spaces vs. tabs came up in our team, we ran a real-time survey to collect thousands of data points around it. We also wanted to settle the debate around other developer issues like SQL vs NoSQL.

Decore needed to enable ad hoc queries in their crypto accounting software service, so they turned to Rockset for fast analytics on DynamoDB.

In this post I explore how to support analytical queries on DynamoDB without prohibitive scan costs - using secondary indexes. I also evaluate the pros and cons of this approach in contrast to extracting data to Athena, Spark or Elastic for analytics

We compare options for real-time analytics on DynamoDB - Elasticsearch, Athena, and Spark - in terms of ease of setup, maintenance, query capability, latency. We also evaluate which use cases each of them are best suited for.

Rockset's schemaless SQL platform automatically infers schema at read time, allowing you to analyze messy data using SQL.

Designed from the ground up for serverless data management, Rockset makes SQL search and analytics simple and accessible.

Fynd uses Rockset to perform fast queries on real-time Kafka event streams, so they can react to consumer behavior as it happens.

An MIT team collaborates with a school in Brazil on a smart city project to analyze weather sensor data using Rockset.

Pittsburgh-based developer Doug Balog collects and analyzes nested JSON weather data to improve pollution forecasts in his community.

Build a chatbot that provides instant responses, leveraging fast SQL queries on CSV data.

Use Rockset's Smart Schema to understand complex, nested JSON and enable immediate queries using SQL on raw data.

Run SQL queries on data from PDF files, and join PDFs with JSON, CSV, XLSX, and other data.

Rockset distributes aggregation queries to reduce query latency and memory requirements. This was an intern project by Ashwath, Rockset's first ever intern.

The Aggregator Leaf Tailer architecture takes advantage of powerful indexing and cloud scalability to enable live analytics on real-time event streams.

Run fast SQL queries on data from DynamoDB tables by continuously ingesting and indexing DynamoDB data through a Rockset-DynamoDB integration.

Build live dashboards by connecting Redash to Rockset to create visualizations quickly and easily.

Run complex SQL across multiple Excel spreadsheets and join XLSX files with JSON, Parquet or CSV data.

Connect Kafka and Rockset to obtain real-time analytics with ad hoc SQL queries on event streams.

How to query nested JSON and CSV using SQL (including joins), without any upfront data preparation or complex data pipelines - for interactive data science using Python notebooks.

Build serverless microservices, data APIs, and data-driven applications. Use SQL to join and query JSON and CSV data using AWS Lambda and Rockset.

Serve a live dashboard using SQL on streaming Twitter data from Amazon Kinesis.

Make raw JSON immediately queryable through fast SQL queries, without ETL, data pipelines, or fixed schema.

David Cohen, System Architect at Intel, explores how RocksDB-Cloud can be be used to build an open-source cloud-friendly storage system.

Rockset Chief Architect Tudor Bosman discusses strong dynamic typing in SQL, and how it is implemented in Rockset.

SQL on unstructured data is hard. But storage and compute in the cloud are making SQL on raw data a reality.

Rockset CTO and co-founder Dhruba Borthakur discusses what Cloud-Native data processing entails, and how best to build for the cloud today.

Rockset CTO and co-founder Dhruba Borthakur discusses the shift from Open Source to Open Services in data infrastructure, and how Open Services will become the new standard.