How Real-Time Updates Work on Rockset with Kshitij

Kshitij Wadhwa:

The ability to get the changes that happen in an operational database and make them available for real-time applications, is a core capability for many organizations. Change Data Capture are design pattern to track data changes is one such approach to monitoring and capturing events in a system. Businesses use CDC from operational databases to power real-time applications and various microservices that demand low data latency. Examples of which include fraud prevention, game leaderboard APIs and personalized recommendation APIs. In these cases, if the data comes in too late, the company and customers are adversely affected.

Kshitij Wadhwa:

From the database perspective, building multiple indices are necessary for serving low latency queries, especially complex queries involving joints and applications. The trade-off here is that, every single update now needs to write to every index causing many random database writes. Traditional data systems are based on B-trees where random writes to the database translates to random writes of storage, presenting in additional computer and I/O.

Kshitij Wadhwa:

This means, real-time updates with high writes won't be able to keep up and result in increased data latency. As I mentioned in the previous slide, that the key problem with multiple indices is updating these indices with very low data latency and operation management. Now I will discuss how Rockset solves it using RocksDB-Cloud. At Rockset, we use RocksDB Cloud as one of the building blocks of Rockset's distributed Converged Index. RocksDB-Cloud is open source and is fully compatible with RocksDB, which is an LSM storage engine. LSM trees are optimized for writes because they turn random writes to different indices into sequential writes to underlying storage.

Kshitij Wadhwa:

To quickly recap, sequential writes to the underlying storage is faster than random writes on storage because it doesn't result in additional I/O, which can be caused by moving data. Being an LSM storage engine, a set of background threads are used for compaction, and compaction is a process of combining a set of files and generating files with over-riding abilities plus from out output files.

Kshitij Wadhwa:

Compaction needs a lot of compute resources. The higher the write rate into the database, the more compute resources are need for compaction because the system is only stable if compaction is able to keep up with the new writes to your database. RocksDB-Cloud server encapsulates a compaction job, with a set of cloud objects, and sends the request to a remote stateless server and does operating compute from storage. This also ensures your queries don't suffer in case of high writes to your collections. All documents to the Rockset collection are mutable and can be updated at the field level. Even if these fields are deeply nested inside arrays and objects. With other data systems, every update to a single field in a document results in re-indexing the whole document leading to high compute and I/O. The patch APIs solves the problem that other databases have with re-indexing the whole document.

Kshitij Wadhwa:

At Rockset we use the patch API to implement any index of the collection. This means that any updates that a collection gets, only part of those fields in a document are re-indexed, while keeping the rest of the fields in that document untouched. This results in Rockset being orders of magnitude, more efficient with compute and I/O compared to other databases.

Kshitij Wadhwa:

As a summary, Rockset makes it possible to provide low latency and high writes with RocksDB-Cloud and remote compaction. Developers also save on compute and I/O with incremental indexing using the Patch API. Overall, real-time insights with millisecond query latency allow applications to move faster. Thanks for listening. If you think Rockset might help navigate your data better, check out talks.rockset.com to try out a quick start guide with a free developer account.