Rockset vs. Elasticsearch for Real-Time Analytics

Elasticsearch does not natively support joins. This results in joining code as part of the application which results in prohibitively expensive queries. Or, do write-time denormalization which limits your flexibility and requires you to know the queries in advance.

Rockset supports a full featured SQL language including joins. A multi level aggregator executes Rockset’s join operator. Aggregators are distributed and the JOINs are executed in a parallel fashion over multiple aggregators for both scalability and speed.

When an index grows in size and data is spread out to shards on different machines, queries become slower and the only option is to create a new index with a larger number of shards and reindex all the data from the existing index to the new index. This is a resource-intensive, costly process. However, reindexing tens to hundreds of terabytes is a resource-intensive, costly process.

Rockset is designed to scale to hundreds of terabytes without needing to ever reindex a dataset. A Rockset index is organized in the form of thousands of micro-shards, and a set of micro-shards combine together to form appropriate number of shards based on the number of available servers and the total size of the index. If the dataset increases in size, a subset of micro-shards are peeled away from every existing shard and made into additional independent shards and these additional shards are distributed to the new machines in your cluster.

Elasticsearch requires deep expertise for controlling costs at scale. It requires configuring clusters with different node types, pre-configuring the number of shards in an index, tuning the amount of CPU per node, configuring thread-pools, and moving indexes between hot-warm-cold nodes to manage the index lifecycle as data ages.

Rockset’s cloud-native serverless architecture is optimized for hands-free operations, while providing visibility and control. With Rockset, you don't need to manage indexes, clusters or shards.

Elasticsearch was created in 2010 when the cloud was nascent. It was optimized for the datacenter and does not decouple compute from storage.

Rockset, on the other hand, is cloud-native and decouples compute and storage. Data is served from hot storage, and automatically backed up on durable cloud storage. Compute resources are scaled independently as required. Compute resources used for ingesting and querying data are also separated so you get fast queries on fresh data. Scaling storage independently from compute allows developers to seamlessly control price-performance.