Rockset Architecture vs Elasticsearch
Separation of compute and storage
Isolation of ingest and query
Yes - separate compute clusters (Virtual Instances) for ingest and query
No - There are dedicated ingestion nodes but indexing, compaction and updates occur on the data nodes
Isolation for multiple applications
Yes - separate compute cluster (Virtual Instance) for each application
Full isolation with replication
Use of storage hierarchy
• SSDs store shared hot data, accessible from any Virtual Instance cluster
• Cloud storage for durability
• Hot, warm and cold storage on disk
• Frozen storage on cloud storage
SaaS - infrastructure, software and cluster ops managed by service provider
On-prem, PaaS options
Rockset is built to be a cloud-only database and does not have a self-managed option. It disaggregates compute from both hot storage and cloud storage, allowing multiple isolated compute clusters to run on the same shared data.
Elasticsearch is an open-source distributed search engine built on Apache Lucene, a full text search library. Elasticsearch is a distributed system, which means that it is designed to operate across multiple nodes, each responsible for a part of the data.
Rockset Ingestion vs Elasticsearch
Managed data connectors to:
• Events streams (e.g. Kafka, Kinesis)
• Database CDC (e.g. MongoDB, DynamoDB, MySQL, PostgreSQL)
• Data lakes (e.g. S3, Google Cloud Storage)
• Logstash JDBC input plugin for relational databases
• Open-source Kafka plugin or Kafka Elasticsearch Service Sink Connector (available only to managed Confluent and Elasticsearch)
• REST APIs or client libraries to sync data directly from the application
Semi structured data
• Ingests JSON and XML without a predefined schema
• Ingests nested data
Yes- Ingests JSON and XML without a predefined schema
Transformations and rollups
Yes - using SQL ingest transformations
Yes - Ingest pipelines can be configured to remove fields, extract values from text and enrich data. Ingest pipelines require ingest nodes in the cluster. Rolling up historical data is in technical preview
Rockset has built-in connectors that manage streaming ingestion from common data sources. It has native support for semi-structured data, so that nested JSON and XML can be ingested and queried as is.
Elasticsearch has a number of integrations as well as a REST API. It is a NoSQL database and natively supports semi-structured data. Transformations typically occur upstream so that data can be modeled for optimal performance before it is indexed in Elasticsearch.
See Rockset in action
Get a product tour with a Rockset engineer.
Rockset Queries vs Elasticsearch
SQL - with extensions for nested data
DSL - domain specific language
No- Need to use workarounds including data denormalization, application-side joins, nested objects or parent-child relationships
• Data APIs - saved SQL queries executed via REST endpoint
• Python, Java, Node.js and Go SDKs
• REST API
Integrations with Tableau, Looker, Grafana, Superset, Power BI, Retoolwhic
• PowerBI, Qlik, Tableau
Rockset supports SQL as its native query language and can perform SQL joins. Users can create data APIs by storing SQL queries in Rockset that are executed from dedicated REST endpoints. Rockset integrates with some common visualization tools, but BI is not Rockset’s primary use case.
Elasticsearch has its own domain specific language (DSL) based on JSON. Joins are not a first class citizen in Elasticsearch requiring a number of complex and expensive workarounds. Elasticsearch is known for its developer tooling and supports a number of client libraries. Kibana is the visualization layer for Elasticsearch and is frequently used for log analytics and monitoring.
Rockset Performance vs Elasticsearch
• Ingests on a per-record basis
• Data latency is typically 1-2 seconds
• Ingests on a per-record or batch basis
• Data latency on a per-record basis is typically 1-2 seconds
Documents are mutable and can be updated at the field level
Update API can update, delete or skip modifying the document. The entire document must be reindexed; in-place updates are not supported
Converged Index, comprising a rowstore, columnstore and inverted index
• Converged Index (row, columnar and inverted index) built on all data by default
• All queries are resolved through the index
50-1000ms queries on 100s of TB
50-1000ms queries on 100s of TBs
Rockset is designed to make streaming data queryable as quickly as possible by avoiding the need to batch data. It also updates documents efficiently by only reindexing fields that are part of an update request. Rockset indexes all data by default, which results in storage amplification but also enables low-latency queries that require less compute.
Elasticsearch is a search engine that utilizes an inverted index. Although this approach leads to storage amplification, it also enables low-latency queries that demand less computation. Elasticsearch is tailored to accommodate large scale, append-only data such as logs, events, and metrics. To manage frequently updated data, users often utilize the Bulk API to minimize computational costs and ensure consistent query performance.
Rockset Scalability vs Elasticsearch
Resize compute clusters (Virtual Instances) via API or in the console
Manually resize machines
• Add or remove compute clusters (Virtual Instances) via API or in the console
• Scale out compute clusters for higher concurrency
• Use separate compute clusters to isolate ingest from query or for multiple isolated applications
• No rebalancing required
• Elasticsearch is horizontally scalable and can scale by adding nodes to the cluster
• When using managed Elastic, autoscaling policies can be used to self-monitor cluster health and it is the responsibility of the operator to update resource allocations either manually or using APIs. Elasticsearch rebalances the data automatically obeying shard allocation rules
• There are many cluster-level operations that need to be monitored when scaling
Rockset Virtual Instances are distributed compute clusters that can be scaled up for faster queries or scaled out for practically unlimited concurrency or if compute isolation is needed. Rockset has shared storage that scales automatically and independently, so no rebalancing is required.
Elasticsearch is horizontally scalable and can scale by adding more nodes to the cluster. Its tightly coupled architecture means that compute and storage scale together for performance. This often results in resource contention and overprovisioning. Scaling Elasticsearch often requires deep expertise as there are many levels of the system that need to be managed- the server, operating system, network and software.