Elasticsearch Architecture vs Rockset
Separation of compute and storage
Isolation of ingest and query
No - There are dedicated ingestion nodes but indexing, compaction and updates occur on the data nodes
Yes - separate compute clusters (Virtual Instances) for ingest and query
Isolation for multiple applications
Full isolation with replication
Yes - separate compute cluster (Virtual Instance) for each application
Use of storage hierarchy
• Hot, warm and cold storage on disk
• Frozen storage on cloud storage
• SSDs store shared hot data, accessible from any Virtual Instance cluster
• Cloud storage for durability
On-prem, PaaS options
SaaS - infrastructure, software and cluster ops managed by service provider
Elasticsearch is an open-source distributed search engine built on Apache Lucene, a full text search library. Elasticsearch is a distributed system, which means that it is designed to operate across multiple nodes, each responsible for a part of the data.
Rockset is built to be a cloud-only database and does not have a self-managed option. It disaggregates compute from both hot storage and cloud storage, allowing multiple isolated compute clusters to run on the same shared data.
Elasticsearch Ingestion vs Rockset
• Logstash JDBC input plugin for relational databases
• Open-source Kafka plugin or Kafka Elasticsearch Service Sink Connector (available only to managed Confluent and Elasticsearch)
• REST APIs or client libraries to sync data directly from the application
Managed data connectors to:
• Events streams (e.g. Kafka, Kinesis)
• Database CDC (e.g. MongoDB, DynamoDB, MySQL, PostgreSQL)
• Data lakes (e.g. S3, Google Cloud Storage)
Semi structured data
Yes- Ingests JSON and XML without a predefined schema
• Ingests JSON and XML without a predefined schema
• Ingests nested data
Transformations and rollups
Yes - Ingest pipelines can be configured to remove fields, extract values from text and enrich data. Ingest pipelines require ingest nodes in the cluster. Rolling up historical data is in technical preview
Yes - using SQL ingest transformations
Elasticsearch has a number of integrations as well as a REST API. It is a NoSQL database and natively supports semi-structured data. Transformations typically occur upstream so that data can be modeled for optimal performance before it is indexed in Elasticsearch.
Rockset has built-in connectors that manage streaming ingestion from common data sources. It has native support for semi-structured data, so that nested JSON and XML can be ingested and queried as is.
See Rockset in action
Get a product tour with a Rockset engineer.
Elasticsearch Queries vs Rockset
DSL - domain specific language
SQL - with extensions for nested data
No- Need to use workarounds including data denormalization, application-side joins, nested objects or parent-child relationships
• REST API
• Data APIs - saved SQL queries executed via REST endpoint
• Python, Java, Node.js and Go SDKs
• PowerBI, Qlik, Tableau
Integrations with Tableau, Looker, Grafana, Superset, Power BI, Retoolwhic
Elasticsearch has its own domain specific language (DSL) based on JSON. Joins are not a first class citizen in Elasticsearch requiring a number of complex and expensive workarounds. Elasticsearch is known for its developer tooling and supports a number of client libraries. Kibana is the visualization layer for Elasticsearch and is frequently used for log analytics and monitoring.
Rockset supports SQL as its native query language and can perform SQL joins. Users can create data APIs by storing SQL queries in Rockset that are executed from dedicated REST endpoints. Rockset integrates with some common visualization tools, but BI is not Rockset’s primary use case.
Elasticsearch Performance vs Rockset
• Ingests on a per-record or batch basis
• Data latency on a per-record basis is typically 1-2 seconds
• Ingests on a per-record basis
• Data latency is typically 1-2 seconds
Update API can update, delete or skip modifying the document. The entire document must be reindexed; in-place updates are not supported
Documents are mutable and can be updated at the field level
Converged Index, comprising a rowstore, columnstore and inverted index
• Converged Index (row, columnar and inverted index) built on all data by default
• All queries are resolved through the index
50-1000ms queries on 100s of TBs
50-1000ms queries on 100s of TB
Elasticsearch is a search engine that utilizes an inverted index. Although this approach leads to storage amplification, it also enables low-latency queries that demand less computation. Elasticsearch is tailored to accommodate large scale, append-only data such as logs, events, and metrics. To manage frequently updated data, users often utilize the Bulk API to minimize computational costs and ensure consistent query performance.
Rockset is designed to make streaming data queryable as quickly as possible by avoiding the need to batch data. It also updates documents efficiently by only reindexing fields that are part of an update request. Rockset indexes all data by default, which results in storage amplification but also enables low-latency queries that require less compute.
Elasticsearch Scalability vs Rockset
Manually resize machines
Resize compute clusters (Virtual Instances) via API or in the console
• Elasticsearch is horizontally scalable and can scale by adding nodes to the cluster
• When using managed Elastic, autoscaling policies can be used to self-monitor cluster health and it is the responsibility of the operator to update resource allocations either manually or using APIs. Elasticsearch rebalances the data automatically obeying shard allocation rules
• There are many cluster-level operations that need to be monitored when scaling
• Add or remove compute clusters (Virtual Instances) via API or in the console
• Scale out compute clusters for higher concurrency
• Use separate compute clusters to isolate ingest from query or for multiple isolated applications
• No rebalancing required
Elasticsearch is horizontally scalable and can scale by adding more nodes to the cluster. Its tightly coupled architecture means that compute and storage scale together for performance. This often results in resource contention and overprovisioning. Scaling Elasticsearch often requires deep expertise as there are many levels of the system that need to be managed- the server, operating system, network and software.
Rockset Virtual Instances are distributed compute clusters that can be scaled up for faster queries or scaled out for practically unlimited concurrency or if compute isolation is needed. Rockset has shared storage that scales automatically and independently, so no rebalancing is required.