ClickHouse vs Apache Pinot
Compare and contrast ClickHouse and Apache Pinot by architecture, ingestion, queries, performance, and scalability.
ClickHouse Architecture vs Apache Pinot
ClickHouse is open source and can be deployed anywhere. Several vendors such as ClickHouse, the company, and Tinybird also offer cloud versions. Compute and storage are tightly coupled, although ClickHouse Cloud was rearchitected to decouple compute and storage. ClickHouse Cloud pricing is based on compute and storage usage.
Pinot is a real-time distributed OLAP datastore that ingests both batch and streaming data. It has a distributed systems architecture that scales both horizontally and vertically, but unlike alternative OLAP databases, it does not decouple storage and compute. It supports both self-managed and PaaS options.
ClickHouse Ingestion vs Apache Pinot
ClickHouse has core integrations from common sources such as Kafka and S3. It recently introduced greater ability to handle semi-structured data using the JSON Object type and automatic schema inference.
Pinot supports high-performance ingest from streaming data sources. Each table is either offline or real time. Real-time tables have a smaller retention period and scale based on ingestion rate while offline tables have a larger retention period and scale based on the amount of data. In order to persistently store the generated segments that make up a table, you will need to change controller and server configs to add deep storage.
ClickHouse Queries vs Apache Pinot
ClickHouse uses SQL for querying, with support for SQL joins. ClickHouse integrates with some common tools for visual analytics, including Superset, Grafana and Tableau.
In Pinot, SQL-like queries are received by brokers and scatter the request between real-time and offline servers. The two tables then process requests, send results back to the broker, and responds with the result. Joins are limited, as is support for UDFs and subqueries, making Pinot more or less useful depending on the use case.
ClickHouse Performance vs Apache Pinot
ClickHouse leverages column orientation and heavy compression for better performance on analytics workloads. It also uses indexing to accelerate queries as well. While ClickHouse use cases often involve streaming data from Kafka, batching data is recommended for efficient ingestion.
Like its competitors, Pinot can achieve sub-second query latency at high concurrency. However, this level of performance requires tuning, management, and deep expertise. Compared with the open-source version, the PaaS versions of Pinot address some of these issues, but similarly require expertise while making tradeoffs affecting query performance.
ClickHouse Scalability vs Apache Pinot
ClickHouse can be used in both single-node and distributed modes. Tight coupling of compute and storage and the need to rebalance data make scaling out more complex, but cloud versions of ClickHouse help automate this process.
Pinot allows for vertical scaling by increasing CPU and memory for each node as well as horizontal scaling by adding additional nodes. Capacity planning is a time-consuming, iterative, and manual task. It involves load testing and tuning across multiple vectors including read QPS, write QPS, number of streaming partitions, daily data size, retention period, types of workloads, number and type of segments, and much more.