Weaviate vs Pinecone
Compare and contrast Weaviate and Pinecone by architecture, ingestion, queries, performance, and scalability.
Weaviate vs Pinecone Ingestion
Ingestion
Weaviate
Pinecone
Streaming and bulk ingestion
Weaviate recommends batching vector embeddings in sizes of 100-300. For large-size files, it recommends breaking them up and ingesting them using libraries like ijson for JSON files and pandas for CSV files. Some manual flushing of batches may be required.
Pinecone supports inserting vectors in batches of 100 vectors or fewer with a maximum size per upsert request of 2MB. Pinecone cannot perform reads and writes in parallel and so writing in large batches can impact query latency and vice versa.
Index updates
Weaviate can update the values of an existing property or an entire object in a schema. Weaviate does use HNSW index type which is more costly when it comes to adding or updating vectors. Weaviate does not support adding or deleting properties to the schema.
Pinecone offers two methods for updating vectors: full and partial updates. Full updates modify the entire dataset and a partial update will update specific fields using a unique identifier.
Embedding generation
Weaviate supports API calls to OpenAI, Cohere and Huggingface to index embeddings.
Pinecone supports API calls to OpenAI, Cohere and HuggingFace to insert and index embeddings.
Size of vectors and metadata
The maximum number of vector dimensions for an embedding is 65,535.
Pinecone supports 40kb of metadata per vector and a maximum vector dimensionality is 20,000. Pod sizes, resource confidurations, are storage bound.
Versioning
There does not appear to be a way to version in Weaviate.
There does not appear to be a way to version in Pinecone.
Weaviate supports batch insertion of vectors and updates and in-place updates for vectors and metadata. Weaviate supports searches across high dimensional vector embeddings.
Pinecone supports batch insertion of vectors and updates and in-place updates for vectors and metadata. Pinecone supports searches across high dimensional vector embeddings.
Weaviate vs Pinecone Indexing
Indexing
Weaviate
Pinecone
KNN and ANN
Weaviate supports KNN and ANN search using HNSW.
Pinecone supports KNN and ANN search. The algorithms leveraged by Pinecone are not documented.
Additional indexes
Weaviate has an inverted index that can be used for filters, hybrid search and BM25 search.
Pinecone supports the creating a single sparse-dense vector for hybrid search. The sparse vector is used for text search and includes support for BM25 algorithms. Because this is a single vector there's no ability to independently weight the sparsity or density of the coordinates of the vector.
Vectorization
Weaviate supports vectorization to speed up query execution.
There is no documentation on vectorization in Pinecone.
Index management
Weaviate users are responsible for configuring and managing indexes and product quantization.
Pinecone handles all index management.
Weaviate supports KNN and ANN search using HNSW indexing algorithms. Weaviate provides inverted indexes and vector search indexes and uses vectorization to speed up query execution. Users are responsible for index maintenance.
Pinecone supports KNN and ANN search. Pinecone supports sparse-dense vectors for hybrid search. Pinecone handles all index management.
Weaviate vs Pinecone Querying
Querying
Weaviate
Pinecone
Metadata filtering
Weaviate supports metadata filtering and hybrid search. Weaviate pre-filters the data and only if a number of records returns (default- greater than 40,000) will it run an ANN search. Otherwise, it uses a brute force exact search.
Weaviate uses a strict schema system with all of the fields and their type specified before the data is indexed.
Weaviate uses a strict schema system with all of the fields and their type specified before the data is indexed.
Pinecone supports metadata filtering and hybrid search. Pinecone filters are applied during the approximate kNN search.
Pinecone supports a limited number of metadata field types. It recommends avoiding indexing high-cardinality metadata as that will consume significantly more memory. The maximum results a query will return with metadata filtering is 1,000.
Pinecone supports a limited number of metadata field types. It recommends avoiding indexing high-cardinality metadata as that will consume significantly more memory. The maximum results a query will return with metadata filtering is 1,000.
Multi-modal models
Weaviate supports multi-modal modules with CLIP.
There is no documentation on multi-modal models in Pinecone.
API (SQL, REST, etc)
Weaviate has RESTful APIs for database management and CRUD operations and a GraphQL API for accessing data objects and search.
Pinecone exposes REST APIs that can be called directly to configure and access Pinecone features.
Weaviate pre-filters data before an approximate kNN search. Weaviate supports a GraphQL API for search.
Pinecone applies a filter during an approximate kNN search. Pinecone supports REST APIs.
Weaviate vs Pinecone Ecosystem
Ecosystem
Weaviate
Pinecone
Integrations (Huggingface, Langchain, etc.)
Weaviate supports API calls to OpenAI, Cohere and HuggingFace to insert and index embeddings. Weaviate has an integration to Langchain and LlamaIndex.
Pinecone supports API calls to OpenAI, Cohere and HuggingFace to insert and index embeddings. Pinecone has an integration to Langchain and LlamaIndex.
Weaviate vs Pinecone Architecture
Architecture
Weaviate
Pinecone
Cloud architecture
Weaviate was built for on-prem and has recently introduced a managed offering. Weaviate has a tightly coupled architecture where CPU, RAM and SSD scale together for ingestion and queries. Weaviate stores its object store and inverted index within the same shard; it places its vector index next to the object store. Users responsible for clusters, shards and indexes. Resharding is an expensive operation.
Pinecone is a cloud-based service deployed partly on Kubernetes with a tightly coupled architecture. Each pod, configuration of resources, has one or more replicas and provides the RAM, CPU and SSD required.
Scalability
Weaviate scales horizontally for ingestion and queries. Replicas to support high QPS use cases are still in development. Dynamically scaling a cluster is not fully supported- nodes cannot be removed if data is present. In this architecture, ingestion and queries use the same CPU and memory resources, there is no resource isolation, allowing for potential resource contention.
Pinecone offers a number of pods, or resource configurations, that can be picked depending on the performance requirements of the vector search. Pods are storage bound so once you cross a threshold you will scale your pod size up (1x, 2x, 4x and 8x) without downtime. It is not possible to scale a pod size down. You can also horizontally scale by adding more pods but this will pause new inserts and index creation. You can also add replicas to increase QPS.
Enterprise readiness
Weaviate does not have case studies of enterprises using their product in production.
Pinecone does not have case studies of enterprises using their product in production. It reported a partial database outage on March 1st, 2023.
Weaviate is built for on-prem with a tightly coupled architecture. Scaling Weaviate requires data and infrastructure expertise and management.
Pinecone is a cloud-service with a tightly-coupled architecture.