Pinecone vs Weaviate
Compare and contrast Pinecone and Weaviate by architecture, ingestion, queries, performance, and scalability.
Pinecone vs Weaviate Ingestion
Ingestion
Pinecone
Weaviate
Streaming and bulk ingestion
Pinecone supports inserting vectors in batches of 100 vectors or fewer with a maximum size per upsert request of 2MB. Pinecone cannot perform reads and writes in parallel and so writing in large batches can impact query latency and vice versa.
Weaviate recommends batching vector embeddings in sizes of 100-300. For large-size files, it recommends breaking them up and ingesting them using libraries like ijson for JSON files and pandas for CSV files. Some manual flushing of batches may be required.
Index updates
Pinecone offers two methods for updating vectors: full and partial updates. Full updates modify the entire dataset and a partial update will update specific fields using a unique identifier.
Weaviate can update the values of an existing property or an entire object in a schema. Weaviate does use HNSW index type which is more costly when it comes to adding or updating vectors. Weaviate does not support adding or deleting properties to the schema.
Embedding generation
Pinecone supports API calls to OpenAI, Cohere and HuggingFace to insert and index embeddings.
Weaviate supports API calls to OpenAI, Cohere and Huggingface to index embeddings.
Size of vectors and metadata
Pinecone supports 40kb of metadata per vector and a maximum vector dimensionality is 20,000. Pod sizes, resource confidurations, are storage bound.
The maximum number of vector dimensions for an embedding is 65,535.
Versioning
There does not appear to be a way to version in Pinecone.
There does not appear to be a way to version in Weaviate.
Pinecone supports batch insertion of vectors and updates and in-place updates for vectors and metadata. Pinecone supports searches across high dimensional vector embeddings.
Weaviate supports batch insertion of vectors and updates and in-place updates for vectors and metadata. Weaviate supports searches across high dimensional vector embeddings.
Pinecone vs Weaviate Indexing
Indexing
Pinecone
Weaviate
KNN and ANN
Pinecone supports KNN and ANN search. The algorithms leveraged by Pinecone are not documented.
Weaviate supports KNN and ANN search using HNSW.
Additional indexes
Pinecone supports the creating a single sparse-dense vector for hybrid search. The sparse vector is used for text search and includes support for BM25 algorithms. Because this is a single vector there's no ability to independently weight the sparsity or density of the coordinates of the vector.
Weaviate has an inverted index that can be used for filters, hybrid search and BM25 search.
Vectorization
There is no documentation on vectorization in Pinecone.
Weaviate supports vectorization to speed up query execution.
Index management
Pinecone handles all index management.
Weaviate users are responsible for configuring and managing indexes and product quantization.
Pinecone supports KNN and ANN search. Pinecone supports sparse-dense vectors for hybrid search. Pinecone handles all index management.
Weaviate supports KNN and ANN search using HNSW indexing algorithms. Weaviate provides inverted indexes and vector search indexes and uses vectorization to speed up query execution. Users are responsible for index maintenance.
Pinecone vs Weaviate Querying
Querying
Pinecone
Weaviate
Metadata filtering
Pinecone supports metadata filtering and hybrid search. Pinecone filters are applied during the approximate kNN search.
Pinecone supports a limited number of metadata field types. It recommends avoiding indexing high-cardinality metadata as that will consume significantly more memory. The maximum results a query will return with metadata filtering is 1,000.
Pinecone supports a limited number of metadata field types. It recommends avoiding indexing high-cardinality metadata as that will consume significantly more memory. The maximum results a query will return with metadata filtering is 1,000.
Weaviate supports metadata filtering and hybrid search. Weaviate pre-filters the data and only if a number of records returns (default- greater than 40,000) will it run an ANN search. Otherwise, it uses a brute force exact search.
Weaviate uses a strict schema system with all of the fields and their type specified before the data is indexed.
Weaviate uses a strict schema system with all of the fields and their type specified before the data is indexed.
Multi-modal models
There is no documentation on multi-modal models in Pinecone.
Weaviate supports multi-modal modules with CLIP.
API (SQL, REST, etc)
Pinecone exposes REST APIs that can be called directly to configure and access Pinecone features.
Weaviate has RESTful APIs for database management and CRUD operations and a GraphQL API for accessing data objects and search.
Pinecone applies a filter during an approximate kNN search. Pinecone supports REST APIs.
Weaviate pre-filters data before an approximate kNN search. Weaviate supports a GraphQL API for search.
Pinecone vs Weaviate Ecosystem
Ecosystem
Pinecone
Weaviate
Integrations (Huggingface, Langchain, etc.)
Pinecone supports API calls to OpenAI, Cohere and HuggingFace to insert and index embeddings. Pinecone has an integration to Langchain and LlamaIndex.
Weaviate supports API calls to OpenAI, Cohere and HuggingFace to insert and index embeddings. Weaviate has an integration to Langchain and LlamaIndex.
Pinecone vs Weaviate Architecture
Architecture
Pinecone
Weaviate
Cloud architecture
Pinecone is a cloud-based service deployed partly on Kubernetes with a tightly coupled architecture. Each pod, configuration of resources, has one or more replicas and provides the RAM, CPU and SSD required.
Weaviate was built for on-prem and has recently introduced a managed offering. Weaviate has a tightly coupled architecture where CPU, RAM and SSD scale together for ingestion and queries. Weaviate stores its object store and inverted index within the same shard; it places its vector index next to the object store. Users responsible for clusters, shards and indexes. Resharding is an expensive operation.
Scalability
Pinecone offers a number of pods, or resource configurations, that can be picked depending on the performance requirements of the vector search. Pods are storage bound so once you cross a threshold you will scale your pod size up (1x, 2x, 4x and 8x) without downtime. It is not possible to scale a pod size down. You can also horizontally scale by adding more pods but this will pause new inserts and index creation. You can also add replicas to increase QPS.
Weaviate scales horizontally for ingestion and queries. Replicas to support high QPS use cases are still in development. Dynamically scaling a cluster is not fully supported- nodes cannot be removed if data is present. In this architecture, ingestion and queries use the same CPU and memory resources, there is no resource isolation, allowing for potential resource contention.
Enterprise readiness
Pinecone does not have case studies of enterprises using their product in production. It reported a partial database outage on March 1st, 2023.
Weaviate does not have case studies of enterprises using their product in production.
Pinecone is a cloud-service with a tightly-coupled architecture.
Weaviate is built for on-prem with a tightly coupled architecture. Scaling Weaviate requires data and infrastructure expertise and management.