This directory contains comprehensive benchmarks comparing SochDB against major vector database competitors.

# Install dependencies
pip install chromadb qdrant-client lancedb faiss-cpu python-dotenv openai

# Run the ultimate showdown
python benchmarks/ultimate_showdown.py

# Run real embedding demo (requires Azure OpenAI)
python benchmarks/real_search_demo.py

Tests SochDB against all available competitors:

• ChromaDB - Python-based, simple embedded database
• Qdrant - Rust-based with excellent filtering
• FAISS - Facebook's C++ library (no persistence)
• LanceDB - Columnar embedded database

Dimensions tested: 384 (MiniLM), 768 (BERT), 1536 (OpenAI)

Demonstrates semantic search using actual Azure OpenAI embeddings. Requires .env with:

AZURE_OPENAI_API_KEY=your-key
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/

Extensive benchmark with real embeddings across multiple test sizes.

Simulates actual RAG (Retrieval-Augmented Generation) workloads:

• Document ingestion
• Semantic search
• Batch queries (concurrent users)
• Filtered search
• Memory usage

Tests SochDB's unique features:

• All commercial embedding dimensions (128-3072)
• Concurrent read/write access
• Batch operation efficiency
• Real embedding performance

Based on testing, SochDB provides:

1. 🚀 Rust-Native Performance - SIMD-accelerated distance calculations (NEON/AVX2)
2. 📦 Truly Embedded - No server required, like SQLite for vectors
3. 🔢 All Dimensions - Supports 128-3072 (MiniLM to OpenAI text-embedding-3-large)
4. 💾 SQL Interface - Query vectors with familiar SQL syntax
5. 🔒 MVCC Transactions - Safe concurrent reads and writes
6. 🕸️ Graph + Vector - Hybrid knowledge graph + semantic search
7. 🐍 Python Simplicity - Native Python bindings via FFI

# Full competitive analysis
cd sochdb-python-sdk
python benchmarks/ultimate_showdown.py

# Real embeddings (requires Azure OpenAI)
python benchmarks/real_search_demo.py

# RAG-realistic workloads
python benchmarks/rag_benchmark.py

# Feature tests
python benchmarks/feature_benchmark.py

For real embedding benchmarks, create .env in the project root:

AZURE_OPENAI_API_KEY=your-key
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_VERSION=2024-12-01-preview

Results are saved to:

• showdown_results.json - Ultimate showdown results
• benchmark_results.json - Competitive benchmark results
• rag_benchmark_results.json - RAG benchmark results
• feature_benchmark_results.json - Feature benchmark results

Based on ANN-Benchmarks (ann-benchmarks.com), VectorDBBench (Zilliz), and Qdrant Benchmarks:

"The speed of vector databases should only be compared if they achieve the same precision." — Qdrant Benchmarks

ANN search is fundamentally about trading precision for speed. Any benchmark comparing two systems must use the same recall threshold (typically 0.95 or 0.99).

Recall@10 = (# of true neighbors in results) / 10

VectorDBBench (github.com/zilliztech/VectorDBBench) tests:

High p99/p999 indicates tail latency issues that affect user experience.

Typical configurations:

• High Recall (0.99+): M=32, ef_construction=256, ef_search=256
• Balanced (0.95-0.98): M=16, ef_construction=128, ef_search=100
• High Speed (0.90-0.95): M=8, ef_construction=64, ef_search=50

1. Same Hardware: All systems must run on identical hardware
2. Same Dataset: Use standard datasets (SIFT, GloVe, DBpedia)
3. Same Recall: Only compare at equivalent precision thresholds
4. Warm Cache: Run warmup queries before measurement
5. Multiple Runs: Report median of 5+ runs
6. Separate Client/Server: Use different machines for client and server (if applicable)

Client: 8 vCPUs, 16 GB RAM (Azure Standard D8ls v5)
Server: 8 vCPUs, 32 GB RAM (Azure Standard D8s v3)
CPU: Intel Xeon Platinum 8375C @ 2.90GHz
Memory Limit: 25 GB (to ensure fairness)

✅ Recall@k vs QPS curves (the gold standard chart)
✅ Multiple precision thresholds (0.90, 0.95, 0.99)
✅ Latency percentiles (p50, p95, p99)
✅ Index build time and memory usage
✅ Dataset and hardware specifications

❌ No recall measurement (speed without accuracy is meaningless)
❌ Single data point (no precision/speed tradeoff shown)
❌ Unknown or unreproducible hardware
❌ Proprietary datasets

Based on industry benchmarks, SochDB targets:

• ANN-Benchmarks: https://ann-benchmarks.com/
• VectorDBBench: https://github.com/zilliztech/VectorDBBench
• Qdrant Benchmarks: https://qdrant.tech/benchmarks/
• Zilliz Leaderboard: https://zilliz.com/benchmark
• Erik Bernhardsson's ANN Benchmarks: https://github.com/erikbern/ann-benchmarks