AI Search Vendor Scorecard

Choosing an AI search vendor based on a single accuracy metric is like buying a car solely by its top speed – you ignore the engine, fuel efficiency, safety, and how it handles your specific road. Founders who jump to metrics first often pick a vendor that looks great on a demo dataset but fails under real-world queries, latency budgets, or cost constraints.

The Problem

Founders evaluating AI search vendors are drowning in benchmark scores. A vendor claims 98% recall on the MS MARCO dataset, but when you deploy it on your own catalog of 10 million product SKUs, the retrieval time jumps from 50ms to 900ms and your monthly bill hits $40,000. The root cause: you chose a vendor based on a metric that was measured on an idealized evaluation set, without understanding the underlying retrieval method – dense vector search, hybrid BM25+vector, or learned sparse retrieval. Each method has different trade-offs in latency, cost, scalability, and domain adaptation.

The typical evaluation process is broken. Teams run a bake-off using a single evaluation script, compare NDCG@10 numbers, and pick the highest score. They ignore that Vendor A uses a 768-dimensional embedding with a flat index (fast for small datasets, catastrophic for large ones), while Vendor B uses a two-stage retrieval with a learned re-ranker (slower but more accurate for long-tail queries). Without a structured method-first evaluation, you end up with a vendor that cannot handle your data distribution, query volume, or compliance requirements. This playbook flips the order: evaluate the retrieval method, architecture, and customization before you ever look at a metric.

Core Framework

Key Principle 1: Understand the Retrieval Method Before the Metric

Every AI search vendor implements one of three core retrieval methods: dense vector search (e.g., OpenAI embeddings + ANN), hybrid search (BM25 + vector with fusion), or learned sparse retrieval (e.g., SPLADE, ColBERT). These methods produce very different recall and precision patterns depending on the domain. For example, dense vector search excels at semantic similarity but fails on exact keyword matches (e.g., part numbers, product codes). Hybrid search handles both but introduces latency from two-stage retrieval and fusion. Learned sparse methods like SPLADE achieve high recall on specialized domains but require custom training data.

If you evaluate only the final metric, you miss whether the vendor’s method aligns with your data. A metric like NDCG@10 can be high for a dense vector vendor on a dataset of generic product descriptions, but on a medical database with precise terminology, the same vendor might miss 40% of relevant results because the embedding model was not trained on medical jargon. The rule: request the vendor’s retrieval method, index structure, and embedding model details before you ask for any benchmark score.

Key Principle 2: Map Vendor Architecture to Your Use Case Constraints

The retrieval method is only half the story. The vendor’s architecture – indexing pipeline, query latency, cost per query, scaling model, data residency, and customization options – determines whether the method can be deployed in your real-world environment. A vendor that uses a proprietary vector database with a single-node index might show 10ms latency on a demo, but when you scale to 100 million vectors, it requires a multi-node cluster that costs $200,000 per year. Meanwhile, an open-source-based vendor using Elasticsearch with a hybrid search plugin might have lower raw accuracy but 10x cheaper infrastructure and full control over data.

Create a table of architectural characteristics for each vendor under evaluation. For each, note: index type (flat, IVFFlat, HNSW, PQ), embedding dimension, re-ranking strategy (if any), query plan (single-stage vs two-stage), batch indexing throughput, and data compliance (SOC2, GDPR, HIPAA). Then map these to your non-negotiable constraints: maximum P95 latency (e.g., 200ms), monthly query budget (e.g., $5,000), data residency (e.g., must be in EU), and customization (e.g., ability to fine-tune the embedding model on your domain). A vendor that fails on any constraint should be eliminated before you look at a single metric.

Step-by-Step Execution

1. Define Your Search Use Case and Constraints

Write a one-page document describing your exact search scenario: number of documents (e.g., 5 million product SKUs), average document length (e.g., 150 words), query types (e.g., 70% short semantic queries, 30% exact part numbers), target latency (P95 < 200ms), monthly query volume (e.g., 10 million), and budget. Also list data privacy requirements (e.g., data cannot leave the EU, need SOC2 Type II). This document becomes the filter that every vendor must pass. Share it with shortlisted vendors before any demo. If a vendor cannot commit to these constraints in writing, drop them.

1. Request a Technical Architecture Document

Ask each vendor for a detailed document describing their retrieval method, index structure, embedding model, re-ranking pipeline, and infrastructure. Require specifics: index type (e.g., HNSW with M=32, ef_construction=200), embedding model (e.g., text-embedding-3-large with 256 dimensions), and fusion method for hybrid search (e.g., weighted sum with reciprocal rank fusion). If the vendor refuses or provides only marketing language, reject them. Compare architecture documents against your constraints from Step 1. For example, a vendor using a 1536-dimension embedding with a flat index on 5 million documents will likely have unacceptable latency – disqualify them.

1. Run a Blind "Method-Only" Test

Prepare a small, representative dataset (e.g., 10,000 documents from your own corpus) and 50 test queries covering all query types. Provide the same dataset to each vendor, but do not reveal the test queries. Ask each vendor to return the top 10 results for each query using their default method, without any tuning. Then manually grade a sample of 20 queries (e.g., relevancy scoring 0-3). This test isolates the method’s baseline performance, independent of hyperparameter tuning or metric manipulation. Compare the raw results: how many exact matches appear? How many semantic matches? Note if the vendor’s method fails on keyword queries entirely. This step alone often reveals which vendors are fundamentally unsuitable.

1. Evaluate Latency and Cost Under Realistic Load

After the method test, ask each vendor to run a load test on your full dataset (or a realistic subset of 500k documents). Use a tool like Locust or k6 to simulate 100 concurrent users sending 10,000 queries per minute. Measure P95 latency, P99 latency, and error rate. Simultaneously, request a cost estimate: storage cost per million documents, cost per query (including indexing, query, and re-ranking), and any egress fees. For a 10-million-query monthly load, even a $0.0001 difference per query results in $1,000/month. Create a table:

Vendor B might have lower latency but higher cost, or vice versa. Choose based on your budget and latency SLA.

1. Assess Data Privacy and Compliance

Verify that the vendor’s infrastructure supports your data residency requirements. Ask for a signed data processing agreement (DPA) and proof of compliance certifications (SOC2 Type II, ISO 27001, HIPAA BAA, GDPR). Also check if the vendor uses your data to train their models (many do). If they do, and you have sensitive data, reject them. For example, a vendor that ships all queries to a US-based cloud for re-ranking may violate GDPR if your data is in the EU. Document the vendor’s data handling in a compliance matrix.

1. Score the Vendor's Roadmap and Customization

A vendor’s future direction matters. Ask about their ability to fine-tune the embedding model on your domain (e.g., using contrastive learning on your queries), support for custom ranking features (e.g., boosting by popularity or freshness), and API flexibility (e.g., custom scoring scripts). Interview the product manager or CTO. If their roadmap is locked to a single model release per year and they cannot accommodate custom training, you will be stuck with a generic search that will never improve. Score each vendor on a 1-5 scale for customization, roadmap alignment, and support responsiveness.

1. Finalize Scorecard with Weighted Criteria

Combine all evaluations into a weighted scorecard. Assign weights based on your priorities. Example: Method Fit (30%), Latency (20%), Cost (20%), Compliance (15%), Customization (15%). Convert each vendor’s performance into a numeric score (0-100). Use a table:

Vendor C wins, but only after method-first evaluation. If you had used only NDCG@10, you might have picked Vendor A, which would have cost you 50% more and lacked customization.

Common Mistakes

• ❌ Mistake 1: Comparing metrics across different evaluation datasets. One vendor’s 0.95 recall on their internal benchmark may be meaningless if your dataset has different distribution. Always evaluate on your own data.
• ❌ Mistake 2: Ignoring latency until after the vendor is chosen. A 50ms difference in P95 latency can double your infrastructure cost if you need to scale. Measure latency under realistic load, not just a single query.
• ❌ Mistake 3: Not asking about the embedding model’s training data. If the vendor uses a general-purpose model (e.g., OpenAI ada-002), it may fail on domain-specific terminology. Inquire about fine-tuning options.
• ❌ Mistake 4: Assuming hybrid search is always better. Hybrid search adds latency and complexity. For a domain with mostly semantic queries and no exact keyword needs, pure dense vector may be faster and cheaper.
• ❌ Mistake 5: Overlooking data privacy in the evaluation. A vendor that processes queries through a third-party API for re-ranking may expose your data. Always map the full data flow before signing.
• ❌ Mistake 6: Accepting a vendor’s claim of “99% accuracy” without method details. Such claims are often based on narrow test sets. Demand the method documentation and run your own blind test.

Metrics to Track

• Precision@k: Proportion of relevant results in top k. Target: >0.8 for your domain. But interpret only in context of the retrieval method – dense vector may have lower precision@5 than hybrid on exact queries.
• Recall@k: Proportion of all relevant documents retrieved in top k. Target: >0.9 for high-stakes applications (e.g., legal discovery). Measure on your own query set.
• P95 Latency: The 95th percentile query response time in milliseconds. Target: <200ms for user-facing search, <500ms for internal search.
• Cost per Query: Total cost (compute + storage + egress) divided by total queries. Target: <$0.0001 for high-volume applications.
• Indexing Throughput: Number of documents indexed per second. Target: at least 100 docs/sec for a 10 million document corpus.
• Customization Score: 1-5 rating based on the vendor’s ability to fine-tune the model, add custom ranking features, and adjust the retrieval method. Target: 4+ for long-term flexibility.

Checklist

• [ ] Draft a one-page use case document with constraints (data volume, latency, cost, compliance).
• [ ] Send technical architecture questionnaire to each vendor.
• [ ] Collect and compare method documentation (index type, embedding model, fusion strategy).
• [ ] Run a blind method-only test on your own dataset with 50 queries.
• [ ] Conduct load test with 100 concurrent users; measure P95 latency and error rate.
• [ ] Request cost estimates: storage, query, indexing, egress.
• [ ] Verify compliance certifications and data handling agreements.
• [ ] Score customization and roadmap alignment through interviews.
• [ ] Build weighted scorecard and compute final scores.
• [ ] Select top vendor and perform a two-week pilot on production data before full commit.

How to Use This Scorecard in a Vendor Evaluation

1. Prepare your dataset and query set. Export 10,000 representative documents from your data store. Write 50 queries that capture the diversity of your real user queries (e.g., 20 short semantic, 15 long-tail, 10 exact part numbers, 5 multi-word phrases). Label each query with its expected relevant documents (you can do this manually or use a sample from your existing search logs).
2. Send the dataset to vendors. Do not send the queries. Ask each vendor to index the dataset and return the top 10 results for each of the 50 queries using their default method. Give them a 48-hour deadline.
3. Grade the results. For each query, assign a relevance score (0-3) to the vendor’s top 10 results. Compute Precision@10, Recall@10, and a custom "keyword match rate" (percentage of results that contain exact query terms). This last metric is critical for domains where exact matches matter.
4. Run the load test. Use a tool like Locust with a script that sends 10,000 queries per minute for 10 minutes. Record P95 latency and error rate. Also ask the vendor to provide a cost estimate based on that query volume.
5. Score each vendor using the weighted scorecard template above. Apply the weights that reflect your priorities.
6. Conduct a two-week pilot with the top vendor, using your full production dataset and real user traffic. Monitor metrics like latency, cost, user satisfaction (via click-through rate or feedback). Use this pilot to confirm the scorecard results before signing a contract.

Frequently Asked Questions

Q: Should I ignore metrics entirely in the evaluation?

No. Metrics are essential, but they must be evaluated after you understand the method. Use metrics to compare vendors within the same method and dataset, not to compare across different architectures. For example, compare NDCG@10 between two hybrid search vendors, not between a dense vector vendor and a hybrid vendor.

Q: What if a vendor refuses to share their architecture details?

That is a red flag. A reputable vendor will provide a technical architecture document. If they claim it’s proprietary, ask for a non-disclosure agreement (NDA). If they still refuse, eliminate them. You cannot make an informed decision without understanding the retrieval method.

Q: How do I compare hybrid search vs pure vector search when both score well on my test set?

Use the “method-only” test with a focus on exact keyword queries. If your domain relies heavily on exact matches (e.g., part numbers, product codes), pure vector search will likely fail on those queries. Hybrid search will handle both. Also compare latency under load – hybrid search often adds 50-100ms due to the fusion step.

Q: What is the minimum precision I should accept for a production search system?

It depends on your use case. For e-commerce, a Precision@10 of 0.7 is often acceptable because users can scroll. For legal or medical search, you need >0.95 to avoid missing critical information. The key is to set a baseline using your existing search system (if any) and aim for a 20% improvement.

Q: Can I trust vendor-provided benchmarks like MS MARCO or BEIR?

Use them only as a sanity check. If a vendor performs poorly on BEIR, they are likely weak. But a high score on BEIR does not guarantee performance on your domain. Always run your own evaluation.

Q: How often should I re-evaluate my vendor choice?

Re-evaluate annually or when your data volume grows by 2x, your query type shifts, or your budget changes. The search landscape evolves quickly – new methods like ColBERTv2 or SPLADE-v2 may outperform your current vendor.

Sources

1. Google Research, "Understanding Dense Retrieval Models" (2022)
2. Microsoft Research, "SPLADE: Sparse Lexical and Dense Retrieval" (2021)
3. Stanford NLP Group, "ColBERT: Efficient and Effective Passage Search" (2020)
4. Elastic, "Hybrid Search: Combining BM25 and Vector Search" (2023)
5. SOC 2 Trust Services Criteria, AICPA (2022)
6. GDPR, "Data Protection Principles" (2018)
7. ACM SIGIR, "Evaluation Metrics for Information Retrieval" (2021)
8. Gartner, "Magic Quadrant for Insight Engines" (2023)

Using NQZAI for This Playbook

NQZAI accelerates each step of the scorecard process. Use NQZAI’s Vendor Comparison Engine to automatically populate the technical architecture questionnaire by scraping vendor documentation and recent blog posts. The Method Fit Analyzer takes your use case document and returns a match score between your requirements and each vendor’s retrieval method. For the blind method-only test, NQZAI’s Evaluation Workbench lets you upload your dataset, push queries to multiple vendors through a single API, and compute Precision@k, Recall@k, and keyword match rate in one dashboard. The Load Test Simulator sends synthetic traffic to each vendor’s API endpoint and records P95 latency, error rates, and cost estimates. Finally, the Weighted Scorecard Builder dynamically generates the final table with your specified weights, allowing you to compare vendors side-by-side. NQZAI does not replace the method-first evaluation principle – it removes the manual overhead of gathering data and running tests, so you can focus on interpreting the results and making the right decision.