Evaluate AI Systems

Model Selection

A strategic guide to selecting foundation models, comparing self-hosting vs model APIs, and critically navigating public benchmarks and leaderboards.

Model Selection

At the end of the day, you don't care which model is universally "the best" — you care which model is best for your application. Model selection maps candidate models along capability, latency, and cost frontiers to find the optimal trade-off.

The Selection Workflow

Model selection is not a one-time decision; it is an iterative discipline revisited across prompt engineering, retrieval tuning, and fine-tuning.

When evaluating candidate models, distinguish between hard attributes and soft attributes:

Hard Attributes (Fixed Constraints)

Attributes governed by organizational policy or model architecture that are impractical or impossible to alter: commercial licenses, data privacy requirements, model size, and hardware limits.

Soft Attributes (Malleable)

Attributes that can be improved through engineering effort: accuracy, factual consistency, output format, or latency (when self-hosting).

Four-Step Evaluation Funnel

Figure 4-5. An overview of the evaluation workflow to evaluate models for your application.

Filter by Hard Constraints: Eliminate models violating legal, compliance, privacy, or infrastructure requirements.
Narrow via Public Benchmarks: Leverage public benchmarks and leaderboards to identify 3–5 top candidate models.
Internal Pipeline Evaluation: Replay representative traffic through custom evaluation suites to determine domain-specific performance.
Production Observability: Continuously monitor model drift, error modes, and user feedback post-deployment.

Model Build vs. Buy: Self-Hosting vs. Model APIs

Deploying a model requires an inference service that receives user prompts, executes forward passes, and returns completions.

Figure 4-6. An inference service runs the model and provides an interface for users to access the model.

Open Weights vs. Truly Open Models

Open Weight: Model weights are publicly downloadable, but training code, curation recipes, and pre-training datasets remain proprietary (e.g., Llama 3, Mistral 7B).
Open Model / Open Data: Both weights and full training datasets are publicly auditable (e.g., OLMo).

Commercial License Traps

Open-weight licenses often contain strict usage conditions:

User Thresholds: Llama 2/3 licenses mandate explicit commercial agreements for products exceeding 700M monthly active users.
Distillation Bans: Many providers forbid using model completions to train or fine-tune competing models.

Seven Critical Decision Axes

1. Data Privacy & Sovereignty

Sending data to external APIs is unacceptable for organizations with strict compliance mandates or air-gapped systems (e.g., Samsung's proprietary data leak via ChatGPT).

2. Data Lineage & Legal Risk

Training data opacity creates copyright exposure. Commercial contracts often provide copyright indemnification; open-source models place legal liability squarely on the user.

3. Capability Frontier

While open models are closing the gap, frontier closed models retain advantages in complex reasoning and tool use due to massive compute investments.

4. Features vs. Transparency

Commercial APIs provide out-of-the-box function calling and JSON modes, but rarely expose token logprobs. Self-hosted models grant full access to logits, embeddings, and weights.

5. API Cost vs. Engineering Cost

APIs charge linearly per token; self-hosting involves fixed GPU clusters and specialized MLOps talent. At high sustained volume, self-hosting becomes drastically cheaper.

6. Version Control & Lock-in

Commercial APIs frequently update or deprecate model snapshots without notice, altering prompt behaviors. Self-hosting allows freezing exact weights indefinitely.

Figure 4-7. The gap between open source models and proprietary models is decreasing on the MMLU benchmark. Image by Maxime Labonne.

Figure 4-8. Why enterprises care about open source models. Image from the 2024 study by a16z.

Comprehensive Comparison Matrix

Dimension	Managed Model APIs	Self-Hosted Open Weights
Data Privacy	Sensitive data leaves infrastructure; risk of provider training on prompts	Zero data egress; runs entirely within private VPC or air-gapped on-premise
Capabilities	Frontier reasoning, multimodal inputs, massive context windows	Competitive for 80% of tasks; slightly lags frontier models
Functionality	Turnkey tool use, structured outputs, managed guardrails	Full logprob access, custom logits processors, arbitrary fine-tuning
Financials	Pay-as-you-go per token; high variable cost at massive scale	High capital expenditure/fixed compute, but near-zero marginal token cost
Operational Burden	Zero infrastructure maintenance; dependent on third-party uptime/SLA	Requires GPU provisioning, cluster autoscaling, quantization, and monitoring
Deployment Edge	Requires active internet connection; subject to global API latency	Supports local, offline, on-device execution (smartphones, IoT, local desktops)

Navigating Public Benchmarks & Leaderboards

Standardized benchmark suites measure broad capabilities across standardized tasks:

BIG-bench: 200+ multi-task capabilities curated by Google Research.
lm-evaluation-harness: EleutherAI's framework unifying 400+ evaluation suites.
OpenAI Evals: Automated framework for evaluating closed models on standard suites.

The Illusion of Simple Averages

Public leaderboards (such as Hugging Face's Open LLM Leaderboard or Stanford HELM) aggregate multiple benchmarks to create unified rankings.

Flaws in Public Leaderboard Rankings

Unweighted Averaging: Simple averages treat an 80% score on math the same as an 80% score on common sense, regardless of difficulty.
Benchmark Correlation: As shown in Table 4-5, MMLU, WinoGrande, and ARC-C have Pearson correlations > 0.85. Averaging them overweights general reasoning while under-representing truthfulness or coding.

Benchmark	ARC-C	HellaSwag	MMLU	TruthfulQA	WinoGrande	GSM-8K
ARC-C	1.0000	0.4812	0.8672	0.4809	0.8856	0.7438
HellaSwag	0.4812	1.0000	0.6105	0.4228	0.4842	0.3547
MMLU	0.8672	0.6105	1.0000	0.5507	0.9011	0.7936
TruthfulQA	0.4809	0.4228	0.5507	1.0000	0.4550	0.5009
WinoGrande	0.8856	0.4842	0.9011	0.4550	1.0000	0.7979
GSM-8K	0.7438	0.3547	0.7936	0.5009	0.7979	1.0000

Benchmark Saturation & Modern Suites

As models conquer legacy benchmarks, evaluations rapidly shift from grade-school math to graduate-level problem solving:

MMLU $\rightarrow$ MMLU-PRO: Expanded distractors, multi-step chain-of-thought requirements.
GSM-8K $\rightarrow$ MATH Level 5: Advanced competition-grade mathematics.
GPQA: Graduate-level physics, chemistry, and biology questions validated by PhD experts.

Are Frontier Models Getting Worse?

Users frequently report that models degrade after provider updates. A Stanford/UC Berkeley study (Chen et al., 2023) confirmed measurable shifts in performance across versions:

Figure 4-9. Changes in the performances of GPT-3.5 and GPT-4 from March 2023 to June 2023 on certain benchmarks (Chen et al., 2023).

Model updates are rarely strictly superior across all dimensions. Fine-tuning to improve safety or instruction adherence frequently causes regressions in coding syntax or niche reasoning. Prompts are not portable across model snapshots.

Data Contamination in Public Benchmarks

Data contamination (data leakage or training on the test set) occurs when evaluation prompts exist inside the model's pre-training web scrape.

In a famous demonstration, Rylan Schaeffer (2023) trained a tiny 1-million parameter model exclusively on public benchmark test sets, achieving near-perfect scores that humiliated massive models.

Detection and Decontamination

N-gram Overlap: Scans the pre-training corpus for exact 8–13 token overlaps with test sets. Highly accurate, but requires access to raw pre-training text dumps.
Perplexity Probing: Identifies test samples where the model exhibits anomalously low perplexity, indicating rote memorization.

Figure 4-10. Relative difference in GPT-3's performance when evaluating using only the clean sample compared to evaluating using the whole benchmark.

Public benchmarks filter out bad models, but they cannot identify the best model for your unique workload. You must engineer a custom, private evaluation pipeline.

Edit this pageorReport an issue

Evaluation Criteria

How to define and calculate criteria for evaluating AI applications, including domain capabilities, factual consistency, safety, instruction-following, and cost-latency tradeoffs.

Design Your Evaluation Pipeline

A practical guide to architecting production evaluation pipelines, defining scoring rubrics, slicing datasets, and avoiding Simpson's paradox.