The Evaluation-Driven Development Approach

AI applications with questionable returns on investment are remarkably common. This happens not only because generative applications are intrinsically hard to evaluate, but also because builders frequently lack observability into how their systems perform in the wild.

• An ML engineer at a used car dealership deployed a model to predict vehicle values based on owner specs. A year later, users liked the feature, but the engineering team had no idea whether predictions were actually accurate.
• During early chatbot hype, enterprises rushed to launch customer support bots without metrics to determine whether they improved or degraded customer experience.

Inspired by test-driven development (TDD) in software engineering, evaluation-driven development establishes evaluation criteria before building.

The Four Evaluation Buckets

Every AI application should begin with a tailored list of criteria categorized across four fundamental buckets:

Domain-Specific Capability

A model's specialized capabilities are fundamentally constrained by its architecture, size, and pre-training data distribution. If an application requires translating Latin to English, a model that never encountered Latin during pre-training simply cannot perform the task.

Thousands of domain benchmarks exist across code generation, debugging, grade-school math, medicine, reasoning, tool usage, and game playing.

Functional Correctness and Execution Efficiency

For code and query generation, evaluation traditionally relies on functional correctness (running tests against generated code). However, correctness alone is insufficient:

• Efficiency Benchmarking: BIRD-SQL evaluates both execution accuracy and runtime efficiency by comparing generated SQL execution times against optimized ground-truth queries.
• Readability & Maintainability: Code that runs but is indecipherable creates immense technical debt. Because readability cannot be measured deterministically, it typically requires subjective scoring via AI judges.

Close-Ended Benchmarks vs. Open-Ended Tasks

Non-coding capabilities are overwhelmingly evaluated using close-ended setups such as multiple-choice questions (MCQs):

Example: MMLU Multiple-Choice Item

Question: One of the reasons that the government discourages and regulates monopolies is that:

• (A) Producer surplus is lost and consumer surplus is gained.
• (B) Monopoly prices ensure productive efficiency but cost society allocative efficiency.
• (C) Monopoly firms do not engage in significant research and development.
• (D) Consumer surplus is lost with higher prices and lower levels of output.

Ground Truth: (D)

MCQs offer clear advantages: they are cheap to grade, yield unambiguous accuracy scores, and have an explicit random guessing baseline (e.g., 25% for 4 options).

Generation Capability

Natural language generation (NLG) evaluation historically tracked two properties:

• Fluency: Grammatical correctness and natural phrasing.
• Coherence: Logical structure across paragraphs.

With frontier models, AI prose is virtually indistinguishable from human writing, making basic fluency rarely a discriminator. Instead, generation evaluation centers on factual consistency and safety.

Factual Consistency

Hallucinations are acceptable in creative storytelling, but fatal in enterprise automation. Factual consistency is measured across two distinct paradigms:

AI as a Judge for Consistency

Prompting an advanced model (e.g., GPT-4) to evaluate factual alignment frequently outperforms traditional ngram metrics (Liu et al., 2023; Luo et al., 2023):

Factual Consistency: Does the summary contain untruthful or misleading facts that are not supported by the source text?

Source Text:
{{Document}}

Summary:
{{Summary}}

Does the summary contain factual inconsistency?
Answer:

Advanced Verification Architectures

1. Self-Verification (SelfCheckGPT): Generates $N$ stochastic completions. If independent samples disagree with one another on factual claims, the primary output is flagged as hallucinated (Manakul et al., 2023).
2. Search-Augmented Factuality Evaluator (SAFE): Google DeepMind's framework (Wei et al., 2024) breaks long responses into atomic claims, issues search queries via Google Search API, and verifies each claim against search results.

Natural Language Inference (NLI)

Factual consistency can be framed as classical textual entailment:

• Entailment: Premise supports hypothesis $\rightarrow$ Factually consistent
• Contradiction: Premise refutes hypothesis $\rightarrow$ Factually inconsistent
• Neutral: Premise neither confirms nor refutes $\rightarrow$ Undetermined

Small cross-encoder models (e.g., DeBERTa-v3-base fine-tuned on MNLI/FEVER) provide rapid, cost-effective entailment classification.

TruthfulQA Benchmark

TruthfulQA spans 817 questions designed around common human misconceptions across 38 categories:

Safety and Moderation

Safety encompasses toxicity, profanity, violent guidance, hate speech, stereotype propagation, and systematic bias.

Moderation Tooling & Benchmarks

• Specialized Classifiers: Smaller models such as Perspective API, RoBERTa toxicity classifiers, or Meta's Llama Guard deliver sub-50ms moderation scoring.
• Benchmark Suites: RealToxicityPrompts (100,000 provocation prompts) and BOLD (bias in open-ended language generation).

Instruction-Following Capability

A model may possess immense domain knowledge yet fail completely if it cannot follow instructions. If a sentiment classifier instructed to output POSITIVE, NEGATIVE, or NEUTRAL responds with HAPPY, downstream parsers break immediately.

Verifiable Instructions: IFEval and INFOBench

Roleplaying and Personas

Roleplaying represents one of the most frequent real-world instruction formats (LMSYS Chatbot Arena study).

When evaluating persona fidelity (e.g., CharacterEval, RoleLLM), models must be scored on:

1. Style & Tone: Maintaining linguistic idiosyncrasies without drifting into generic assistant prose.
2. Negative Knowledge: Refusing to discuss facts or capabilities that the persona cannot possess (e.g., preventing a non-playable video game character from leaking future plot points).

Cost and Latency

High-quality responses are useless if delivered too slowly or expensively for the economics of your product.

Multi-Objective Pareto Optimization

Inference tradeoffs rarely offer an absolute optimum:

• If latency is non-negotiable (e.g., real-time autocomplete), establish strict P90/P99 latency thresholds, eliminate failing models, and optimize quality among survivors.
• Key Latency Metrics: Time to First Token (TTFT), Inter-Token Latency (ITL), and End-to-End Query Duration.

Commercial APIs vs. Self-Hosted Economics

• Model APIs: Charge linearly per input and output token. Marginal cost per token remains relatively flat as traffic expands.
• Self-Hosted Clusters: High fixed infrastructure and engineering cost, but marginal token cost drops dramatically as throughput approaches GPU saturation.