Pointwise Versus Comparative Evaluation

Comparative Evaluation in AI

In AI, comparative evaluation was first used in 2021 by Anthropic to rank different models. It also powers the popular LMSYS's Chatbot Arena leaderboard that ranks models using scores computed from pairwise model comparisons from the community.

Many model providers use comparative evaluation to evaluate their models in production. Figure 3-10 shows an example of ChatGPT asking its users to compare two outputs side by side. These outputs could be generated by different models, or by the same model with different sampling variables.

Figure 3-10. ChatGPT occasionally asks users to compare two outputs side by side.

For each request, two or more models are selected to respond. An evaluator, which can be human or AI, picks the winner. Many developers allow for ties to avoid a winner being picked at random when drafts are equally good or bad.

When collecting comparative feedback from users, one challenge is to determine what questions can be determined by preference voting and what shouldn't be. Preference-based voting only works if the voters are knowledgeable in the subject. This approach generally works in applications where AI serves as an intern or assistant, helping users speed up tasks they know how to do—and not where users ask AI to perform tasks they themselves don't know how to do.

Comparative evaluation shouldn't be confused with A/B testing.

From Matches to a Ranking

Each comparison is called a match. This process results in a series of comparisons, as shown in Table 3-5.

Table 3-5. Examples of a history of pairwise model comparisons.

The probability that model A is preferred over model B is the win rate of A over B. We can compute this win rate by looking at all matches between A and B and calculating the percentage in which A wins.

If there are only two models, ranking them is straightforward. The model that wins more often ranks higher. The more models there are, the more challenging ranking becomes. Let's say that we have five models with the empirical win rates between model pairs, as shown in Table 3-6. It's not obvious, from looking at the data, how these five models should be ranked.

Table 3-6. Example win rates of five models. The $A >> B$ column denotes the event that $A$ is preferred to $B$.

Given comparative signals, a rating algorithm is then used to compute a ranking of models. Typically, this algorithm first computes a score for each model from the comparative signals and then ranks models by their scores.

Comparative evaluation is new in AI but has been around for almost a century in other industries. It's especially popular in sports and video games. Many rating algorithms developed for these other domains can be adapted to evaluating AI models.

Through this lens, model ranking is a predictive problem. We compute a ranking from historical match outcomes and use it to predict future match outcomes. Different ranking algorithms can produce different rankings, and there's no ground truth for what the correct ranking is. The quality of a ranking is determined by how good it is in predicting future match outcomes. My analysis of Chatbot Arena's ranking shows that the produced ranking is good, at least for model pairs with sufficient matches. See the book's GitHub repo for the analysis.

Challenges of Comparative Evaluation

With pointwise evaluation, the heavy-lifting part of the process is in designing the benchmark and metrics to gather the right signals. Computing scores to rank models is easy. With comparative evaluation, both signal gathering and model ranking are challenging. This section goes over the three common challenges of comparative evaluation.

Scalability bottlenecks

Comparative evaluation is data-intensive. The number of model pairs to compare grows quadratically with the number of models. In January 2024, LMSYS evaluated 57 models using 244,000 comparisons. Even though this sounds like a lot of comparisons, this averages only 153 comparisons per model pair (57 models correspond to 1,596 model pairs). This is a small number, considering the wide range of tasks we want a foundation model to do.

Fortunately, we don't always need direct comparisons between two models to determine which one is better. Ranking algorithms typically assume transitivity. If model A ranks higher than B, and B ranks higher than C, then with transitivity, you can infer that A ranks higher than C. This means that if the algorithm is certain that A is better than B and B is better than C, it doesn't need to compare A against C to know that A is better.

There's also the challenge of evaluating new models. With independent evaluation, only the new model needs to be evaluated. With comparative evaluation, the new model has to be evaluated against existing models, which can change the ranking of existing models. This also makes it hard to evaluate private models. Imagine you've built a model for your company, using internal data. You want to compare this model with public models to decide whether it would be more beneficial to use a public one. If you want to use comparative evaluation for your model, you'll likely have to collect your own comparative signals and create your own leaderboard or pay one of those public leaderboards to run private evaluation for you.

The scaling bottleneck can be mitigated with better matching algorithms. So far, we've assumed that models are selected randomly for each match, so all model pairs appear in approximately the same number of matches. However, not all model pairs need to be equally compared. Once we're confident about the outcome of a model pair, we can stop matching them against each other. An efficient matching algorithm should sample matches that reduce the most uncertainty in the overall ranking.

Lack of standardization and quality control

One way to collect comparative signals is to crowdsource comparisons to the community the way LMSYS Chatbot Arena does.

The benefit of this approach is that it captures a wide range of signals and is relatively difficult to game.

However, the downside is that it's hard to enforce standardization and quality control. First, anyone with internet access can use any prompt to evaluate these models, and there's no standard on what should constitute a better response. It might be a lot to expect volunteers to fact-check the responses, so they might unknowingly prefer responses that sound better but are factually incorrect.

Some people might prefer polite and moderate responses, while others might prefer responses without a filter. This is both good and bad. It's good because it helps capture human preference in the wild. It's bad because human preference in the wild might not be appropriate for all use cases. For example, if a user asks a model to tell an inappropriate joke and a model refuses, the user might downvote it. However, as an application developer, you might prefer that the model refuses. Some users might even maliciously pick the toxic responses as the preferred ones, polluting the ranking.

Second, crowdsourcing comparisons require users to evaluate models outside of their working environments. Without real-world grounding, test prompts might not reflect how these models are being used in the real world. People might just use the first prompts that come to mind and are unlikely to use sophisticated prompting techniques.

Among 33,000 prompts published by LMSYS Chatbot Arena in 2023, 180 of them are "hello" and "hi", which account for 0.55% of the data, and this doesn't yet count variations like "hello!", "hello.", "hola", "hey", and so on. There are many brainteasers. The question "X has 3 sisters, each has a brother. How many brothers does X have?" was asked 44 times.

If a public leaderboard doesn't support sophisticated context construction, such as augmenting the context with relevant documents retrieved from your internal databases, its ranking won't reflect how well a model might work for your RAG system. The ability to generate good responses is different from the ability to retrieve the most relevant documents.

From comparative performance to absolute performance

For many applications, we don't necessarily need the best possible models. We need a model that is good enough. Comparative evaluation tells us which model is better. It doesn't tell us how good a model is or whether this model is good enough for our use case. Let's say we obtained the ranking that model B is better than model A. Any of the following scenarios could be valid:

You need other forms of evaluation to determine which scenario is true.

Imagine that we're using model A for customer support, and model A can resolve 70% of all the tickets. Consider model B, which wins against A 51% of the time. It's unclear how this 51% win rate will be converted to the number of requests model B can resolve. Several people have told me that in their experience, a 1% change in the win rate can induce a huge performance boost in some applications but just a minimal boost in other applications.

When deciding to swap out A for B, human preference isn't everything. We also care about other factors like cost. Not knowing what performance boost to expect makes it hard to do the cost–benefit analysis. If model B costs twice as much as A, comparative evaluation isn't sufficient to help us determine if the performance boost from B will be worth the added cost.

The Future of Comparative Evaluation

Given so many limitations of comparative evaluation, you might wonder if there's a future to it. There are many benefits to comparative evaluation.

First, as discussed in "Post-Training" on page 78, people have found that it's easier to compare two outputs than to give each output a concrete score. As models become stronger, surpassing human performance, it might become impossible for human evaluators to give model responses concrete scores. However, human evaluators might still be able to detect the difference, and comparative evaluation might remain the only option. For example, the Llama 2 paper shared that when the model ventures into the kind of writing beyond the ability of the best human annotators, humans can still provide valuable feedback when comparing two answers (Touvron et al., 2023).

Second, comparative evaluation aims to capture the quality we care about: human preference. It reduces the pressure to have to constantly create more benchmarks to catch up with AI's ever-expanding capabilities. Unlike benchmarks that become useless when model performance achieves perfect scores, comparative evaluations will never get saturated as long as newer, stronger models are introduced.

Comparative evaluation is relatively hard to game, as there's no easy way to cheat, like training your model on reference data. For this reason, many trust the results of public comparative leaderboards more than any other public leaderboards.

Comparative evaluation can give us discriminating signals about models that can't be obtained otherwise. For offline evaluation, it can be a great addition to evaluation benchmarks. For online evaluation, it can be complementary to A/B testing.