# How the meta-leaderboard works: our methodology

> Bayesian pairwise rank aggregation explained. How we combine three public TTS leaderboards into one trustworthy ranking using fractional scaling, coverage smoothing, Borda counts, and Schulze tiebreaks.

## Key Information
- Published: Aug 13, 2026
- Category: Methodology
- Summary: Most aggregated rankings hide compromises behind a single number. Our methodology is public, versioned, and every score traces to a source snapshot you can download.

## Content
The hard part of comparing TTS models is not finding data. The hard part is combining data from sources that disagree about which model is best. Each public leaderboard comes with built-in blind spots.

Artificial Analysis runs blind preference tests but only covers models that vendors submit. The Humanness Index measures how human a voice sounds but uses a smaller sample pool. Voice Arena collects real listener votes but the voter population shifts over time. None of these boards are wrong. They are all incomplete.

Simple averages make things worse. Take the mean of each model's scores and you let a single number one placement on a large board dominate the result. A model that ranks first on a 1000-model board gets a huge score advantage. That tells you the board is big, not that the model is better.

Medians ignore information. Drop the highest and lowest scores and you throw out signal from boards that actually differentiate models well. You lose the shape of the data entirely.

We needed something better. So we built a four-phase pipeline that treats each board's ranking as evidence, not truth.

## What is Bayesian pairwise rank aggregation and why does it matter for TTS?

Bayesian Pairwise Rank Aggregation, or BPRA, is a statistical method for combining head to head comparisons into a single consensus ranking. It works by taking pairwise inputs (model A beat model B, model C lost to model D) and using probability models to estimate the true skill of each item in the field. The output is not just a ranking. It's a ranking with uncertainty attached: confidence intervals that tell you how sure the system is about each position.

This matters for TTS because there is no single objective test for voice quality. Every board measures something slightly different. Preference is subjective. Listening conditions vary. Listener populations drift. BPRA handles all of that noise by design. It was originally developed for aggregating crowdsourced labels and chess rankings, where the same problems (unreliable judges, missing data, circular preferences) show up constantly.

Every board we aggregate already uses a Bayesian pairwise method internally. Artificial Analysis runs an Elo system, which is a simplified Bayesian pairwise model. Voice Arena uses Bradley-Terry, a close cousin. The Humanness Index uses its own pairwise voting pipeline. They all solve the same core problem: given noisy head to head votes, find the true ranking. They just solve it at different scales with different priors.

Our meta-leaderboard does not replace those methods. It aggregates their outputs. We treat each board's ranking as evidence in a higher-level Bayesian model, combining three posterior distributions into one.

## The problem with individual leaderboards

Each board tells you something real about voice quality. But each board also has its own methodology, its own blind spots, and its own biases baked into the data. You cannot compare scores across boards directly. A 95 on the Humanness Index is not the same as a 1233 Elo on Artificial Analysis. They are different scales, different sampling methods, different populations of voters.

Artificial Analysis runs controlled blind preference tests. Listeners hear two clips without knowing which model produced them and pick the one they prefer. The Elo rating then updates based on who won and who lost. But the board only includes models that vendors submit. If a vendor does not submit, their model is invisible. The field is defined by who pays attention to the board, not by who builds good TTS.

The Humanness Index asks listeners to rate how human a voice sounds on a 0-100 scale. It is a direct measure of naturalness, which is a critical dimension for voice apps. But the sample pool is smaller than Artificial Analysis, and the rating scale introduces its own biases. Different listeners anchor on different parts of the scale. One person's 85 is another person's 92. The board normalizes for this, but normalization is itself a modeling choice that introduces uncertainty.

Voice Arena runs head to head tournaments with real listener votes. The tournament format means the voter population shifts from match to match. A model that competes on Tuesday faces a different set of voters than a model that competes on Thursday. The Elo system handles this better than raw averages, but it still means the ranking is sensitive to timing. A model that entered early and accumulated many votes looks different from a model that entered yesterday and has sparse coverage.

No single board tells the whole story. A model that tops Artificial Analysis might sound robotic but win on clarity. A model that tops the Humanness Index might sound natural but lack expressiveness. A model that tops Voice Arena might appeal to the current voter demographic but fall flat with a different audience. Each board is a partial view.

The problem gets worse when you try to combine them manually. You end up with a spreadsheet of numbers from different scales, different sample sizes, and different timeframes. There is no clean way to say "this model is better than that one" across all three boards without making arbitrary weighting decisions.

## How BPRA works at the board level

Before explaining how we combine boards, it helps to understand how each board already uses Bayesian pairwise methods internally. The framework is the same across all three, even though the implementations differ.

### Pairwise input

Every board starts with head to head comparisons. In Artificial Analysis, listeners hear two clips and choose one. In Voice Arena, the tournament pairs models directly. In the Humanness Index, each voice gets rated, and those ratings are compared pairwise in the background. The input is always the same: a set of ordered pairs where one item beat or was preferred over another.

### Prior beliefs

The model starts with a prior probability distribution for each model's hidden skill level. This is the "prior" in Bayesian. Before seeing any data, the model assumes every model is roughly average. The prior then updates as evidence arrives. In Elo systems, this prior is encoded as the starting rating (usually 1000 or 1200). In Bradley-Terry models, the prior is a logit distribution over win probabilities.

### Likelihood update

When a new comparison arrives, the model computes the probability of that outcome given the current skill estimates. If model A is rated 1100 and model B is rated 1000, the model expects A to win about 64% of the time. If A loses, that outcome is unlikely under the current estimates. So the likelihood is low, and the update is large. The model shifts its estimates to better explain the observed result.

### Posterior inference

After all comparisons are processed, the model outputs a posterior distribution over each model's skill. This is the ranking with uncertainty. In Elo systems, the posterior is approximated by the final rating and a rating deviation. In Bradley-Terry models, you get full posterior distributions that can be sampled and compared. The output is not a flat ordered list. It is a set of probability distributions that tell you how likely each model is to be better than every other model.

This is the part most casual readers miss. When you see a ranking on Artificial Analysis, the order is not determined by a simple sort of Elo numbers. It is the result of a full Bayesian inference that accounts for the number of comparisons, the strength of opponents faced, and the variance in outcomes. The same model can have different confidence intervals on different boards because the data quality differs.

## Our meta-aggregation: combining three boards into one score

Our meta-leaderboard runs a four phase pipeline on the output of each source board. The goal is to produce a single 0-100 score for every model that appears on at least one board, with the score reflecting the model's performance across all available evidence.

### Phase 1: fractional scaling

Every board contributes the same thing to the pipeline: a model's rank. Rank one is always 1.0. The last row is always 1 divided by N, where N is the number of models on that board. This means third place on a 1000-model board is worth more than third place on a 10-model board. That is intentional. Being in the top 0.3% of a large field is harder than being in the top 30% of a small one.

The formula is simple: (N - R + 1) / N, where R is the rank and N is the board size. A rank of 3 on a 10-model board gives (10 - 3 + 1) / 10 = 0.8. The same rank on a 1000-model board gives (1000 - 3 + 1) / 1000 = 0.998. Same rank. Very different signal strength. Fractional scaling prevents board size from distorting the aggregate. A rank is not just a position. It is a position relative to the field size, which carries information about how competitive that placement actually was.

### Phase 2: Bayesian coverage smoothing

A model that appears on a single board and dominates there looks great on paper. But that model has never been tested against the full field. Coverage smoothing pulls models with few board appearances toward the field average using a Bayesian prior. The pull is gentle: a single strong placement still counts. It just does not count as much as a model that proves itself across every board. You don't get full credit for showing up once.

The math works like this. We compute the mean fractional score across all models on all boards. That is the prior. For each model, we weight the prior against the model's observed scores, with the weight determined by how many boards the model appears on. A model on all three boards gets almost no pull toward the mean. A model on one board gets a noticeable pull. The pull shrinks as more boards cover the model.

This is the phase that catches overconfidence. A model with one perfect score on one small board might look like a contender on paper. Coverage smoothing flags the missing data and adjusts. The model can still rise, but it needs to show up on more boards first.

Consider a concrete example. Model X ranks first on Voice Arena, with 10 models total, giving a fractional score of 1.0. But X appears on no other boards. The global mean fractional score across all models is about 0.6. With coverage smoothing, X's final score lands around 0.73, pulled noticeably toward the mean. Compare this to model Y, which ranks second on all three boards. Y's fractional scores are 0.9 on each board. With full coverage, Y gets almost no pull and stays near 0.9. Despite X being number one on one board, Y beats X in the aggregate because Y proved itself across more environments.

### Phase 3: ecosystem-normalized Borda count

This is the core aggregation step. For each pair of models, we count how many other models each one beats across every board. A model ranked first on every available board beats every other model on every board. That scores a perfect 100. Miss a board entirely and your ceiling drops, no matter how dominant you are on the boards you did enter.

The Borda count is ecosystem-normalized, which means it accounts for board size. Beating 500 models on a 1000-model board counts the same as beating 5 models on a 10-model board. Each board contributes equally to the total points available, regardless of how many models it lists. The math ensures no single board dominates the aggregate just because it happens to list more models.

Here is how the normalization works in practice. Board A has 1000 models and board B has 10 models. Without normalization, beating 500 models on board A would count 100 times more than beating 5 models on board B. That is wrong. The board size is an artifact of the board's coverage, not a measure of how competitive the ranking is. Ecosystem normalization divides each board's contribution by the total points available on that board, so each board contributes equally to the final pool.

The result is a 0-100 score that reflects a model's performance relative to the entire ecosystem it competes in. A model that enters all three boards can score up to 100. A model that enters only one board is capped at 33, regardless of how dominant it is there. We are explicit about this ceiling. You can do the same math yourself.

### Phase 4: Schulze tiebreak

Ties happen. Two models with identical aggregate scores need a tiebreaker, and random assignment is not acceptable. Schulze's method looks at the strongest chain of board-level wins between two models. If model A beats model B on three boards but loses on one, and B has no chain of wins back to A, then A wins.

The method works by constructing a directed graph of pairwise wins across boards. Each edge in the graph represents how many boards prefer one model over another. Schulze then finds the strongest path from each model to every other model. The strongest path is not necessarily the one with the most direct wins. It is the one with the highest bottleneck: the weakest link in the chain determines the path strength. This handles circular preferences (A beats B, B beats C, C beats A) better than simpler methods do.

Schulze gives you a strong ordering even when the aggregate scores are extremely close. It is the same method used in some political elections and open-source project votes. We chose it because it handles ties without introducing randomness or arbitrary tiebreakers. Every model's final position is deterministic given the same source data. You can reproduce the result yourself.

## Why this beats simple aggregation methods

You might wonder why we did not just average the scores. Or take the median. Or pick the best of N boards. Each of those methods has existed for decades and each one breaks in predictable ways.

### Simple average

Averaging scores across boards is the most common approach. It is also the most misleading. A model that ranks first on a 1000-model board gets a massive boost from that single placement, even if it performs poorly or disappears on the other boards. The average hides the fact that the model was only tested on one board. It looks like a top contender. It is actually a one-board wonder.

Take a concrete example. Model A ranks first on Artificial Analysis (1000 models) and ranks 50th on the Humanness Index (50 models) and is absent from Voice Arena. Model B ranks third on all three boards. Simple averaging makes Model A look dominant because its 1000-model board overwhelms the smaller boards. Our pipeline handles this correctly: fractional scaling normalizes the board sizes, coverage smoothing penalizes Model A for missing a board, and the Borda count caps Model A's ceiling. Model B beats Model A in our system, which is the right result.

### Median

Medians avoid the extreme-value problem of averages but introduce a different flaw: they throw away information. If a model appears on three boards and ranks 1st, 2nd, and 10th, the median is 2nd. You lost the signal from the 10th place finish, which might tell you something important about the board where the model struggled. Medians are safe but wasteful. They discard shape.

### Best of N

Taking the best score across any board sounds reasonable until you realize every model has at least one board where it shines. Every model is someone's favorite. Best-of-N turns every model into a winner, which defeats the purpose of aggregation. It is the same problem as letting each student grade their own exam and take the highest result.

### Elo blending

Elo blending combines raw pairwise data across boards using a shared Elo scale. It is the most sophisticated alternative and the one we considered most seriously. But it requires raw pairwise data from every board. We don't have that. We have each board's output ranking, not the individual match results. Artificial Analysis publishes Elo ratings but not the full match history. Voice Arena publishes rankings but not the raw vote data. The Humanness Index publishes scores but not the individual listener ratings.

Elo blending would force us to reverse-engineer pairwise data from rankings, which introduces its own biases and assumptions. The four-phase pipeline works with the data we actually have: rankings. It doesn't require access to internal board data that the board operators do not publish.

## Confidence: how sure are we about each score?

Every model on the board carries a confidence tag: High, Medium, or Low. These tags are not part of the score. They are a separate signal that tells you how much evidence supports that position.

The confidence label combines three factors.

Coverage. How many boards cover this model? A model on all three boards starts with the highest coverage signal. A model on one board starts low.

Agreement. Do the boards agree about where this model belongs? If all three boards rank the model similarly, agreement is high. If the boards place the model in wildly different positions (first on one board, last on another), agreement is low. Disagreement between boards is informative: it suggests the model's quality is context-dependent.

Each board's own confidence intervals. Every board already publishes confidence information. Artificial Analysis reports Elo deviations. Voice Arena reports rating uncertainty. The Humanness Index reports sample sizes. We incorporate these as weights. A model with wide confidence on every board gets lower agreement weight, because the boards themselves are uncertain.

The final label maps like this.

High. The model appears on at least two-thirds of boards. The boards broadly agree about its quality. The board-level confidence intervals are narrow. This is a model you can bet on.

Medium. Two or more boards cover the model, but either the boards disagree or at least one board has wide confidence. The score is reliable but worth verifying against your own testing.

Low. A single board is the only source, or the boards strongly disagree on the model's position, or the board-level confidence is very wide. The score is provisional. Use it as a lead, not a decision.

Confidence matters because not all scores are equally trustworthy. A model with a 90 and High confidence is a safer choice than a model with a 92 and Low confidence. The higher score might look better on paper, but the data behind it is thinner. We want you to see the difference. High confidence means the model has been tested widely and consistently. Low confidence means the data is sparse or contradictory. Your risk tolerance determines which matters more.

## This is live now

The meta-leaderboard runs daily. When Artificial Analysis, the Humanness Index, or Voice Arena publish a new snapshot, our pipeline pulls it, runs all four phases, and updates every model's score. No manual intervention. No editorial curation. The same formula runs on the same data for every model.

Models enter and leave the source boards all the time. A vendor submits a new model to Artificial Analysis and it appears in our pipeline the next day. A model drops off the Humanness Index and our coverage smoothing adjusts its pull toward the mean. The meta-board tracks it all automatically.

Every score traces back to a snapshot of each source board. You can download those snapshots and run the same formula. The aggregation code is open source. If you disagree with a ranking, you can trace exactly why. The Schulze chain, the coverage pull, the fractional scaling, every step is logged. We built this so you can verify, not trust.

If you are choosing a TTS model for a production app, the meta-leaderboard is the starting point. Pull up the board, sort by aggregate score, check the confidence tags, and dig into the source boards for the kind of quality you care about. No single board is complete. But the combined evidence across three boards, processed through a four-phase Bayesian pipeline, is the best way to answer the question that matters: which TTS model should you actually build on?

## Frequently asked questions

### How often does the data refresh?

The pipeline runs daily. When a source board publishes a new snapshot, the scores recompute automatically on the next cycle. You never wait more than 24 hours for updated rankings.

### Can I download the source data?

Yes. Every snapshot from every source board is archived. You can download the raw rankings and run the aggregation yourself. The code is public.

### What happens when a model leaves all boards?

The model stays in our database with a Low confidence tag and a score that reflects its last known position. After 30 days with no updates, it is removed from the main board but remains accessible through the archive.

### Why not include more boards?

We plan to. The pipeline is designed to accept any board that produces a ranked list of models. Each new board adds evidence and narrows the confidence intervals. We are actively integrating additional TTS leaderboards.

### How do you handle models that rename?

Model names change. Vendors rebrand, release new versions, or split model lines. We track these through manual curation with versioned notes in the snapshot archive. A rename triggers a Low confidence tag until the model establishes its position across boards.

## Links
- Read full article: https://texttospeech.com/blog/our-methodology
- All posts: https://texttospeech.com/blog
- Leaderboard: https://texttospeech.com/