Before starting a pairwise ranking session, the practical question is: how long will this take? The answer depends on how many items you have and how smart the comparison order is. This article gives you both formulas, tables you can use for planning, and the practical rules of thumb that follow from them.
The Round-Robin Baseline: n(n−1)/2
If you want every item compared against every other item exactly once — a full round-robin — the number of matchups is:
Comparisons = n × (n − 1) / 2
where n is the number of items. Intuition: each of the n items meets every other item, which double-counts each pairing, so you divide by 2.
| Items (n) | Round-robin comparisons |
|---|---|
| 5 | 10 |
| 10 | 45 |
| 15 | 105 |
| 20 | 190 |
| 30 | 435 |
The growth is quadratic — double the items, roughly quadruple the comparisons. That's why a 30-item round-robin (435 matchups) is a bad afternoon for anyone, while 5 items (10 matchups) is a two-minute warm-up.
Do you ever need the full round-robin? Yes, in specific cases:
- You need win-rate statistics per item, not just an order. With every matchup recorded you can say "feature A won 80% of its comparisons" and measure how close the middle of the pack really is.
- Voters are comparing remotely (survey-style), each voter sees only some pairs, and you want every pair covered by someone.
- The list is short (under ~8 items), so the full coverage costs almost nothing.
The Merge-Sort Shortcut: ~n · log₂(n)
For producing a ranking, you don't need every pair. A well-designed comparison tool borrows the strategy of merge sort: to combine two already-ranked groups, it only compares across the boundary until one group runs out, and each answer eliminates several possible matchups at once. The number of comparisons needed scales as approximately:
Comparisons ≈ n × log₂(n)
| Items (n) | Round-robin n(n−1)/2 | Merge-sort ≈ n·log₂(n) | Reduction |
|---|---|---|---|
| 5 | 10 | ~12 (≈10) | — (too small to benefit) |
| 10 | 45 | ~25 | ~44% fewer |
| 15 | 105 | ~45 | ~57% fewer |
| 20 | 190 | ~70 | ~63% fewer |
| 30 | 435 | ~115 | ~74% fewer |
(The small n case is the exception: below about 8 items the overhead of the approach isn't worth it, and the two strategies converge.)
Two honest caveats about the shortcut:
- The number varies with your answers. Merge sort's comparison count depends on how often your answers let it skip ahead. The table values are typical, not exact — a session of 20 items usually lands in the 60–75 range, not always exactly 70.
- You get an order, not per-item statistics. Wins recorded along the way are partial, so "A beat 7 of 9 opponents it met" is a sample, not a full record.
For most prioritization decisions that trade is a clear win: you save 60–75% of the effort and the ranking is just as valid.
Worked Example: Ranking a 12-Item Backlog
Say your team has 12 feature requests and you want a priority order before planning week. Here's the arithmetic:
- Round-robin: 12 × 11 ÷ 2 = 66 comparisons. As a solo ranking at ~8 seconds each, that's about 9 minutes — tolerable. As a team session with 60–90 seconds of debate per matchup, it balloons to 1.5–2 hours, which is why teams don't do it.
- Merge-sort approach: roughly 12 × log₂(12) ≈ 43 comparisons, and in practice often fewer as answers eliminate several matchups at once. Team session cost: about an hour — a realistic single agenda slot.
- Trim first? At 12 items, trimming isn't urgent. But if two of those 12 are clearly blocked, removing them saves another 15–20 comparisons and sharpens the debate. Always check for free wins before starting.
The general pattern: solo ranking copes with bigger lists because per-comparison cost is tiny; group sessions are where the count formula really matters, because each matchup carries the cost of everyone's attention.
What About Inconsistent Answers?
Real humans are not perfectly consistent — on a bad day you might pick A over B in one matchup and effectively rank B above A through another chain of comparisons. This happens in both approaches, but it surfaces differently:
- In a round-robin, contradictions show up as win-count ties. That's not a flaw — two items with equal wins are genuinely hard to separate, and acknowledging that beats manufacturing a false order.
- In a merge-sort pass, an inconsistency simply nudges the final position of one item; the overall ranking stays stable because most positions were decided by clear chains of wins.
The practical takeaway: don't agonize over individual matchups. The strength of the method comes from the aggregate of many small judgments, which averages out single slips — the same reason it beats trying to rank everything in one judgment.
Practical Guidance on List Size
Knowing the formulas, here's how to size a session in practice:
The sweet spot is 8–15 items. At 10 items you're looking at roughly 25 quick comparisons in quick mode — a comfortable 10–15 minutes including discussion. Fatigue stays low, and the top of the ranking is reliable.
Above ~20 items, trim before you compare. Comparisons get slower and sloppier as attention drains. Before ranking 40 backlog entries:
- Delete anything with an obvious blocker (infeasible, out of strategy, done elsewhere).
- Merge near-duplicates — five variants of "improve onboarding" become one item.
- If you still have 25+, rank them in two passes: a coarse pass to find the obvious bottom half, then a careful pass on the survivors.
Below 5 items, pairwise still works but is overkill. With 3–4 items you can usually just discuss and decide. The method earns its keep from about 6 items up, where mental bookkeeping starts to fail.
Budget time per comparison, not just comparisons. Solo ranking: 5–10 seconds per matchup. Team session with debate: 60–90 seconds per matchup. A 12-item team session in quick mode is roughly 35 matchups × 75 seconds ≈ 45 minutes — a solid agenda slot, and worth planning for.
If you're choosing between this method and others (a weighted matrix, RICE, and so on), the count tables above are only half the story — see our comparison of prioritization methods for when each approach fits.
Curious how the counts feel in practice? PairwisePro's free pairwise comparison tool uses the optimized approach automatically — paste a list and the tool tells you how many matchups to expect.