Pairwise comparison is a method for ranking a set of options by comparing them two at a time and asking which of the two is preferred. Each judgment is simple, and once enough pairs have been judged, the results add up to a complete ranking from first to last.
The method also goes by the names paired comparison, pairwise ranking and the pairwise comparison method. Psychologists have used it since the 1920s to measure preferences that people can't put a number on directly, and today it shows up in product backlogs, hiring panels, survey research and everyday personal decisions. This page explains how it works, walks through a full example with a comparison matrix, covers how many comparisons you need, and compares the main variants. If you'd rather try it than read about it, the free pairwise comparison tool ranks a list in your browser with no signup.
Why compare in pairs?
Ranking ten things at once is hard. You have to hold all ten in your head, keep shifting them around, and the items in the middle of the list tend to get less attention than the obvious first and last. Rating each item on a 1–10 scale has a different problem: most things end up as a 7 or 8, and the scores don't force you to choose.
A pairwise comparison asks a much smaller question: of these two, which matters more? People answer that question quickly and fairly consistently, even when they couldn't explain how to score either option. Repeat it across enough pairs and the overall order follows from your own choices. Nobody has to argue for a full ranking up front.
How pairwise comparison works, step by step
- Write the list. Put down every option you want to rank, one per line. Keep items at the same level of detail: "Improve onboarding" and "Change the button color on step 3" don't compare well.
- Fix the question. Decide on one question you will ask for every pair, such as "Which delivers more value to customers?" or "Which would I rather do this summer?" Changing the question halfway through mixes up two different rankings.
- Compare pairs. Present two items and pick one. In a full round-robin every item meets every other item once. In an adaptive approach, a sorting algorithm picks the next pair based on your earlier answers so that fewer comparisons are needed.
- Record the results. Each time an item is chosen, it scores a win. If the two are genuinely equal, record a tie (often counted as half a win).
- Turn wins into a ranking. Sort the items by their win count (or, with adaptive sorting, read off the sorted order). The item with the most wins is first.
- Sanity-check the result. Look at the final order. If something looks wrong, revisit the specific pairs that put it there instead of shuffling the list by hand.
The pairwise comparison tool handles steps 3 to 5 for you: paste your list, answer the matchups, and the ranking appears with each item's wins and win rate.
A worked example: ranking five product features
A small software team has budget for one big feature next quarter and wants a priority order for five candidates:
- Search: full-text search across all projects
- Dark mode: a dark color theme
- CSV export: download any table as a spreadsheet
- Mobile app: a native app for phones
- SSO: single sign-on for company accounts
The question for every pair is: "Which will matter more to our customers next quarter?" With five items, a full round-robin takes 5 × 4 ÷ 2 = 10 comparisons. Here are the team's answers:
| # | Pair | Winner |
|---|---|---|
| 1 | Search vs Dark mode | Search |
| 2 | Search vs CSV export | Search |
| 3 | Search vs Mobile app | Mobile app |
| 4 | Search vs SSO | Search |
| 5 | Dark mode vs CSV export | CSV export |
| 6 | Dark mode vs Mobile app | Mobile app |
| 7 | Dark mode vs SSO | SSO |
| 8 | CSV export vs Mobile app | Mobile app |
| 9 | CSV export vs SSO | SSO |
| 10 | Mobile app vs SSO | Mobile app |
The pairwise comparison matrix
The same answers fit into a pairwise comparison matrix. Each row is an item, each column is an opponent. A 1 means the row item beat the column item, a 0 means it lost, and the diagonal is empty because an item never plays itself. Every 1 in the matrix has a matching 0 on the mirrored side of the diagonal.
| Row beats column → | Search | Dark mode | CSV export | Mobile app | SSO | Wins |
|---|---|---|---|---|---|---|
| Search | — | 1 | 1 | 0 | 1 | 3 |
| Dark mode | 0 | — | 0 | 0 | 0 | 0 |
| CSV export | 0 | 1 | — | 0 | 0 | 1 |
| Mobile app | 1 | 1 | 1 | — | 1 | 4 |
| SSO | 0 | 1 | 1 | 0 | — | 2 |
The row totals are the win tally. As a check, the wins add up to 4 + 3 + 2 + 1 + 0 = 10, one win per comparison.
From win tally to ranking
Each item played 4 matchups, so the win rate is wins ÷ 4:
| Rank | Feature | Wins | Win rate |
|---|---|---|---|
| 1 | Mobile app | 4 | 100% |
| 2 | Search | 3 | 75% |
| 3 | SSO | 2 | 50% |
| 4 | CSV export | 1 | 25% |
| 5 | Dark mode | 0 | 0% |
The team now has a clear order, and every position can be explained. Search ranks above SSO because the team picked it head to head, and the mobile app is first because it won every matchup it played. When someone asks "why isn't dark mode higher?", the answer is in the matrix: it lost all four of its comparisons.
This example came out perfectly consistent, with one item at each possible win count. Real sessions are messier, and the pitfalls section below covers what to do when they are.
How many comparisons does pairwise comparison need?
Comparing every item with every other item takes n × (n − 1) ÷ 2 comparisons, where n is the number of items. The count grows quickly:
| Items (n) | Full round-robin n(n−1)/2 | Adaptive merge sort ≈ n·log₂(n) |
|---|---|---|
| 5 | 10 | ~10 |
| 10 | 45 | ~25 |
| 15 | 105 | ~45 |
| 20 | 190 | ~70 |
| 30 | 435 | ~115 |
The shortcut in the right-hand column comes from merge sort. When two already-sorted groups are merged, the algorithm only compares items at the front of each group, and every answer settles several positions at once because of transitivity: if A beats B and B is already known to beat C, there's no need to ask about A and C. The exact count depends on your answers, so treat the column as a typical figure rather than a fixed number.
The practical upshot: a full round-robin is fine up to about 8 items, but for longer lists an adaptive approach saves most of the work. For more tables, worked arithmetic and advice on list size, see how many pairwise comparisons you need.
Pairwise comparison methods and variants
"Pairwise comparison" covers a family of methods. They share the same basic judgment (A or B?) and differ in which pairs get asked and how the answers are turned into a ranking or a score.
Round-robin win tally
Every pair is compared once and items are sorted by wins, as in the example above. It's simple, transparent and easy to audit, and it gives you a full win record for every item. The cost is the quadratic number of comparisons, which makes it practical mainly for short lists or for surveys where many people each judge a few pairs.
Adaptive sorting (merge sort and similar)
A sorting algorithm chooses the next pair based on what it already knows, so you only answer the comparisons that are needed to establish the order. This is how the PairwisePro tool works: it uses a merge-sort strategy that needs roughly n × log₂(n) comparisons, supports ties and undo, and shows each item's wins and win rate at the end. The trade-off is that you get a reliable order but not a full head-to-head record for every pair. The head-to-head ranking guide covers this style of "this or that" ranking in more detail.
Elo ratings and the Bradley–Terry model
When comparisons are incomplete, noisy or come from many different people, statistical models estimate a score for each item from the pattern of wins and losses.
- The Bradley–Terry model assigns each item a strength p and assumes the probability that item i beats item j is pᵢ ÷ (pᵢ + pⱼ). The strengths are fitted to all observed comparisons at once, so the model copes with missing pairs and with disagreements between voters.
- Elo ratings, best known from chess, update two items' scores after every single comparison: the winner gains points and the loser loses them, with a bigger swing when the result is a surprise. Elo is convenient for continuous, crowd-sourced voting where comparisons never really stop.
Both produce a numeric score as well as an order, which is useful when you want to know how far apart items are. They also need many more judgments than a single person would want to make, so they suit large-scale voting rather than personal prioritization.
Analytic Hierarchy Process (AHP)
The Analytic Hierarchy Process, developed by Thomas Saaty, asks how much more important one item is than another, using a 1–9 scale:
- 1: equally important
- 3: moderately more important
- 5: strongly more important
- 7: very strongly more important
- 9: extremely more important
- 2, 4, 6, 8: intermediate values
If A is rated 3 against B, then B is automatically 1/3 against A, so the matrix is reciprocal. The priority weights come from the matrix's principal eigenvector (a common hand approximation is to normalize each column and average across each row). AHP also computes a consistency ratio; a value above about 0.10 is the usual signal that the judgments contradict each other and should be revisited.
AHP is often used to derive criteria weights for a larger decision. It's more expressive than a simple win/loss judgment, but it asks more of the person comparing, and it's usually applied to fewer than ten items at each level of the hierarchy.
Pairwise comparison vs ranking vs rating
There are three common ways to put a list in order. They suit different situations.
| Approach | What you do | Strengths | Weaknesses |
|---|---|---|---|
| Direct ranking | Drag the whole list into order | Fast for short lists | Gets unreliable past 6–7 items; middle positions are guesswork |
| Rating / scoring | Give each item a score (1–5, 1–10) | Each item judged on its own; quick to collect | Scores bunch together; no forced trade-offs; people use scales differently |
| Pairwise comparison | Pick the better of two, repeatedly | Easy judgments; forces trade-offs; full order with an audit trail | More steps than a single ranking; needs one clear question |
A related choice is between pairwise comparison and a weighted decision matrix. A matrix scores each option against several explicit criteria (cost, effort, impact) and multiplies by weights; it works best for a handful of options with measurable criteria. Pairwise comparison works best for longer lists judged on overall value or gut preference. The decision matrix vs pairwise comparison article goes through when to use which, and if a matrix fits your problem you can build one with the weighted decision matrix calculator.
Where pairwise comparison is used
- Product features and backlogs. Teams rank feature requests, bugs or roadmap themes against a question like "which moves our main metric more?" The result is an order the team can defend in planning.
- Hiring and evaluation. Comparing candidates two at a time against the same role criteria reduces the drift that comes from interviewing people days apart. The same idea applies to grant applications, design submissions and award shortlists.
- Personal decisions. Where to travel, which job offer to take, which side project to start, what to read next. Pairwise comparison is a good way to find out what you actually prefer when the options feel equally attractive.
- Research and surveys. Paired comparison is a long-standing technique in psychology, marketing research and sensory testing, because respondents find "which of these two?" easier than rating scales. Survey tools often show each respondent a subset of pairs and combine the answers with a model such as Bradley–Terry.
- Group alignment. When a team disagrees about priorities, ranking the list pairwise surfaces exactly which trade-offs people disagree on.
For a broader introduction aimed at prioritization, read what pairwise prioritization is. If you're choosing software, the pairwise ranking tools compared roundup looks at the options side by side.
Common pitfalls and how to handle them
Intransitive loops
Sometimes the answers form a cycle: A beats B, B beats C, and C beats A. In a round-robin tally the three items end up with the same number of wins against each other. This usually means the items are close, or that you switched between two different criteria without noticing (for example, judging A vs B on cost and B vs C on impact). Restate the question, look at the three pairs together, and decide again. If the loop survives a second look, the items really are about equal and their order matters less than you think.
Ties
Allowing ties keeps you from inventing a preference that isn't there. In a tally, a tie usually counts as half a win for each item. Use them sparingly, though: if many answers are ties, the question is probably too vague. Items that end with equal win counts can be separated by their direct head-to-head result.
Comparison fatigue
Quality drops when people answer too many matchups in one sitting. Answers get faster and more random. To keep sessions manageable:
- Trim the list first: remove items that are obviously out, and merge near-duplicates.
- Use an adaptive method instead of a full round-robin for anything over about 10 items.
- Take a break in long sessions. The PairwisePro tool saves an unfinished session in your browser so you can resume where you left off.
A vague or shifting question
"Which is better?" invites everyone to use their own definition. Write the question down before you start and keep it visible during the session.
Items at different levels
Comparing "Rebuild the billing system" with "Fix a typo on the pricing page" produces lopsided, uninformative matchups. Group items at a similar size, or rank big themes first and their details separately.
Frequently asked questions
What is the pairwise comparison method in simple terms?
You take a list of options, look at them two at a time, and pick the one you prefer in each pair. Count how often each option wins (or let a sorting algorithm order them for you) and you have a ranking. It replaces one hard decision with many easy ones.
What is a pairwise comparison matrix?
It's a square table with the same items along the rows and the columns. Each cell records the result of comparing the row item with the column item: 1 for a win and 0 for a loss in a simple tally, or a 1–9 intensity value and its reciprocal in AHP. Adding up each row gives the item's total score.
How many comparisons do I need for 10 items?
A full round-robin needs 10 × 9 ÷ 2 = 45 comparisons. An adaptive merge-sort approach typically needs around 25, because each answer rules out several other pairs. See the comparison count guide for other list sizes.
Is paired comparison the same as pairwise comparison?
Yes. "Paired comparison", "pairwise comparison" and "pairwise ranking" describe the same basic technique of judging items two at a time. Some fields prefer one term (paired comparison is common in psychology and sensory testing; pairwise comparison is more common in decision-making and product management), but the method is the same.
Is there a free pairwise comparison calculator?
Yes. The PairwisePro tool is free, runs in your browser and needs no account. Enter up to 100 items, answer the matchups with your mouse or keyboard (ties and undo are supported), and get a ranked list with wins and win rates. You can share the result as a link or export it as text, CSV or JSON.
Rank your own list
The quickest way to understand pairwise comparison is to run one. Paste a list of five to fifteen things you've been meaning to prioritize, such as features, ideas, tasks or places to go, and answer one question for each pair. A few minutes later you'll have an order you can explain.
Start a free pairwise comparison: no signup, nothing to install.