Somebody posts the crew on Wednesday, adds a penalty average, and the replies fill up with people who have already decided how Sunday goes. The number is usually real. What is missing around it is everything that makes the number mean something.
Here is what a crew report has to show you before you should believe it.
1. How many games the number is built on
A crew works one game a week. The regular season is seventeen. That is a thin base for estimating a rate, and it gets thinner the moment you slice it by team or by foul type.
“This crew is bad for the Bears” is almost always built on the handful of games where that crew met that team. Three or four games. You would not accept a quarterback rating built on four throws.
A report that gives you a rate without the sample behind it is not giving you a rate. It is giving you an anecdote with a decimal point.
2. Which foul type, not just the total
Total flags per game is the least useful officiating number on the internet. It adds a first quarter false start to a fourth quarter defensive pass interference and calls them both “1.”
The foul type is what connects to your game. A crew that calls defensive pass interference more than the league average matters when two vertical passing offenses meet. It barely matters when two run heavy teams grind at each other for three hours.
Ask which flag, then ask whether this matchup produces the situations that draw it.
3. Whether the schedule is doing the work
Crews do not draw random games. A crew assigned to more games involving heavily penalized offenses will post a higher flag rate without officiating any differently than anyone else.
That is the confound that kills most crew analysis, and almost nobody adjusts for it. Separating crew tendency from schedule is the actual work. A raw leaderboard has not done it.
If a report cannot tell you whether it adjusted for opponent, treat its rankings as a description of what happened rather than a prediction of what will.
Correction, added after we tested this. We tested this over 2021 to 2025 and the schedule effect is close to zero. Removing both teams moved the crew spread from 0.677 to 0.674 flags per game. Crew assignment turns out to be near random with respect to how penalized the teams are, so this question matters much less than we implied. The sample-size question in point 1 is the one that actually bites. See the analysis.
4. Whether the crew is even the same crew
Crews are built around a referee before the season and work together week to week. Swing officials fill in when somebody is unavailable, and the label on the crew does not change when they do.
This matters more than it sounds. Offensive holding comes from the umpire. Pass interference comes from the officials working downfield. A crew carrying a substitute at umpire is not quite the crew the holding numbers describe.
Across seasons it matters more still. Personnel moves between crews. The referee’s name stays the same while the thing being measured does not. We wrote about which official actually throws which flag in how crews get assigned and what the name tells you.
5. What the league average is
A crew that calls holding at some rate is meaningless on its own. The only question is whether that rate is above or below what everybody else does, and by how much.
Every number we publish carries the league average next to it and the sample it was built from. Not because it looks rigorous. Because a rate without a baseline is a number you cannot argue with, and the whole point is to be able to argue.
The five-line version
| What to ask | Good answer | Walk away if |
|---|---|---|
| How many games? | A full season or more of that crew’s work | The sample is not stated at all |
| Which foul? | Broken out by penalty type | One combined flags-per-game number |
| Adjusted for opponent? | Says so, and says how | A raw leaderboard with no method |
| Same personnel? | Notes substitutions and season boundaries | Multi-season stats presented as one crew |
| Versus what? | League average printed beside every rate | A rate with nothing to compare it to |
The habit that matters most
Decide what would change your mind before you look anything up. Write down the number that would make you drop the theory.
If no number would, you are not measuring the crew. You are venting at it, which is fine, but it is a different activity and it should not come with a decimal point attached.
Whether any of this moves the result is a separate question, and a genuinely open one. We took it apart in what the evidence actually says about officiating and outcomes.