Two weeks ago we published a piece arguing that crew penalty leaderboards are confounded by schedule. Crews do not draw random games, we said, so a crew working more games against heavily penalized offenses will post a higher flag rate without officiating any differently. Pulling those apart, we wrote, is the actual work.
We did the work. That claim was wrong.
What we did
Five regular seasons, 2021 through 2025. 1,356 games, 17 referees per season, 14 to 17 games per crew per season, 16,278 flags. League average 12.0 flags per game, which matches published norms and is the first thing to check when the numbers come out of a pipeline you wrote yourself.
The data is public. Play by play and officials assignments both come from nflverse, which is MIT licensed. Nothing here touches a commercial feed, so anyone can rerun it and tell us we are wrong.
The question the leaderboards skip is not whether crews differ. It is whether they differ more than luck would produce anyway. A crew works about 17 games a season, and at that sample size random variation alone manufactures a gap between the busiest and quietest crew. So the number that matters is not the spread. It is the spread compared against what identical crews would produce.
We built that comparison by permutation. Shuffle which crew worked which game, holding each crew’s game count fixed, recompute the spread, repeat 5,000 times. That gives the distribution luck produces. Then see where the real number falls inside it.
We split the fouls into two groups. Low discretion ones are close to binary: false start, offside, neutral zone, delay of game. Either the tackle moved or he did not. High discretion ones need a judgment about whether contact was significant or a block materially restricted: offensive and defensive holding, pass interference, illegal contact, unnecessary roughness, roughing the passer. If crew identity matters anywhere, it should matter more in the second group.
What came back
| Penalty group | Observed spread | Spread from luck alone | p | Verdict |
|---|---|---|---|---|
| All penalties | 0.68 | 0.46 | 0.007 | Real |
| High discretion | 0.56 | 0.30 | under 0.001 | Real, and the widest gap |
| Low discretion | 0.25 | 0.25 | 0.55 | Indistinguishable from luck |
Spread is the standard deviation of per-crew mean flags per game, across the 17 crews with at least 40 games in the window.
The low discretion row is the one worth sitting with. On false starts and offside, seventeen crews are statistically indistinguishable from seventeen coin flips. The spread that exists is exactly the spread luck produces. Whatever else is true about officiating, nobody is calling more false starts than anybody else.
On judgment calls the gap is wide and it is not close. The observed spread is nearly double the noise floor.
The claim that did not survive
We expected schedule to explain a meaningful share of the crew gap. So we removed it: least squares on indicators for both teams in each game, then recomputed crew spread on what was left.
| Penalty group | Crew spread, raw | After removing both teams |
|---|---|---|
| All penalties | 0.677 | 0.674 |
| High discretion | 0.555 | 0.555 |
| Low discretion | 0.248 | 0.235 |
Three thousandths of a flag per game. On the group where the crew effect is strongest, the adjustment changes nothing at all to three decimal places.
Schedule is not the confound. In this window crew assignment is close to random with respect to how penalized the teams are, which is what you would expect if the league assigns crews without regard to matchup, and is a quiet point in the league’s favor. Our original framing implied the leaderboards were contaminated by something that is not contaminating them.
The rest of that article stands. Sample size is still thin, the discretionary split still matters, and a raw leaderboard is still a bad way to argue. But the specific claim that schedule does meaningful work is not supported, and we have corrected the article rather than quietly leaving it up.
Does it persist?
A real habit should turn up again next season with mostly the same people in the same positions. Noise should not. Correlating each crew’s rate in one season against the next, across 65 crew-season pairs:
| Penalty group | Year over year correlation |
|---|---|
| High discretion | +0.37 |
| All penalties | +0.21 |
| Low discretion | +0.04 |
The same pattern from a completely different direction. Judgment calls carry a signal that survives into the following season. Mechanical fouls carry nothing. A correlation of +0.04 is a coin flip with extra steps.
Note what +0.37 does and does not mean. It is a real, repeatable tendency. It also leaves most of the variance unexplained, so it will not tell you what happens in your game on Sunday.
What we would want a critic to hit
- Seventeen crews is a small n. The permutation test handles the games-per-crew problem. It does not fix the number of crews.
- The referee is a proxy for the crew. Personnel move between seasons and swing officials substitute within them, so the year over year figure measures a unit that partly changes. That should push the correlation down, meaning true persistence is probably higher than +0.37, not lower.
- We counted flags, not enforcement. The count includes declined penalties, about 1.6% of flagged plays. Weighting by leverage instead of counting would measure impact better and is a different piece of work.
- Per game, not per opportunity. Offensive holding should arguably be normalized to pass-blocking snaps. That the team adjustment changed nothing suggests this matters less than we assumed, but it is not the same test.
- A crew tendency is not a bias. Nothing here says any crew favors anyone. It says some crews call judgment fouls more often than others, in both directions, against everybody.
Reproduce it
Two public files. Play by play at github.com/nflverse/nflverse-data/releases/download/pbp/play_by_play_YYYY.parquet for 2021 through 2025, and officials at github.com/nflverse/nflverse-data/releases/download/officials/officials.parquet.
Join old_game_id in the play by play against game_id in the officials file. Both are strings and they match exactly; we checked 285 of 285 games in 2024. Filter officials to position == "Referee" to identify the crew, count rows where penalty == 1 per game, then permute crew labels while holding each crew’s game count fixed.
If you rerun this and get something different, we want to know. The point of publishing the method is that the claim can be killed by someone who is not us.
What it means on Sunday
Check the crew, but check the right number. Total flags per game is the weakest version of this. What carries signal is the rate on judgment fouls, and only that.
And if your complaint this week is about false starts, the data says leave the crew out of it. That one is your left tackle.
For what a crew report has to show you before you believe any of it, see the five questions worth asking. For the piece this article grades, see bias or style.