Defensive holding is five yards and an automatic first down. That is what the rulebook charges. It is not what it cost your team.
Yardage measures field, not damage. The same flag is worth almost nothing in the first quarter and can end a game in the fourth. To know what a call actually cost, you need a number that knows the situation the flag landed in. That number is win probability added.
Yardage tells you the penalty, not the damage
Take a ten yard holding call on the offense. First and ten at your own 25, six minutes in, tie game. Now it is first and twenty. Annoying, and that is all it is.
Same flag, second and two at the opponent 15, down four, two minutes left. Now it is second and twelve on a drive that has to end in a touchdown.
Identical penalty. Identical yardage. Any measure that treats those two the same is not measuring impact.
Down conversion widens the gap further. Defensive holding and defensive pass interference carry automatic first downs, and a fresh set of downs is often worth more than the yards stapled to it. Sports Info Solutions studied three seasons of penalties and found defensive fouls on third down worth roughly twice the expected point value of the same fouls on first down. They put defensive pass interference near one and a half expected points, the most damaging defensive foul in the book.
What a win probability model knows, and what it does not
A win probability model answers one narrow question. Given this exact game state, what share of historically comparable situations ended in a win for the team with the ball?
It is empirical, not a simulation. The open source model in nflfastR is a gradient boosted tree trained on two decades of play by play data, and it reads a short list of inputs:
- Seconds remaining in the half and in the game
- Score differential, adjusted so that a lead matters more as time drains
- Field position
- Down and yards to go
- Timeouts remaining for each side
- Home field, and which team receives the second half kickoff
Look at what is missing. The baseline model does not know which quarterback is playing, that your left tackle just limped off, or that it is raining sideways. Some variants add a pregame estimate of team strength, which is one reason two models can watch the same snap and disagree.
The check that matters is calibration. Of all the moments a model called 30 percent, did about 30 percent end in wins? The current nflfastR model is tight on that test, with reported calibration error well under one percentage point. Calibration is what makes these numbers usable at all. It is not a promise about any single play.
Win probability added is just a subtraction
WPA is win probability after a play minus win probability before it. That is the whole definition.
For a penalty, the before state is the field position, down, distance, clock and score at the snap. The after state is the same game with the flag enforced. Subtract. What is left is how many percentage points of win probability the call moved.
Three properties come with that, and they are worth holding onto:
- It is zero sum. Whatever one team gains, the other loses. No play helps both.
- It is measured in percentage points of win probability, not in yards and not in points. A late game pass interference call moves one team’s chances by whatever the model says it moved them, no matter how far the ball travelled.
- It is descriptive. WPA tells you what happened to the estimate. It forecasts nothing, and it rates nobody’s skill.
Enforcement sets the number, not the name of the penalty. A declined flag has a WPA of zero by construction, because the game state never changes. A spot foul on a deep ball can move the ball further than any yardage table suggests. The number follows what the crew actually enforced.
The same flag, three different games
Take one defensive holding call, five yards and an automatic first down, and drop it into three game states.
First quarter, tied, third and 7 at your own 30. The offense keeps the ball on a drive that had a long way to go anyway, and the model barely notices. A rounding error.
Third quarter, trailing by 17, same down and distance. Close to nothing again, for the opposite reason. At that deficit, almost nothing either team does moves the estimate much.
Fourth quarter, trailing by 4, third and 7 at your own 30, 1:48 on the clock and one timeout left. The drive was one stop from over and now it is alive. That one is the game.
Same five yards every time.
This is the most useful thing WPA does. It separates the flags that felt bad from the flags that mattered, and those are not the same set.
Leverage is doing most of the work
Leverage is how much any single outcome can swing the estimate from a given state. It is high when the game is close and late, on third and fourth down, and in the red zone. It is near zero in a tied first quarter and in a four score blowout.
So a game full of flags can finish with net WPA near zero because every one of them landed in a dead moment, while a game with three flags turns on one of them. Most penalties cluster near the bottom of the impact scale. A thin top slice carries almost all of the damage.
That imbalance is why raw penalty counts are a bad way to argue about officiating. If you want that argument, have it with the right numbers: see which teams get flagged most, and whether that is bias or style.
The game where the flag was never thrown
This is where most penalty analysis quietly cheats, so here it is with the hedging stripped out.
A flag with a WPA of minus 12 percentage points does not mean your team would otherwise have won. It means the model’s estimate of your chances was 12 points higher one moment earlier. Those are two different claims, and only one of them is supported.
Nobody gets to watch the game where the flag stayed in the pocket. In that game the clock runs differently, the next play call is different, and every snap after it is different. WPA accounts for one state transition. It does not simulate the other universe. Stacking up a team’s WPA losses and calling the total “the refs cost us the game” is a claim the math does not make.
One more limit belongs here. WPA says nothing about whether the call was correct. A perfectly officiated holding call can be the most expensive play of your season, and a blown call in a second quarter blowout is free. Correctness is a rules question, which is why the crowd verdict sits next to the cost instead of inside it.
Two models, one play, two answers
Win probability models are built differently, and they agree less than the clean decimal on your screen implies. Reviewing Super Bowl LI, Michael Lopez compared public models through the game and found gaps between the highest and lowest estimates reaching roughly ten percentage points. Across the plays he checked, fewer than half had all the compared models agreeing even on the direction of a play’s WPA.
The answer is not to throw out the metric. It is to stop performing a precision that is not there:
- Name the model. Every cost figure we publish says which win probability model produced it, and which version of it.
- Round. “About four points of win probability” is a defensible sentence. “4.27 percent” is theater.
- Treat small gaps as noise. A flag worth a fraction of a percentage point and a flag worth twice that are both small.
- Use WPA to rank and compare. Do not use it to adjudicate.
How to read the number without overclaiming
Three questions get you to an honest verdict on any call.
- How big was the swing? That is the WPA number, rounded.
- How leveraged was the moment? Late, close and third down means a big number was on the table. First quarter means it was not.
- Was the call defensible under the rule as written? Separate question, separate evidence.
Answer all three and you have something better than a grievance. You have the number. For the live version of that skill, see how to read a call while it is happening. Before drawing a conclusion out of one Sunday, see whether officiating measurably changes outcomes.
For where this fits alongside everything else worth having open on a Sunday, see what to have on your second screen during a game.
In the ChallengeFlag app, every flagged play carries its win probability cost next to the community rating from fans who scored the call one to five. The cost is math. The verdict is opinion. They stay in separate columns.