Q&A With Brian Burke Of Advanced NFL Stats: An Outsider's View Of Arian Foster And Shaun Cody

If statistics were a person, I'd feel really bad for them.  They'd be the weird kid in the corner who knows more than anyone else, but is constantly misunderstood.

All the other numbers would make fun of statistics and steal their lunch money.  They'd grow up and constantly be misquoted and getting in trouble, but they're too quiet to be able to stand up for themselves.  Everybody thinks they're a bunch of liars. 

Their position is not helped by the fact that they are completely and totally unemotional, and all the girls think they're weird. 

The problem with statistics, though, isn't that they're liars.  All they do is calculate numbers, but those numbers are often misinterpreted.  The biggest misinterpretation is that any single statistic can tell a complete story.  It often (read:  always) takes multiple stats to truly understand what's going on.

This is especially true when it comes to football.  No single stat can give a true understanding of who's better, who had the biggest contribution, or how to predict future performance. As such, stats continue to be branded with the scarlet number.

But in the past few years, stats have finally started getting their due.  Everybody knows about Football Outsiders and the fine work being done there (shout out to Rivers!) but there are others, and one that I'm a big fan of is Advanced NFL Stats. 

The founder of Advanced NFL Stats is Brian Burke, and he was kind enough to answer a few questions I had regarding his results.  Before you hit the jump, though, let's all thank Brian by jumping over to his site and clicking on the ads repeatedly.

The first is Win Probability Added (WPA).  Burke calculates the probability of victory during every play in the game using a model that includes a myriad of factors such as down and distance, field position, current score, time in game, etc.  For each play, if the play increases the win probability, he assigns the players involved value equal to the amount that the probability increased.  Similarly, if the probability decreases, he decrements each players score by the appropriate amount.  A far better explanation is included here. 

Expected Points Added is a similar metric in that for each game scenario (including field position, down and distance, etc), a team can expect a certain amount of points.  As each play is run, the expected points will either increase or decrease, depending on the result of the play, and he assigns credit accordingly.  Again, a better explanation is here.

So, with that said, here's a brief Q&A between myself and Brian.

Me: 

Brian,

1. In last week's victory over the Steelers, Arian Foster was huge.  He ran for over 150 yards, including a huge 42 yard touchdown run that put the Texans ahead for good.  Other highlights included a big 18 yard run on 2nd and 11 from the Texans' own 5, where he put an incredible move on Polamalu.  Yet your stats had him with a 0.09 WPA and a -2.6 EPA.  I presume this is because the team was unable to run the ball in the fourth quarter and he ended up with a low success rate, but can you help me understand how his overall performance resulted in a low WPA and a negative EPA?
   
   
2. Game observations tell me that Shaun Cody is terrible.  He's constantly pushed out of the way and struggles to hold the point (even though he's supposed to be shooting the gap in Wade Phillips' system).  The Texans are consistently run on up the middle and, according to Football Outsiders, opposing teams have run 82% of those plays up the middle.  Yet you have him ranked 17th in the DT rankings -- which is approximately 873 places higher than I would have expected -- with a 0.25 WPA and a 7.3 EPA.  Can you also help me understand this?

Thanks,

Robert

Brian:

Robert,

+0.09 WPA is actually pretty good for a single game for a RB. His EPA is low for a few reasons. The string of runs at the end of the game (since his TD run) were for: -2, 3, 2, 3, 1, 2, 2, and 5 yds. It's not as costly in terms of WPA because those runs burned clock, but it's costly in terms of moving the chains--the things that SR and EPA measure.

17th for DTs is not that great. There are quite a few 3-4 DTs that get labeled DEs because of the scheme. The vast majority of Cody's impact has come on one single play so far this season--the fumble recovery vs IND. Notice that he's nearly at the bottom of the list in terms of Success Count with only 4. And he's at the very bottom of the list of qualifying DTs for Tackle Factor with only 0.25. Both of those numbers are very poor for a starter. No wonder teams are running at him.

 

Brian

Me:

Thanks for the response.  I'd like to ask some follow-up questions though. 

First, on Arian:  My first non-quantitative reaction to the game was that Foster was the biggest reason that the Texans won (offensively at least), so I would have originally expected him to have the highest score on the team.  I would imagine that Matt Schaub's higher score of 0.18 is because even though Schaub didn't have excellent traditional stats, he had a much better success rate (this would also explain Owen Daniels' score -- the plays he made were tremendously valuable).  This leads me to a two part question:  1) how do you factor in the notion that players are reliant on the performances of others (Arian on his blockers in this case, but I also noticed that in New Orleans Andre Johnson has a negative WPA and EPA which I attributed to bad passes from Schaub (there was one in the red zone, the interception, and the final 4th down prayer (and no, there is no limit to the amount of parentheticals I will put inside other parentheticals))), and 2) what's the appropriate way to use these stats?  Is it correct to say that Schaub was twice as productive as Foster?  Can we say that he had a bigger impact than Foster?  Are we really overvaluing Foster's value because of the highlight-iness of his plays? 

On Cody:  Looking at Cody's Success Count of 4 means that of all the plays he's been involved in, only 4 have been successful.  Does that take into considerations the plays where his "involvement" included getting pushed five yards off the line of scrimmage by the guard/center?  Also, his Tackle Factor of 0.25 means that he makes 75% fewer tackles than what's expected of a nose tackle.  Does that compare him to all DTs or only to Nose Tackles, and does it account for those in 3-4 schemes vs 4-3 schemes?  Also, in regards to Cody's fumble recovery, he didn't really have a tremendous impact on that play, but was kind of lucky that the ball bounced to him.  In general, fumble recoveries tend to have a lot of luck involved.  How much does that factor into WPA and EPA calculations and is there a reason you've decided to include them in your data set as opposed to just discarding them as "lucky plays"?

Brian:

I bet you're not alone. RBs just have way too many short gains, which are a big drag on their WPA and EPA. It's not entirely their fault. Coaches just call too many running plays given today's game. For that reason, I prefer to compare RBs to other RBs rather than to QBs in these advanced metrics.

Regarding Schaub--he had a solid game. His 3 biggest plays were 2 completions early in the 4th, both for conversions. Both were to Owen Daniels, one for 9 and another for 30 yds, worth 0.04 and 0.08 WPA respectively. He had another big play early in the game on 3rd and 10--a completion to Daniels for 18 yds worth 0.08 WPA. No interceptions, no fumbles, and no sacks. Really solid.

It's definitely correct to say Schaub had a bigger impact than Foster, but that's almost always true when comparing QBs and RBs. That said, neither Schaubs nor Fosters plays were solo acts. There were 10 other teammates on the field.

WPA and EPA is not split among players. If a play is a +0.05 WPA play, both passer and receiver get credited in my implementation. I realize this isn't the only way to look at things, but it captures the full story of the game. It follows what's called a counter-factual epistemology. (Without successful performance by both Player x and Player y, this play would not have happened. Alternatively, if either player failed in his job, the play doesn't happen.)

Foster's role is overvalued not because of flashy play, I believe, but because of his fantasy value. He plays on a team with a great running scheme, solid o line, and a great passing game that puts him near the goal line frequently and puts defenses on their heels.

Regarding Cody and defensive stats--The SC counts plays in which the player was named in the official play-by-play that resulted in a setback for the offense. Setback is defined as causing negative EPA. If a player is named in the play-by-play for making a tackle that results in a positive outcome for the offense, I don't count that against him. What was he supposed to do, let the ball carrier run by? That's why +WPA and +EPA can only be positive for defenders. A SC of 4 really bad for a DT who has played 3 games.

Your definition of TF is almost right on. A TF of 1.0 represent the number of tackles (and assists) you'd expect by a player at a particular position, assuming he played all the snaps of a game. That rarely happens, which is why TF averages well below 1.0. It's for all DTs, and does not distinguish between nose, 3-technique, or other variants of DT, nor does it distinguish between 3-4 and 4-3. I can add that functionality, and although the differences are relatively small, it's on the list of things I'll do when I have more time.

Unfortunately, "S.Cody blown off the line and allows a 7-yard gain by R.Mendenhall," is never included in the play-by-play. So the stats won't capture that kind of thing, except for the fact that he missed an opportunity for a successful play, and his stats will indirectly reflect that.

Sure, most fumble recoveries are totally random, but they did happen. The player was in the right place at the right time, was alert and aware, and was quick enough and strong enough to get and hold possession. Stats like +WPA, and +EPA should be thought of as 'playmaking' stats. They capture, at most, half the picture. But that's 50% more than we had before. Defensive stats, like simple tackles or total sacks, can paint very misleading pictures.

So, what did we learn here?

First off, there are no perfect football statistics.  Each analysis gives us a view from one particular angle, but no statistic definitively provides a complete picture.

With respect to Arian, it's interesting to note the difference between a fan's perspective and the heartless perspective of numbers.  As fans, we vividly remember the brilliant first down run from within our own 5 and the incredible 42 yard touchdown, but we tend to forget the missed opportunities to keep the chain moving in the fourth quarter. 

This is not intended to discredit Arian (he is still tremendously valuable and Brian's stats from the 2010 season bear that out), but what I think is really telling is that even though Matt Schaub didn't have very impressive traditional numbers, he was tremendously valuable in helping the Texans win.  And isn't that what this is all about?

As for Cody, the reason I included him in the discussion is because there's actually been a bit of disagreement as to how good or bad he's really been.  He had some nice plays against Pittsburgh, but as I mentioned with Arian, we as fans tend only to remember the highlight plays, whereas the figures look at all of them.  He really hasn't made the plays expected of him.

Again, this doesn't finalize any arguments -- in fact, it would be wrong to look at this as anything other than a very cursory look -- but hopefully you find this an interesting perspective, and I hope that you all head over to Brian's site.  He's got some great info over there. 

He's also got some great sponsors.  Maybe not quite as good as Samsung and Guinness, but you should check those out too.