Paranjai Patil and Shekhar Shah

Introduction

What is the primary objective for a General Manager in today’s NFL? Is it to maximize their team’s chances of winning the Super Bowl, regardless of the potential drawbacks of that strategy? Answering these questions is not a straightforward task for most teams, making it challenging to execute decisions with a singular goal in mind.

Deciding to extend their team’s quarterback to a second contract is perhaps the most crucial point at which a General Manager has to answer those questions. GMs are often forced to decide between re-signing their QBs or allocating premium draft capital towards new talent. Most opt for the safer play, re-signing their quarterbacks despite potential ceiling limitations. This piece will attempt to inform decisions on quarterback second contracts, using multiple logistic regression models to evaluate success for a quarterback and their team over the second phase of their careers.

Creating our Dataset

Using nflfastR play-by-play data, we were able to collect QB statistics and aggregate them into season totals. We used 100 pass attempts as the minimum cutoff to qualify for the dataset. Inspired by Predicting the Quarterback-MVP, we used a ranked version of each statistic (ranked against other QBs that year) to account for league-wide trends in QB stats. From nflfastR, we were able to acquire the following statistics:

  • Total Games Played
  • Passing/Rushing/Total Yards Per Game Rank
  • Passing/Rushing/Total TDs Per Game Rank
  • CPOE Rank
  • Average Win Probability Added Rank
  • EPA Rank
  • Completion Percentage Rank

After that, we narrowed down our dataset to include only QBs drafted after 2006 and removed quarterbacks who did not meet the pass attempt threshold in at least three of their four seasons in the NFL.

We then added PFF Ultimate Data to our dataset to further measure how well quarterbacks performed. From PFF, we were able to add the following statistics:

  • PFF Passing Grade
  • Big Time Throw Percentage
  • Turnover Worthy Play Percentage
  • Sack Rate
  • Scramble Rate

Next, we had to combine our data so that each quarterback only appeared once in the dataset and had all the variables named relative to their year in the league (for example, epa_rank from a player’s second season becomes epa_rank2 after the dataset is altered).

To adjust for some players having three years’ worth of data while others had four, players had only their last three years of data accounted for. Finally, we added the following general statistics from Pro Football Reference as well as other NFL statistics pages:

  • Regular Season Wins/Losses
  • Playoff Wins/Losses/Appearances
  • Conference Championship Appearances
  • Super Bowl Appearances/Wins
  • Pro Bowls Made

We also added the same statistical data from Years 5-8 of a player’s career to use as potential output variables to test our model on, as well as contract data from Over The Cap to predict second contracts.

Our final dataset contained 44 quarterbacks, with 37 used as training data and 7 to test our predictions.

Contract Model

Once we had all the variables, the first thing we wanted to do was set a baseline for what a team should expect to pay their quarterback through their second contract. Our contract data ranged from 2009 to 2023, so we had to find a metric that would withstand inflation over roughly 15 years.

Simply using average salary per year would not work because of the clear relationship between the APY of the max contract compared to the year it was signed.

We also tested APY as a percentage of the total cap for the year, but that metric also increased year over year. Quarterback salaries have been inflating faster than the cap, so we could not use % of the cap.

Finally, we started using APY relative to the previous year’s max APY as our metric of choice. For example, the max APY in 2021 was Patrick Mahomes’ 45M APY contract, and in 2022 Aaron Rodgers signed a 50.3M APY contract, an 11.7% increase over the previous max APY.

As seen below, there is minimal correlation between the year the contract was signed and the previous maximum APY. Because the metric had little noticeable pattern in its year-over-year change, we were able to use it to project contracts.

Once we had an inflation-proof metric, we were ready to train our model using XGBoost based on the several QB statistics we used to measure skill and performance. To account for the fact that the dataset had few rows but a relatively large number of columns, we used Ridge regularization as well as Lasso regularization. This prevented the model from overfitting and allowed for more feature diversity in the model.

We tested this model specifically for QBs in the 2020 class (Joe Burrow, Tua Tagovailoa, Justin Herbert, and Jalen Hurts). The model projected a coefficient that we multiplied by the previous year’s maximum APY to return an Expected APY for the QB’s second contract. To project the 2020 class signing this year, we multiplied the coefficient by the $50.3M APY of Aaron Rodgers’ deal.

These expected APYs can be used as an informed starting point for contract negotiations for their franchise quarterbacks. The predictions given by the rest of the models can be examined assuming the QBs received this amount of money. A contract paying them more or less money than these projections will likely impact the following estimates. 

Logistic Regression Models

For the logistic regression models, we chose to use GLM instead of XGBoost because of its simplicity. We narrowed down the set of variables by excluding most Year 1 variables as they tended not to matter as much as the Year 2 and Year 3 variables.

To reduce overfitting, we applied regularization techniques, this time focusing on the more important features rather than diversifying them. Some of the models were tested on data after 2016, and while the training vs. testing accuracy (~90% vs. ~80%) still indicated overfitting, the results were reasonable.

To compare our projections to an alternative for teams, we analyzed 1st round QBs on their rookie deals drafted between 2006-2019 and displayed the percentage of rookies who achieved the objectives.

Playoff Appearance Model

The first output we predicted using GLM was a binary Playoff Appearance variable, indicating whether a QB’s team made the playoffs at least once in Year 5 to Year 8 of the QB’s career. Out of the 37 QBs in the training dataset, 19 made the playoffs once. The model generated the following results.

Like many of the other models, it is heavily reliant on win percentage, but it’s worth noting that this one is the most reliant on win percentage. Most other coefficients are generally in line with the other models.

All of the QBs from the 2020 class, as well as Kyler and first-round rookies on average, have win percentages over 50%. The model rates Daniel Jones relatively low, despite leading his team to the playoffs just last year. These data points are especially useful for teams with a lack of playoff appearances over several years, helping them assess their QB’s ability to lead the team to the playoffs.

Conference Championship Appearance Model

The next output we tested was the binary Conference Championship Appearance variable, indicating whether a QB’s team reached the Conference Championship in Year 5 to Year 8 of their careers. Out of the 37 QBs in the training dataset, only 7 accomplished this feat. The projections for the upcoming QBs are displayed below.

This model also considers win percentage as the most important variable. Compared to the other models, it places higher importance on turnover-worthy play rate and CCG appearances. For teams aiming for a deep playoff run, a QB who only helps them make the playoffs may not be the one to take their team far in the playoffs.

Contract projections seem to significantly impact this metric. Burrow performs well despite high projections, but the model is skeptical of Hurts, Herbert, and Kyler. Due to the advantage of rookie contracts, a relatively high 17.5% of QBs on their rookie deal were able to reach the CCG. Tua is the only QB besides Burrow to exceed that number, but his projection could be much lower if he receives more than the projected $34.5M APY.

Pro Bowl Appearance Model

In addition to team metrics, we also wanted to evaluate individual accomplishments such as reaching the Pro Bowl. Out of the 37 QBs in the training set, 12 of them reached a Pro Bowl in Years 5-8. The model outputs the following results.

Win percentage is again the dominant feature, although not as pronounced as in other models. Playoff appearances are relatively more important, while turnover-worthy plays have less influence in this model.

The results of this model are relatively normal, with Joe Burrow, Jalen Hurts, and Justin Herbert leading the pack. For front offices, measuring a player’s individual success is critical in assessing their value when playing under a significantly higher second contract. While Pro Bowls aren’t a direct indicator of individual success, they often separate average from elite QB play.

Win Percentage Model

We also examined whether a quarterback could maintain a win percentage over 50% in Years 5-8 of their career. Out of the 37 quarterbacks, 15 of them achieved this status. The model’s testing results are as follows.

Interestingly, the win percentage model is the only one where win percentage is not the most important variable; in fact, it has a negative coefficient. The model, however, places a lot of emphasis on reaching/winning games in the playoffs.

Joe Burrow again leads the pack, followed closely by Jalen Hurts. On average, first-round rookies do not fare as well, but their projection is still higher than that of Tua Tagovailoa and Daniel Jones.

Super Bowl Model

Finally, the last and perhaps most important metric we tested was the odds of winning the Super Bowl. Out of the 37 training quarterbacks, only 3 achieved this feat: Joe Flacco, Nick Foles, and Patrick Mahomes. The model’s results are as follows.

The Super Bowl model once again considers win percentage as the most important factor. Conference Championship Game appearances have a greater impact in this model than in any other, which makes sense as playoff experience can contribute to winning a Super Bowl. This explains why Burrow is significantly ahead of the field, even more so than in the other metrics.

Note that the 2.5% of rookie quarterbacks to achieve this feat represents 1 out of 40 quarterbacks, with Patrick Mahomes being the one. Tua surpasses this threshold, while Herbert and Hurts do not. This supports the theory that large contract projections can negatively influence higher-level playoff success.

Conclusions

These models support the conclusion that Joe Burrow is in a league of his own, even when compared to Justin Herbert and Jalen Hurts. He finishes first in every metric except for playoff appearances, where he still ranks a very close second. This assessment is made even with the highest contract projection. The models strongly endorse the immediate re-signing of Burrow.

One interesting pattern that emerged is that in certain metrics evaluating playoff success, such as Super Bowl wins and Conference Championship appearances, Tua outperforms “better” QBs like Hurts and Herbert. This flips when looking at other metrics like win percentage and playoff appearances, which seem to be less dependent on having one truly great season.

The models indicate that paying a great QB a significant amount of money will not affect their ability to maintain lower levels of success, such as being above .500 and making the playoffs. However, it might ultimately hinder their team’s ability to go far in the playoffs.

This information can help teams like the Chargers decide whether or not to pay Herbert. While it may seem foolish to consider not re-signing Herbert, if the team is willing to take the risk of not being a perennial playoff contender in exchange for a slightly higher chance of making a deep playoff run, it could make sense. Instead of paying Herbert more than Mahomes and trying to compete with a weaker team, trading him for a treasure trove of draft picks and players could increase the team’s upside, if that is what they value most.

Another factor worth noting is the value of a first-round QB. In our study, more often than not, first-round QBs are projected to be better than serviceable QBs like Daniel Jones and sometimes even better than players like Tua Tagovailoa. It is important to acknowledge that contending teams like the Giants and Dolphins might not have access to the top first-round QBs. However, this difference is somewhat offset by the stronger rosters these teams have around their QB compared to teams picking at the top of drafts.

This further reinforces the notion of not paying top dollar in the second-year contract for QBs and suggests that trading for multiple first-round picks may provide a team with an almost equivalent probability of success in the playoffs as having an expensive QB would. The value of QBs on rookie contracts is heavily emphasized in this model, emphasizing the critical importance for teams to carefully consider all options before committing a significant portion of their salary cap to a QB.

Of course, trading away a star QB like Justin Herbert or Jalen Hurts is a longshot and may be detrimental to team building and chemistry. However, this model indicates that all options should be considered, and how the team chooses to handle a QB’s second contract is one of the most crucial steps in building a contending franchise.

Future Steps

One aspect we intend to explore in the future is the precise impact of the discrepancy between our projected contracts and the actual contracts received by QBs for our other projections. Our model would prove particularly valuable during the negotiation process, enabling teams to assess the long-term implications of a player’s APY on the team’s future success. This would provide teams with projections for every potential contract value they offer to a quarterback. Such information can aid in determining a maximum contract that a team is willing to offer based on the acceptable reduction in other probabilities.

In the future our models can also be trained with different outputs in order to analyze different aspects of success for quarterbacks. While our model focuses on QB’s, leveraging APY and contract information would be useful for other positions as well. These models can help form contract expectations for other players and project whether a player will meet performance objectives, such as reaching a specific sack total or number of receiving yards on their second contract. 

Code: https://github.com/Paranjai-Patil/QBModel