The Simple Rating System: Bringing order (kinda) to chaos

[Ed. Note: This is the first post by my good friend and stat guru Chase Stuart. For at least the rest of this season, Chase will be contributing his unique perspective to the site. Chase has previously contributed to the New York Times Fifth Down and the Pro-Football Reference Blog. You can also follow Chase on twitter.]

The last two seasons, I have published college football ratings using the Simple Rating System. Before explaining how the Simple Rating System (SRS) works, allow me to first explain what the SRS is trying to do (and just as importantly, what it’s not trying to do).

A want and enjoyment of numerosity

Most rating systems fall into one of two categories. A rating system could simply replicate the standings in any particular league; such a rating system would best be described as retrodoctive or explanatory. A retrodictive rating system fits the data to explain what happened in the past. The BCS computer ratings are mostly retrodictive; so are player or team ratings that give significant weight to high-leverage plays that tend to be highly random (clutch play, fumble recovery rates, etc.). An explanatory rating system tries to measure how much a team or player has accomplished in the past; it does not attempt to answer the question “what will happen next?” When Bill Parcells said “You are what your record says you are,” he’s championing retrodictive ratings. So was Rich Kotite when, coaching the 7-2 Eagles in 1994, he said to the media: “Judge me by my record.” An explanatory rating system would say that Kotite and his Eagles were doing well; but it would never have predicted that Kotite would go 4-35 over the next — and final — 39 games of his career.

The other type of rating system is a predictive system, which works as they name implies: it tries to predict the future. Here is a useful chart detailing some of the differences between the two in college football rating systems. Predictive rating systems are not very concerned with wins and losses; instead, they focus on more granular pieces of data. The best and most obvious example of a predictive rating system would be the formulas used by the folks in Vegas. Those who make point spreads aren’t disturbed if their rankings place Team A, which has “accomplished less” than Team B, higher up in their rankings. This weekend produced a useful example. No purely retrodictive rating system would put the Oklahoma Sooners ahead of the Kansas State Wildcats. Oklahoma was 6-1 but lost to a mediocre Texas Tech team; Kansas State was undefeated and had beaten some solid teams, albeit in less than thrilling fashion. Both the BCS ratings and the Associated Press’ rankings had Kansas State over Oklahoma, because those systems are designed to acknowledge accomplishments. But despite being the higher ranked team and playing at home, Kansas State was a 14-point underdog to the Sooners. And Oklahoma promptly went into Manhattan and blew out the Wildcats, 58-17.

The SRS is a predictive system, which means it could theoretically place a 3-5 team ahead of a 7-1 team. As a sanity check, it usually tends to correlate pretty well with the point spread in most games (and it’s worth trying to understand the deviation when the lines do not match up). But as the name implies, it’s simple. The SRS does not factor in the thousands of pieces of data one could place into a rating system, trading precision for elegance and ease of understanding. Here’s how it works:

The SRS takes only two factors into account: strength of schedule and adjusted margin of victory. Each game is given equal weight. Therefore, the sum of a team’s SOS and MOV rating is its SRS rating. A team could have an SRS rating of 60 by having an MOV of 30 and an SOS of 30, or an MOV rating of 40 and an SOS of 40. Once you have the SRS scores for each team, it’s very simple to understand how the system arrived at those ratings. Further, the numbers the system spits out are easy to understand: if Team A has a rating of 55 and Team B has a rating of 44, it means that Team A is predicted to be 11 points better than Team B. The units here mean exactly what you think they mean.

It’s complicated to create these ratings, but I’ve done that in Excel for you. The tricky part is that each team’s strength of schedule is dependent on the ratings of each of their opponents, which is dependent on the ratings of each of their opponents, which includes the original team we’re trying to rate. If you adjust each team’s rating over hundreds of thousands of iterations, eventually the ratings converge, and we’re left with “true” ratings.

One last note: I wrote about that the SRS uses adjusted margin of victory. What does that mean? For starters, the road team is given 3 points for each game (but there is no home or road team for neutral site games). After that adjustment, all wins and losses of between 7 and 24 points as exactly that. So a 24-10 road win goes down as +17 for the road team, -17 for the home team. Wins of 7 or fewer points are scored as 7-point wins and losses of 7 or fewer points are scored as 7 point losses (except that road losses of 3 or fewer/home wins of 3 or fewer are graded as 0 point ties). This gives a very minor boost to teams that win by a couple of points. Finally, wins of more than 24 points/losses of more than 24 points are scored as the average between the actual number and 24. This is to avoid giving undue credit to teams that run up the score.

Now, the college football SRS ratings after week 9:

Rk  Team                 Con   G    MOV      SOS      SRS      W-L
1.  LSU                  SEC   8    24.5     43.9     68.4     8-0
2.  Alabama              SEC   8    27.4     40.7     68.1     8-0
3.  Oklahoma St          B12   8    22.0     45.2     67.2     8-0
4.  Stanford             P12   8    27.4     39.2     66.5     8-0
5.  Oklahoma             B12   8    21.5     44.1     65.6     7-1
6.  Boise St             MWC   7    22.7     41.2     63.9     7-0
7.  Oregon               P12   8    20.7     42.0     62.7     7-1
8.  Wisconsin            B10   8    21.5     38.0     59.5     6-2
9.  Michigan             B10   8    17.4     40.6     57.9     7-1
10. Texas A&M            B12   8     9.9     46.3     56.2     5-3
11. Arizona St           P12   8    13.1     42.7     55.8     6-2
12. Notre Dame           IND   8     9.1     45.6     54.8     5-3
13. Nebraska             B10   8    12.3     42.4     54.8     7-1
14. Houston              CUS   8    22.9     30.5     53.4     8-0
15. Missouri             B12   8     5.8     47.6     53.3     4-4
16. South Carolina       SEC   8    11.4     41.5     52.9     7-1
17. Michigan St          B10   8     9.1     43.7     52.9     6-2
18. Texas                B12   7     8.7     44.1     52.8     5-2
19. Clemson              ACC   9    12.1     40.2     52.3     8-1
20. Georgia              SEC   8     9.1     43.1     52.3     6-2
21. Southern Cal         P12   8     5.5     46.7     52.2     6-2
22. TCU                  MWC   8    16.2     34.7     50.9     6-2
23. Ohio State           B10   8     5.8     44.7     50.5     5-3
24. Arkansas             SEC   8    12.3     37.8     50.0     7-1
25. Toledo               MAC   8     9.8     40.1     49.8     5-3
26. Florida St           ACC   8    14.1     35.5     49.6     5-3
27. Virginia Tech        ACC   9    11.9     37.5     49.4     8-1
28. Penn State           B10   9     8.4     40.8     49.3     8-1
29. Kansas St            B12   8     6.4     42.6     49.0     7-1
30. Cincinnati           BgE   7    17.4     31.5     48.9     6-1
31. Baylor               B12   7     3.5     45.1     48.6     4-3
32. Southern Miss        CUS   8    15.8     32.4     48.1     7-1
33. West Virginia        BgE   8     9.9     38.0     48.0     6-2
34. Georgia Tech         ACC   9    12.7     34.7     47.4     7-2
35. Florida              SEC   8     3.9     43.4     47.3     4-4
36. Tennessee            SEC   8    -2.4     49.3     47.0     3-5
37. Miami FL             ACC   8     5.0     41.6     46.6     4-4
38. Temple               MAC   8    14.9     31.3     46.2     5-3
39. Illinois             B10   9     6.2     39.7     45.9     6-3
40. Washington           P12   8     3.0     42.8     45.8     6-2
41. North Carolina       ACC   9     5.3     39.8     45.1     6-3
42. Tulsa                CUS   8     4.0     40.9     44.9     5-3
43. Auburn               SEC   9    -0.3     45.3     44.9     6-3
44. Mississippi St       SEC   8     4.8     40.0     44.7     4-4
45. South Florida        BgE   7     8.0     36.5     44.5     4-3
46. Utah                 P12   8     1.1     42.9     43.9     4-4
47. Texas Tech           B12   8     4.9     38.9     43.9     5-3
48. Arizona              P12   8    -5.1     48.6     43.6     2-6
49. Vanderbilt           SEC   8     1.1     42.3     43.4     4-4
50. Iowa                 B10   8     9.2     34.0     43.2     5-3
51. San Diego St         MWC   7     3.2     39.6     42.8     4-3
52. Rutgers              BgE   8     7.3     35.5     42.8     5-3
53. Nevada               WAC   8     4.9     37.3     42.2     5-3
54. SMU                  CUS   8     3.0     38.7     41.7     5-3
55. Pittsburgh           BgE   8     1.1     40.1     41.2     4-4
56. California           P12   8     0.8     40.1     41.0     4-4
57. Arkansas St          Sun   8     9.6     31.0     40.5     6-2
58. Air Force            MWC   8     2.3     38.0     40.3     4-4
59. Brigham Young        IND   9     3.2     37.1     40.3     6-3
60. Iowa St              B12   8    -6.8     46.4     39.6     4-4
61. Purdue               B10   8     2.3     37.2     39.4     4-4
62. Northern Illinois    MAC   8     6.4     32.5     38.9     5-3
63. Utah St              WAC   7     4.4     34.4     38.8     2-5
64. UCLA                 P12   8    -6.8     45.4     38.6     4-4
65. Louisiana Tech       WAC   8     0.6     38.0     38.6     4-4
66. Syracuse             BgE   8     0.6     37.9     38.5     5-3
67. Northwestern         B10   8     0.5     37.4     37.9     3-5
68. Virginia             ACC   8     1.5     36.3     37.8     5-3
69. Washington St        P12   8    -0.4     38.1     37.7     3-5
70. Louisville           BgE   8     0.3     37.3     37.5     4-4
71. Ohio U.              MAC   8    10.9     26.6     37.5     5-3
72. Western Michigan     MAC   9     4.1     33.4     37.5     5-4
73. Wake Forest          ACC   8     1.4     36.0     37.4     5-3
74. Hawai`i              WAC   8     7.2     29.9     37.0     5-3
75. Navy                 IND   8    -3.3     40.0     36.7     2-6
76. Central Florida      CUS   8     8.9     26.5     35.4     4-4
77. North Carolina St    ACC   8    -0.9     36.3     35.4     4-4
78. Oregon St            P12   8    -7.1     42.5     35.4     2-6
79. Maryland             ACC   8    -8.3     43.4     35.2     2-6
80. Mississippi          SEC   8    -9.4     44.5     35.1     2-6
81. Fresno St            WAC   8    -5.1     40.1     35.0     3-5
82. Connecticut          BgE   8    -0.3     35.1     34.8     3-5
83. UTEP                 CUS   8    -0.6     34.8     34.3     4-4
84. Wyoming              MWC   7     1.1     33.0     34.1     5-2
85. Marshall             CUS   9    -5.7     39.5     33.9     4-5
86. Florida Int'l        Sun   8     2.8     31.1     33.9     5-3
87. Louisiana-Lafayette  Sun   9     5.2     28.3     33.6     7-2
88. Bowling Green        MAC   9    -1.1     34.3     33.2     4-5
89. San José St          WAC   8    -6.4     39.4     33.0     3-5
90. Miami OH             MAC   8    -4.6     37.3     32.8     3-5
91. Boston College       ACC   8    -5.7     38.0     32.3     2-6
92. Duke                 ACC   8    -5.4     37.7     32.2     3-5
93. East Carolina        CUS   8    -3.9     35.6     31.7     4-4
94. Kentucky             SEC   8    -9.4     40.7     31.3     3-5
95. Ball St              MAC   9    -6.7     37.7     31.0     5-4
96. Colorado             P12   9   -16.6     46.8     30.2     1-8
97. Army                 IND   8    -0.4     30.4     30.0     3-5
98. Minnesota            B10   8   -13.8     43.6     29.8     2-6
99. Eastern Michigan     MAC   8    -2.6     32.1     29.4     5-3
100.Rice                 CUS   8   -13.0     41.9     28.9     2-6
101.Kansas               B12   8   -19.4     47.8     28.4     2-6
102.Western Kentucky     Sun   8    -2.4     30.4     28.0     4-4
103.New Mexico St        WAC   8    -3.8     31.6     27.8     3-5
104.Louisiana-Monroe     Sun   8    -5.9     33.2     27.3     2-6
105.Kent St              MAC   8   -12.1     39.0     26.9     2-6
106.North Texas          Sun   9   -11.7     38.4     26.8     3-6
107.Indiana              B10   9   -12.4     38.9     26.6     1-8
108.Buffalo              MAC   9   -10.4     36.9     26.5     2-7
109.Idaho                WAC   8    -9.4     35.6     26.1     1-7
110.Middle Tennessee St  Sun   7    -4.0     29.3     25.3     2-5
111.Central Michigan     MAC   9    -7.6     32.6     25.1     3-6
112.Troy                 Sun   7    -9.5     34.3     24.8     2-5
113.Colorado St          MWC   8    -5.4     29.5     24.1     3-5
114.UNLV                 MWC   7   -18.0     37.7     19.7     2-5
115.Akron                MAC   8   -15.5     33.5     18.0     1-7
116.Florida Atlantic     Sun   7   -19.2     37.0     17.7     0-7
117.Alabama-Birmingham   CUS   8   -17.3     34.8     17.6     1-7
118.Tulane               CUS   9   -11.9     27.8     15.9     2-7
119.Memphis              CUS   9   -16.0     29.9     13.9     2-7
120.New Mexico           MWC   8   -26.9     39.4     12.5     0-8

Here are the same standings but listed by conference:

Rk  Team                 Con   G    MOV      SOS      SRS      W-L
19. Clemson              ACC   9    12.1     40.2     52.3     8-1
26. Florida St           ACC   8    14.1     35.5     49.6     5-3
27. Virginia Tech        ACC   9    11.9     37.5     49.4     8-1
34. Georgia Tech         ACC   9    12.7     34.7     47.4     7-2
37. Miami FL             ACC   8     5.0     41.6     46.6     4-4
41. North Carolina       ACC   9     5.3     39.8     45.1     6-3
68. Virginia             ACC   8     1.5     36.3     37.8     5-3
73. Wake Forest          ACC   8     1.4     36.0     37.4     5-3
77. North Carolina St    ACC   8    -0.9     36.3     35.4     4-4
79. Maryland             ACC   8    -8.3     43.4     35.2     2-6
91. Boston College       ACC   8    -5.7     38.0     32.3     2-6
92. Duke                 ACC   8    -5.4     37.7     32.2     3-5
8.  Wisconsin            B10   8    21.5     38.0     59.5     6-2
9.  Michigan             B10   8    17.4     40.6     57.9     7-1
13. Nebraska             B10   8    12.3     42.4     54.8     7-1
17. Michigan St          B10   8     9.1     43.7     52.9     6-2
23. Ohio State           B10   8     5.8     44.7     50.5     5-3
28. Penn State           B10   9     8.4     40.8     49.3     8-1
39. Illinois             B10   9     6.2     39.7     45.9     6-3
50. Iowa                 B10   8     9.2     34.0     43.2     5-3
61. Purdue               B10   8     2.3     37.2     39.4     4-4
67. Northwestern         B10   8     0.5     37.4     37.9     3-5
98. Minnesota            B10   8   -13.8     43.6     29.8     2-6
107.Indiana              B10   9   -12.4     38.9     26.6     1-8
3.  Oklahoma St          B12   8    22.0     45.2     67.2     8-0
5.  Oklahoma             B12   8    21.5     44.1     65.6     7-1
10. Texas A&M            B12   8     9.9     46.3     56.2     5-3
15. Missouri             B12   8     5.8     47.6     53.3     4-4
18. Texas                B12   7     8.7     44.1     52.8     5-2
29. Kansas St            B12   8     6.4     42.6     49.0     7-1
31. Baylor               B12   7     3.5     45.1     48.6     4-3
47. Texas Tech           B12   8     4.9     38.9     43.9     5-3
60. Iowa St              B12   8    -6.8     46.4     39.6     4-4
101.Kansas               B12   8   -19.4     47.8     28.4     2-6
30. Cincinnati           BgE   7    17.4     31.5     48.9     6-1
33. West Virginia        BgE   8     9.9     38.0     48.0     6-2
45. South Florida        BgE   7     8.0     36.5     44.5     4-3
52. Rutgers              BgE   8     7.3     35.5     42.8     5-3
55. Pittsburgh           BgE   8     1.1     40.1     41.2     4-4
66. Syracuse             BgE   8     0.6     37.9     38.5     5-3
70. Louisville           BgE   8     0.3     37.3     37.5     4-4
82. Connecticut          BgE   8    -0.3     35.1     34.8     3-5
Rk  Team                 Con   G    MOV      SOS      SRS      W-L
14. Houston              CUS   8    22.9     30.5     53.4     8-0
32. Southern Miss        CUS   8    15.8     32.4     48.1     7-1
42. Tulsa                CUS   8     4.0     40.9     44.9     5-3
54. SMU                  CUS   8     3.0     38.7     41.7     5-3
76. Central Florida      CUS   8     8.9     26.5     35.4     4-4
83. UTEP                 CUS   8    -0.6     34.8     34.3     4-4
85. Marshall             CUS   9    -5.7     39.5     33.9     4-5
93. East Carolina        CUS   8    -3.9     35.6     31.7     4-4
100.Rice                 CUS   8   -13.0     41.9     28.9     2-6
117.Alabama-Birmingham   CUS   8   -17.3     34.8     17.6     1-7
118.Tulane               CUS   9   -11.9     27.8     15.9     2-7
119.Memphis              CUS   9   -16.0     29.9     13.9     2-7
12. Notre Dame           IND   8     9.1     45.6     54.8     5-3
59. Brigham Young        IND   9     3.2     37.1     40.3     6-3
75. Navy                 IND   8    -3.3     40.0     36.7     2-6
97. Army                 IND   8    -0.4     30.4     30.0     3-5
25. Toledo               MAC   8     9.8     40.1     49.8     5-3
38. Temple               MAC   8    14.9     31.3     46.2     5-3
62. Northern Illinois    MAC   8     6.4     32.5     38.9     5-3
71. Ohio U.              MAC   8    10.9     26.6     37.5     5-3
72. Western Michigan     MAC   9     4.1     33.4     37.5     5-4
88. Bowling Green        MAC   9    -1.1     34.3     33.2     4-5
90. Miami OH             MAC   8    -4.6     37.3     32.8     3-5
95. Ball St              MAC   9    -6.7     37.7     31.0     5-4
99. Eastern Michigan     MAC   8    -2.6     32.1     29.4     5-3
105.Kent St              MAC   8   -12.1     39.0     26.9     2-6
108.Buffalo              MAC   9   -10.4     36.9     26.5     2-7
111.Central Michigan     MAC   9    -7.6     32.6     25.1     3-6
115.Akron                MAC   8   -15.5     33.5     18.0     1-7
6.  Boise St             MWC   7    22.7     41.2     63.9     7-0
22. TCU                  MWC   8    16.2     34.7     50.9     6-2
51. San Diego St         MWC   7     3.2     39.6     42.8     4-3
58. Air Force            MWC   8     2.3     38.0     40.3     4-4
84. Wyoming              MWC   7     1.1     33.0     34.1     5-2
113.Colorado St          MWC   8    -5.4     29.5     24.1     3-5
114.UNLV                 MWC   7   -18.0     37.7     19.7     2-5
120.New Mexico           MWC   8   -26.9     39.4     12.5     0-8
4.  Stanford             P12   8    27.4     39.2     66.5     8-0
7.  Oregon               P12   8    20.7     42.0     62.7     7-1
11. Arizona St           P12   8    13.1     42.7     55.8     6-2
21. Southern Cal         P12   8     5.5     46.7     52.2     6-2
40. Washington           P12   8     3.0     42.8     45.8     6-2
46. Utah                 P12   8     1.1     42.9     43.9     4-4
48. Arizona              P12   8    -5.1     48.6     43.6     2-6
56. California           P12   8     0.8     40.1     41.0     4-4
64. UCLA                 P12   8    -6.8     45.4     38.6     4-4
69. Washington St        P12   8    -0.4     38.1     37.7     3-5
78. Oregon St            P12   8    -7.1     42.5     35.4     2-6
96. Colorado             P12   9   -16.6     46.8     30.2     1-8
1.  LSU                  SEC   8    24.5     43.9     68.4     8-0
2.  Alabama              SEC   8    27.4     40.7     68.1     8-0
16. South Carolina       SEC   8    11.4     41.5     52.9     7-1
20. Georgia              SEC   8     9.1     43.1     52.3     6-2
24. Arkansas             SEC   8    12.3     37.8     50.0     7-1
35. Florida              SEC   8     3.9     43.4     47.3     4-4
36. Tennessee            SEC   8    -2.4     49.3     47.0     3-5
43. Auburn               SEC   9    -0.3     45.3     44.9     6-3
44. Mississippi St       SEC   8     4.8     40.0     44.7     4-4
49. Vanderbilt           SEC   8     1.1     42.3     43.4     4-4
80. Mississippi          SEC   8    -9.4     44.5     35.1     2-6
94. Kentucky             SEC   8    -9.4     40.7     31.3     3-5
57. Arkansas St          Sun   8     9.6     31.0     40.5     6-2
86. Florida Int'l        Sun   8     2.8     31.1     33.9     5-3
87. Louisiana-Lafayette  Sun   9     5.2     28.3     33.6     7-2
102.Western Kentucky     Sun   8    -2.4     30.4     28.0     4-4
104.Louisiana-Monroe     Sun   8    -5.9     33.2     27.3     2-6
106.North Texas          Sun   9   -11.7     38.4     26.8     3-6
110.Middle Tennessee St  Sun   7    -4.0     29.3     25.3     2-5
112.Troy                 Sun   7    -9.5     34.3     24.8     2-5
116.Florida Atlantic     Sun   7   -19.2     37.0     17.7     0-7
53. Nevada               WAC   8     4.9     37.3     42.2     5-3
63. Utah St              WAC   7     4.4     34.4     38.8     2-5
65. Louisiana Tech       WAC   8     0.6     38.0     38.6     4-4
74. Hawai`i              WAC   8     7.2     29.9     37.0     5-3
81. Fresno St            WAC   8    -5.1     40.1     35.0     3-5
89. San José St          WAC   8    -6.4     39.4     33.0     3-5
103.New Mexico St        WAC   8    -3.8     31.6     27.8     3-5
109.Idaho                WAC   8    -9.4     35.6     26.1     1-7

Again, these rankings are far from perfect. But at least in theory, they’re pretty simple and easy to understand. They don’t get at everything we want to know, but you can quickly scan any team’s line and get a good sense of how their season has been. Each team’s rating is simply the sum of their (adjusted) margin of victory and their sum of their opponent’s average (adjusted) MOV. The SRS puts equal weight on all games, something the brain is not wont to do.

By way of example, let’s look at the craziness Texas Tech fans have had to deal with the past three weeks. Against Kansas State (SRS rating of 49.0), the Red Raiders lost at home by 7, so that grade counts as a 39 for purposes of the SRS. Then playing in Norman against the Sooners (SRS of 65.6), Tech somehow managed to upend Oklahoma by three points (+7 adjusted MOV, for an SRS grade of 72.6). This past weekend, the second worst team in the conference, Iowa State (SRS of 39.6), blew the doors off of Texas Tech, 41-7. That goes down as an adjusted MOV of -30.5, for an SRS grade of 9.1. There’s no way to make sense of a team with SRS grades of 39.0, 72.6 and 9.1 in three consecutive weeks.

One thing the SRS and almost everyone else agrees on: it’s pretty hard to separate LSU from Alabama.