Home
Our Approach to Rating Players
------------------------------
Our philosophy
--------------
The most important point we can make about our approach to rating players is
that we rate players based on performance, not reputation. This philosophy
influences the meaning of our ratings and how we come up with them in the
first place.
Our outfielder throwing rating, for example, measures the fielder's ability
to prevent runners from taking extra bases and to throw them out when they
try to advance. A fielder earns a good rating by positioning himself well,
getting to the ball quickly, making a quick release, throwing with power,
throwing with accuracy, and throwing to the right base. Someone with a
powerful arm might still get a subpar rating if he doesn't get into throwing
position quickly, throws wildly, or often throws to the wrong base.
Similarly, our running rating is not just a measure of raw speed, but also
the ability to read the ball off the bat, get a good jump, and make good
decisions about when to go for the extra base. Speed does matter, and
you'll find plenty of speedsters with top baserunning ratings. But there
are other players who run the bases well because their instincts are
so good, and some swift runners who don't make the most of their chances.
And our rating for defensive range measures the ability to make plays. An
infielder earns a good rating through positioning, quickness, soft hands,
and effective throwing (quick release, arm strength, accuracy). It's not
always the flashiest player who makes the most of the balls that are hit
his way.
In other words, the things you can see when you watch them play, and the
things the scouts often talk about, don't always translate into results.
Sometimes the things you cannot see -- such as the mental aspects of the
game -- are as important or more important in how the player actually
performs in game action. Our results-oriented analysis often uncovers
players with good instincts who accomplish a lot more than the scouts might
have predicted based on their raw tools.
The role of analysis
--------------------
Because our ratings measure overall ability to succeed in a certain phase of
the game, we look for ways to evaluate performance by analyzing play-by-play
data. This approach is not a radical one. Baseball people have been doing
this for over a century to measure batting and pitching performances. We
don't, after all, give the batting title to the guy with the prettiest swing,
we give it to the player who hit for the highest average. We don't give the
Cy Young to the pitcher with the best mechanics, we give it to the one who
was most effective. Using statistics to evaluate performance is part of the
tradition of the game.
But this tradition usually extends only to hitting and pitching. You never
hear a television or radio analyst talk about meaningful measures of
baserunning, throwing or defense. Instead, they talk about their
impressions of the player -- how fast he looks, his quickness, strength and
athleticism. If we applied the same standard to hitters, we'd never talk
about on-base percentage or slugging average. If we did the same with
pitchers, we'd never talk about walk-strikeout ratios and hits per nine
innings.
Our approach is to apply the time-honored tradition of using well-crafted
statistics to evaluate baseball performance. The difference is that we
don't stop at hitting and pitching. We design ways to measure results in
all phases of the game.
We sometimes find players whose performance is better or worse than you
would guess by watching them a few times a year. And our ratings are
occasionally at odds with the opinions being expressed by some of
baseball's most famous writers and TV personalities. But we sincerely
believe that doing original research into player skills is an important
part of producing an accurate baseball simulation.
Suppose a player has a reputation for great defense but our analysis doesn't
show a superior performance. If we give in to public opinion and rate him
higher than his performance justifies, we'd have these options:
* reduce the range rating of one of his teammates so the team's defense
isn't overrated
* reduce the effectiveness of the team's pitchers to compensate for the
extra plays this player will now make in the simulation
* disregard these side effects and allow the player, the team, and its
pitchers to produce better results than they should
We don't think it's fair to downgrade teammates so we can give a popular
player a better rating than he deserves. And we don't think you'd want us
to disregard the side effects and publish a season database with players and
teams who will overperform. So we do our best to rate players based on
performance, even if that means we might take a little heat for a few of
our ratings.
Bunting Ratings
---------------
Assigning bunt ratings would be easy if all we had to do was look at the
leaders in sacrifice bunts each year. But it's not that simple, because
a player's batting stats don't tell us how many times he failed to get a
bunt down when called upon. And we have separate ratings for sacrificing
and bunting for a hit, and if you're limited to the official batting stats,
there's no way to tell how many bunt singles a player legged out this year.
So we wrote a program to compile bunting information from the play-by-play
and pitch-by-pitch data we work with each year. This program looks at
three situations -- bunting for a hit, sacrifices, and squeeze bunts -- and
counts the number of opportunities to bunt, the number of bunt attempts and
the results of those attempts. The results include separate categories for
bunt hits, reaching on an error, being put out at first with other runners
advancing, popping out, grounding into a fielders choice in which a lead
runner is retired, grounding or popping into a double play, and fouling off
one or more bunt attempts before swinging away.
The number of bunts fouled off is quite important. Typically, in ordinary
sacrifice situations, batters are successful in advancing one or more
runners more than 85% of the time. Provided they got the ball in play,
that is. But roughtly 35% of the time, they bunt foul once or twice before
swinging away. Counting this type of failure drops the success rate to 57%.
In other words, there are many more bunting "failures" due to fouled bunts
than to poorly-placed bunts, and we would be missing a lot of important
information if we tried to measure bunting ability without considering fouls.
So our bunt ratings take all of these factors into consideration. And it
sometimes means that a player with loads of successful sacrifices is given
a mediocre bunt rating because he also failed on a lot of attempts.
Running and throwing
--------------------
There are no official statistics for baserunning, though you may occasionally
see some data on how often runners go from first to third on a single. Our
studies examine how often runners took extra bases on hits and fly balls,
taking into account where the ball was hit (it's much easier to go to third
on a ball hit to right), the number of outs (getting the jump with two outs
makes a big difference), and whether the playing surface is grass or turf
(ground balls reach the outfielder more quickly on old-style artificial turf,
so it's harder to take the extra base on many singles.) We also look at how
often each runner was thrown out on the bases.
Before wrapping up our running ratings, we also review a report showing the
number of times players were used as or replaced by a pinch runner. If a
manager frequently uses player A to run for player B, it's a good indication
that the manager feels A is a much better runner. It doesn't mean that A is
a great runner or that B is a lousy runner. It could be an average runner
replacing a poor one, or it could be an excellent runner replacing an average
one. That's why we look at who's replacing whom, not just the raw count
of pinch-running appearances or replacements.
Outfield assist totals give us some insight into outfielder throwing, but
they can also be misleading. Some great throwing outfielders have their
assist totals cut down by their opponent's unwillingness to run on them in
the first place. And not all assists are created equal. Some assists result
when a throw nails a lead runner, while others occur when the lead runners
score but the batter is out at second advancing on the throw. So we've
written a program that allows us to see how many quality assists each
outfielder earned during the season.
Our throwing ratings focus on the ability to prevent the lead runners from
advancing (and to throw out those who try) using a combination of skills:
getting to the ball quickly, getting the throw off quickly, and throwing
with strength and accuracy. Some players with strong arms are not very
accurate. Some players with weak arms compensate by getting into throwing
position more quickly. As with all of our ratings, we measure results, not
just how often a player looks good making a throw.
Stolen base defense
-------------------
When rating pitchers and catchers for the ability to prevent stolen bases,
it's important to go beyond looking at simple statistics such as the
percentage of runners thrown out by a catcher. Every play starts with the
ball in the pitcher's hand, and some studies suggest that pitchers may be
twice as important as catchers in preventing steals.
It's entirely possible for a catcher to look good or bad, not because of
his own ability to throw, but because of the pitchers he is working with.
Likewise, a catcher with a cannon for an arm can make his pitchers look good.
So our base stealing studies measure performance by pitcher-catcher tandems.
By comparing a pitcher's results with different catchers and vice versa, we
learn more than we would if we merely looked at totals for the pitchers and
catchers alone.
We also log all stolen base attempts, so if there's a question about the
rating of a particular pitcher and catcher, we can see who was stealing
bases against them. If, by chance, the league's best runners were on base
a lot in your games, it's hard for you to look good. On the other hand,
if you were giving up stolen bases to runners who don't run all that often,
it can mean that you're viewed as easy pickings.
Measuring Defensive Range
-------------------------
Defensive playmaking ability -- turning batted balls into outs -- is one
of the hardest baseball skills to evaluate.
Tradional fielding statistics -- putouts, assists, and errors -- tell us
how many plays a fielder was involved in, but they don't tell us how many
times he had an opportunity to make a play. A sure-handed shortstop may
have very few errors but let his team down by failing to reach ground balls
that other shortstops turn into outs. You can't figure that out just by
looking at traditional fielding statistics.
Some publications rank players based on range factors (putouts plus assists
per game) or fielding runs (a metric developed by Pete Palmer for the Total
Baseball encyclopedia). But these statistics are based on the traditional
statistics and therefore suffer from the same limitations. Specifically,
rankings based on range factors assume that each of those players had the
same number of opportunities to make plays. But they don't for several
reasons:
- some pitching staffs strike out more hitters and therefore allow fewer
batted balls to enter the field
- the mix of ground balls and fly balls can vary quite substantially from
one pitching staff to another
- the mix of left- and right-handed pitchers varies from team to team,
meaning that those teams face different percentages of left-handed hitters,
which in turn affects how many ground balls are hit to each side of the
infield
- some pitchers are better than others, and weaker pitchers present fielders
with a higher percentage of difficult chances
- some ballparks require that fielders position themselves differently,
affecting how many batted balls can be reached with normal effort
For all of these reasons, it's much better to count the number of
opportunities presented to each fielder and the number of times each fielder
was able to turn those opportunities into outs. To that end, we have
developed a series of analytical tools that use play-by-play data to
assess fielding ability. Our methods, which are described in detail in an
article ("Evaluating Defense") that you can find on our web site, take all
of the above-mentioned factors into account.
We believe this analytical approach is much better than relying on the
subjective opinions of baseball writers and broadcasters. And it is much
better than relying on the Gold Glove voters to identify the best fielders.
Much too often, these observers focus almost all of their attention on one
aspect of defensive performance -- errors -- and neglect the ability to
reach batted balls in the first place.
Wrapping up
-----------
We put a lot of effort into our player ratings each year. It would, of
course, be much easier if we just carried forward a player's ratings from
year to year or if we based our ratings on what we hear in the media. But
we don't believe we'd be doing our job if we did that.
Everybody knows that batting and pitching performances go up and down over
the course of a player's career. We believe the same is true of other skills
such as defense and baserunning. Using sophisticated analytical techniques
and the best information available helps us spot changes in performance when
they happen, not later when the media picks up on them.
But judgment and common sense are essential, too. Many players do not get
enough playing time in a single season to enable us to reach conclusions
based on numbers alone. Sometimes the way a player is used can produce a
biased view of his abilities. So we always look at the player's results in
the context of his team, career (including minor league records for younger
players) and an overall understanding of how baseball works.