Evaluating the Bullpen: Shutdowns and Meltdowns

The evaluation of relievers might be evolving more slowly than any other area of baseball, but there are statistics out there that are attempting to weight relievers’ roles more accurately. We’ll get to those, but first, some discussion on closers.

The shiny toy in any team’s bullpen is its closer. Managers (usually) take their best relievers, make them the closers, and let them pitch the ninth inning in save situations. The most common measurement of closer’s skill is measured by saves and blown saves, and closers are paid handsomely for accumulating the former. But the problem is that saves—and reliever outings in general—are not all created equally.

Wilhelmsen throwing MartiniTom Wilhelmsen entered a game in the ninth against the White Sox back in April with a comfortable two-run lead and nobody on base. He proceeded to walk three batters, allowing one to score, and he put the winning run on first base. With a strikeout, the Mariners escaped with a one-run win and the Bartender picked up a save. Compare that to his performance a few weeks ago against the Angels when he entered the game with a one-run lead and had to face Albert Pujols, Josh Hamilton, and Mark Trumbo. A perfect inning later, and Wilhelmsen got another save—but hardly the same as his save against the White Sox earlier in the year.

The advent of win expectancy allowed for the formulation of a new set of statistics for relievers: shutdowns and meltdowns. Those links are good primers on the subject, but here’s a quick rundown through examples:

A reliever comes in with his team down a run in the seventh inning. “Win expectancy” says a typical team in this situation has a 26 percent chance to win. Three outs later, he leaves the game having allowed no runs, improving his team’s chances of winning to 30 percent. After his team knocks one home to tie the score, the reliever is brought back in for the top of the eighth inning, and again he slams the door. His team began the eighth with a 50 percent chance of victory, and it leaves the top half of the eighth with a 59 percent win expectancy. Overall, that pitcher helped to improve his team’s chances by 13 percent, and that, my friends, is a “shutdown.”

But a week later it’s the bottom of the eighth inning, and the reliever’s team is down one run. His team is down, but not out, living on a 12 percent chance to win. The reliever gives up two more runs on a single and a homerun, and now—down three runs going into the ninth—his team has just a three percent chance of winning. The reliever cost his team nine percent in win expectancy. That’s a bad thing, and he’s going to get a “meltdown.”

That man was actually Carter Capps, first against the Angels and then against the Pirates. Though he didn’t throw a single pitch in the ninth inning, his pitching was still important in both games.  The Mariners were able to win against the Angels partly because Capps kept them in the game, but then—even though they didn’t end up scoring anymore runs against the Pirates—he made it a lot harder for the M’s to win that game.

These statistics, like most, aren’t perfect—closers, for instance, get a bit of an advantage by almost always pitching in the ninth—but I think shutdowns and meltdowns help to articulate the work that middle relievers do a lot more accurately than holds and saves. Serving drinks in the middle innings can be just as important as the Bartender’s orchestration of closing time in the ninth. So with that, let’s take a look at the M’s bullpen in terms of shutdowns and meltdowns, courtesy of Fangraphs:

Name pLI SD MD
Tom Wilhelmsen 1.47 7 0
Stephen Pryor 1.65 4 0
Oliver Perez 0.66 2 0
Carter Capps 0.93 5 3
Yoervis Medina 0.37 0 0
Lucas Luetge 0.23 1 1
Hector Noesi 0.17 0 0
Bobby LaFromboise 0.79 0 0
Kameron Loe 0.45 1 1
Blake Beavan 0.32 0 2
Charlie Furbush 0.84 0 4

Though Wilhelmsen has seen a better part of the workload, Pryor has actually performed admirably in the most stressful circumstances according to his average leverage index (pLI), which at 1.65 is well above the 1.00 average. You’ll also notice that, while our bartender has ten saves, he only has seven shutdowns. Often, three run leads in the ninth inning are safe enough that a pitcher can’t record a shutdown for his work. In either case, Wilhelmsen’s and Pryor’s results are flawless. Charlie Furbush, on the other hand, has had a tough year to this point, pushing the Mariners significantly further out of contention four times.

As a side note, the chart above can also articulate something about bullpen usage. Wilhelmsen and Pryor are the only pitchers whose average usage occurs during abnormally stressful situations. Eric Wedge has placed increased faith in those two. We can and should still evaluate relievers in terms of strikeout rates, homerun rates, walk rates, and even ERA if the sample size is large enough. But as Dave Cameron argues in that article, this should really kill the saves statistic.