This. The problem with Rivera is the problem with closers, which is the problem with relief pitchers generally. They just don't pitch enough to accrue that much value individually. And like you said, they're all pretty much failed starters. Even if Rivera is the greatest reliever ever, where does that put him among all pitchers ever? Pretty far down the list, actually. Let alone adding in position players.
Here's a thought experiment: if Rivera is really among the greatest players ever, and "the closer" is really so important, then the Dodgers are being pretty stupid wasting Clayton Kershaw as a starter, no? He's the best pitcher on that team, everyone would agree. They should give up his 7-8+ innings each start and put him in the pen where he throws 1 inning at a time.
But of course that would be stupid.
By that same logic the Jays were wasting Halladay all those years as a starter throwing 230 innings, when they could have had him as the closer throwing like 65 innings a year.