This doesn't disprove your general point, but both Olerud and Pujols were excellent defensively. A -5 def guy is an excellent defender relative to the rest of 1B, because it takes into account the harsh positional penalty since it's a much easier position to field. Pujols from 2006-2008 is basically as good as you'll ever see in terms of defensive value added at first base.