I think you are absolutely correct. Scouts absolutely LOVE to talk about guys "filling out" and moving down the defense spectrum. Whether that be 3B --> 1B, SS --> 3B, CF --> LF... it seems like every prospect who is is 6'2", 6'3" or greater gets some caveat about it.
There are probably historical examples of this happening, but "filling out" is almost never the primary cause. Players age, lose range, wear down, are no longer competent at positions... even the one example that comes immediately to mind, Miguel Cabrera, wasn't a case of him all of a sudden being too big to play third base.
I think the myth of "filling out" and losing defensive skill or utility is just a weird way for scouts to express that they don't think the player is good enough at a position to play there long term. Like, if you have a fringe capable shortstop, of course if he gains 20 pounds and loses a bit of range, you'll move him off the position.