Very true.
Say a pitcher has a the best & fastest pickoff move ever, where no one can steal on him and any runner that gets on base would have to physically stand on the base otherwise he'd be picked off. That non-pitch skill (a pickoff) would factor in to how good the pitcher is. But by Grant's breakdown it wouldn't, because the pitcher is not in the act of delivering the ball to the plate.