I think you're overstating this. We accept the time it takes for a coach to send a signal to a 3rd base coach, who then sends a signal to a batter after every pitch (with runners on base)...but the catcher can't check his cheat sheet during that time?
We also give the batter time to step out, adjust his s***, collect himself and think about the patterns this pitcher has (which he just studied off an iPad before coming to the plate), but the pitcher/catcher shouldn't be given the same opportunity?
We have a pitch clock in place - whatever you do in that time should be up to you.