The Raps just don't have good defenders on the team. Derozan, Ross and even Lowry give up a ton of straight line drives. Carroll used to be good but since the injuries he's been pretty hit or miss. Joseph is really the only good perimeter defender on the team so when has a bad game it really comes out.
And once you penetrate JV isn't good enough to protect the rim.