New to the site and first post so be gentle
It sounds to me, (picturing it mentally) that it could be a backlift problem. Obviously i've not watched you net so can only speculate, but make sure that your backlift is straight back towards your off stump and not towards gully/slips. The reason i feel it might be backlift related is you say its mainly your drives and forward defenses where your backlift is key.
If it is off towards slips then it will naturally angle in towards your pads, nothing beats standing infront of a mirror and practicing this.