If you were batting and a bowler who had been bowling right-arm over for a few overs suddenly decided he would run in and bowl left-arm over (without telling you or the umpire) which then bowled you I'm sure you wouldn't be happy. It's exactly the same with the switch-hit, if you take stance as a right hander you shouldn't be able to decide half way through a bowler's delivery stride you want to take stance as a left hander.
The 'leg-spinners are allowed to bowl googlies' argument isn't valid either, there's nothing in the rule book that says a leg-spinner has to spin the ball away from a right handed batsman, whereas someone who says they're going to bowl right-arm over HAS to bowl right-arm over.