That stings. Do you "mark" the hole location with an awl or other pointed tool? That usually prevents the drill bit "walking."