I bet it's badly mushroomed. I know they didn't come out when I had the nut off. Actually, if I remember right, the set screw was in the bottom of the gear box and the brass was still in place.Actually this is the best method and way better than double nuts for precision bearings.
Is that brass piece free to move or has it gotten mushroomed on the top side?
I'm going to make a couple new slugs and then pull the cover off again. I'm anticipating drill bits and easy outs to remove the old ones. I still don't think they should be backing off the way it is, but I'm going to start with the easy stuff.