What I tend to do is set the part up and find center. Put in a (small) center drill and move either X or Y (lock the one you don't move) 1/2 the width of the socket across the diagonals.
For a hex, the diagonal is 1/cos x w/2 = 1/2 the diagonal. (w is the width across the flats, or 9/16. w/2 = 9/32). In this case 9/32" x 1/cos30° = .3248"
I then use the center drill to pop a divot deep enough to start a 1/16" drill. Rotate the index 60° and make the next center pop, etc around to the 6th hole. Then change to a 1/16" drill and drill all the spots down to the expected depth.
Then I rotate the index 30° and keeping the 60° spacing on the sector arms, reset them to this new starting point. Now you are centered 1/2 way between 2 points & parallel to one flat. The table has to be moved back toward center until the endmill is _inside_ a line that would connect the 2 dots on the axis you locked. Don't cut too close to the line to start, there is a lot of deflection on small endmills.
Lock the axis that located the end mill inside and perpendicular to the line. Now use the axis that was first locked and start milling the flats, visually staying inside the line Once you've gone around the whole way, measure the width across flats and step out accordingly to finish size. The 1/16" holes will be about 1/32 deep in the corners, but they make it easy to file the corners straight after milling close with a 1/8" or 3/32 endmill. You can even put a square bit in drill chuck in the mill spindle and slot out the corners by manually operating the quill (spindle off, obvioulsy
) to shave out the excess in the corners.
If you need corners that don't have the slight round "over-run", just subtract the necessary amount from the size bit you use to define the corners, from the amount you step out at the beginning.
Here's an example of what the process looks like. I needed a 3/4" square collet, but offset to the center. IOW, not a stock square collet. So made it the way described above, this shows how the corners look. In case it is objectionable for your purpose, just subtract the bit width. For me, it makes it really easy to square the corners
slotting out the corners
smt