Stick a carbide tipped bar in and bore it, up past the actual engagement length/area.
If you bore it on a lathe, the relief is not a precision tolerance, so holding the solid register end in a collet or dialing in on a 4 jaw chuck is close enough for axial alignment even if the project collet projects a bit.
If you bore it in the mill with the bar in a V-block in the vise, put aluminum shims, or short pieces of aluminum TIG rod in each slit so the collet can be closed up tight in the spindle taper to prevent rattle and chatter. (Obviously?) use sensitive manual feed, or power down to a stop. If the length of the bore needs increased, perhaps use a length of allthread & nut for the drawbar so as not to drill off the end of the original. You can decide later whether to reduce the originals length, or not.
There's nothing sacred about collets.
smt