Found a couple of photos of a large-ish Madison reamer arbor on eBay. Notice the undercut in the slot. There's a matching thicker section on the reamer cutter, and that retains the cutter when withdrawing the tool from the hole.
While you can think of these as counterbore cutters, they are not intended for massive metal removal. I found some suggestions that the leading edge should have a taper of 1/32" or 1/16" at 45 or 30 degrees, depending on material hardness, and that taper represents the upper limit on how much the tools is expected to take off the radius.
The tool you show in your photos is obviously a slightly different style, but it's the same general type of tool.