if your old chuck is out then all you do is clean both the socket and the taper real well, then gently press the taper into the socket with the drill press's own quill. If the tapers are right it goes together alot easier than it comes apart.
that hole is probably for a drift wedge, you tap it in and it pops out the taper. instead of whailing on it cut a piece of metal into a 15 degree wedge and tap it gently in, the old taper will pop out. prying with a screw driver may or may not work.
YmmV but this is how I do it (mounting chuck to arbor).
I clean the male (usually Jacobs) taper with brake cleaner. I clean the female taper with brake cleaner. I make sure there are no burrs and use a gentle stone if there are, then repeat cleaning...usually new parts are pretty good.
I gently assemble the parts. I hold the chuck in my left hand and a 24 oz ball peen hammer in my right. I strike the sharpest blow I can tempered with zero chance of a mis-hit. Usually the hammerhead travels mostly vertical instead of a circumferential arc
They have never come apart before and the chuck does not experience any denting forces heavier than what my hand can backup.
If you want to mount chuck-arbor to the DP spindle, I'd clean both first and again check for burrs. Usually seating the chuck with a good "thunk" by vertical hand-generated-impact only will lock it nice and tight. You will need a mild-steel drill-drift to remove it.