If most say, don't worry about it. I may get dual contact and just keep those for when running harder materials (steel,etc), thoughts?
I think non-dual contact holders work great for drills and taps.
The big advantage of dual-contact toolholders is in increased rigidity. That rigidity only comes into play with lateral loading on the end of the tool/toolholder assembly. Tools that go straight up and down do not load the tool laterally so the rigidity doesn't matter anywhere near as much as it does in, say, a long-reach milling application.
(Yes, there are some arguments about Z reproducibility too, but I have never had a substantial problem with this in real life, so....)