I would look into getting a stub 3-flute carbide endmill ground flat as a finishing tool, it would be much more stable than the 2-flute you're using now. I'd keep the dwell as short as possible, but still ensure at least a full revolution occurred during it. Mount it in a short, rigid holder for best effect.
I think the point about the burnisher being used for too long is a good one, but it's worth some careful experimenting on some scrap 316L before you commit. You can work harden 316 very quickly, and going beyond that surface hardening "ideal time" may lead to fatigue failure and spalling of the upper surface.