Not unlogical that a two-flute would do that on a plunge. likely one flute is high and catching and that would pull the EM off-center until both flutes are working.
I used to put more primary clearance on plunge use end mills, but that would not help your problem. A slower feed at the very start might help.
You might look at the end sharpening and see the tooth that may be high.
.oo2 would be enough to make the EM wonder at the start of a plunge.
Using the shortest EM the job will allow might help.