I'm going to chime in here, I been machining annealed 4140 and was having trouble getting a good finish as well until I found the following:
Chipbreaker geometry matters. Look for something like this:
I used that very insert and was able to get a very good finish at 1000+rpm, and low (about...