Willy,
I hear you, there are several things I do to get speeds, feeds, sfm, etc.
Get chart, call your tooling rep. or through some mud against the wall and see what sticks. I work 99% with cast iron so I am pretty familiar with settings.
Use the numbers from the chart or rep as a starting point, I always start conservatively and work from there. Familiarize your self with tool geometry, positive neg, neutral rake nose radius etc. It takes time but you will be able to obtain how cutting is going by sound, feel (placing your finger on tool holder; sensing for vibrations, sound as well.
In closing IMHO it is a mix of art, science, and your senses. Keep your setup as rigid as possible. I keep a note book of different speeds and feeds for turning and milling what works what does not. When milling large 200#+ pieces I brace with wood sometimes to mitigate chatter. Also don't be afraid to fine tune using your spd feed overrides. Each piece has it own natural resonance, at which it will shake rattle and roll and chatter itself to pieces just a little tweak on the over rides will get you into that sweet spot. Just the way I approach "dialing in".
Sure there are other folks here that have MORE knowledge that I will chime in with better ideas for you.
Doug