I'm really curious what the "slow" button is unless it's a jogging button to 'bump" the spindle. With the other stuff you mentioned, there has to be some sort of motor control box somewhere, probably down with the motor.
If you go with a VFD, you cannot have switches/controls between the VFD and the motor. However, a good VFD (I like Hitachi) can be programmed to do forward/reverse. Speed control can be handled by knob on the VFD or a remote potentiometer. A sensorless vector type VFD will retain more torque at lower speeds. On other types, torque falls off pretty much directly with speed.
I just set up my vertical mill with a remotely mounted VFD (nice washdown rated box from Surplus sales up on the wall) and ran multiconductor wire down to a control box at the mill. Forward/rev, speed, and a BRB (big red button) in case things go awry. Also put in one of those digital tachometers you can get for under 100 bux. The control box is a 6x6 plastic box I found at the big orange box, it has a rubber gasket seal to keep the insides clean.
You could conceivably use your existing start/stop/reverse switches with a VFD if you wired them to control the VFD rather than the motor.
The mill has step pulleys I use for major speed changes, but I don't use them near as much now that I have speed control in a dial. If I wanted to switch from say a 1/4" EM in aluminum (very fast) to a 3/4" in steel (much slower), I'd use the pulleys and fine tune with the VFD.
A rotary converter would let you just plug your lathe (or other three phase machine) in and go with the original controls. A VFD really just wants to be wired to one machine.
As far as size, it is to be matched to the motor rating. You need the info off the data plate on the motor for either option really.