it can matter whether the inverter is expected to deal with that when designed or not, but there isn't any inherent problem of blowing up the devices.
For example: I have what amounts to a 3 HP single phase VFD, actually a 2500W 120V inverter, in the shed out back. It runs off solar charged batteries.
I regularly run an air compressor and drill press from it, and other equipment such as angle grinder, hand corded drills, etc as needed. The normal machine controls are used, I just flip the device power switch when I want to turn on any device, and turn off the switch on the device when I am done. The inverter stays on as long as I am using tools.
The inverter is a direct output type, no transformer. This is really the same as one pole of a 3phase inverter, and this usage has caused no problems in 12 years of use, nor would I expect it to.
However, the inverter is designed for at least a 2:1 power ratio, IOW it can produce double power for short term. Some VFDs are like that, although 1.5x is possibly more common.
The switch is not the problem. The turn-on surge of the motor, which is essentially the locked rotor current, is the issue. It will cause an OC trip on the VFD if the inverter is not over-sized.