My thoughts are you have 2 issues going on here, contributing to the problem.
First is the RPC size. Most converters have a minimum size load motor you can run without overheating it. This is due to the amount of capacitance provided in the unit, being sized for the maximum converter load. With too much capacitance for a small motor, the voltage being produced will be high, driving the load motor magnetics into saturation. Saturation and high voltage, causes excessive currents.
Your highest voltage was stated at 275V. This should be around 260V maximum with the small load connected. The phases with the highest voltages will be the ones with the highest currents. The converter needs to be re-tuned for a smaller motor if you want to get the output voltages in line.
Second is the overload relay. As stated earlier, the motor is pulling over its rating, causing a trip. Helping to facilitate that trip quicker is the overload type. IEC type adjustable overloads normally have built in differential protection, to guard against single phasing. This means that when the current between lines is unequal, it causes the relay to trip faster than it would with a balanced overload.
Bottom line is you need to balance the output of the large RPC, in order to be able to run a small motor successfully, without killing it and the overload relay.
SAF Ω