The NEC requirement for a phase converter is to set the overcurrent at not more than 125% of the "nameplate full load single phase input".
That's fine, except you don't know that rating.
The 40HP motor has an NEC rating full load current (3 phase) of 104A on 230V. You won't load it to full load as a converter, and so you won't draw the full load current.....
The 40 HP motor can supply about 25 to 30 HP worth of motors, with no single one bigger than 25 HP. Of that 30 HP, the phase converter supplies 1/3, or about 10 HP, since it produces output only for the 3rd (generated) leg.
Using that as a starting point, look at a 10 HP single phase motor. It draws (per NEC rating) 50A at 230V.
But, the no-load draw of a 40HP motor is larger than that of a 10 HP. Estimating the no-load draw of the 10 HP motor at about 35% of full load, it comes to about 18A. The idle of teh 40 HP, at around the same percentage, would then be about 35A.
Figuring the difference to be the actual load draw for 10 HP, then I would estimate the full load amperage of the nominal 40 HP motor used as a converter as about 35 + 32, or about 67A (35A no load, plus 50-18 = 32, total 67A). Then, 125% of that is about 85A. A 90A breaker may be the closest you find, so a 90A breaker is indicated.
In reality, the single phase no load input will be a bit more than the 3 phase no load input current, so it could easily be 40 to 45A. Then the same addition would give about 95A, or rounding to a breaker value, 100A.
This is very rough estimation, but the logic is defendable. If you know the no load single phase draw of the motor, you can use that instead of the no load estimate, and get a better number.
You probably can't get a better number unless you load the thing to a 10 HP motor output current and measure.