That is not how it works.
Especially in the US.
The agreement may not be legal and or enforcable where You are, yes.
But You or anyone or any company can always be sued for any reason whatsoever, especially in the US.
So the other company can sue You, and drag on the proceedings, and force complex discovery etc. on You that You will need to pay for, pay to Your lawyers paralegals billing You 100$/hr each.
As long as they have money and are willing to spend it, they can increase the ticket size until YOU have to settle, lose, because You cannot afford to pay the paralegals 100$/hr x maybe 3 or maybe 2000$/day, for doing essentially nonsense clerical work.
In Finland, such a suit would be thrown out of court immediately, first hearing, and the other party would usually pay Your costs due to an obviously frivolous suit.
But the US is not Finland.
We employ 16 lawyers in corporate and international law and taxation, clients from 77 countries.
My wife is the founder whom I helped start the business.
Avalanding.es.
The US system is heavily geared towards moneyed interests able to support long very expensive legal proceedings.
The UK, somewhat similar.
The scandinavian countries, almost none.
A case like yours would be thrown out on it´s merits, first hearing, scandinavia.
And the other party would pay all your costs.
Spain, all over the place.
Nonsense claims get thrown out but politically powerful and rich lawyers and clients find workarounds.
Your lawyer was technically right, but possibly/probaly did not explain the situation fully.
That is how some of them get their clients, usually clients in desperate straits with short discovery timelines.
We don´t do any of that kind of business at all.
(details and anecdotes deleted.)
Suggest sending an email (written proof, valid in court) asking when or how a possible lawsuit could get thrown out in court based on the agreement being invalid.
Next, not in same email or day, asking for an example of such a court case decision.
If they ask for an extra 100$, pay it, on condition You get a satisfactory answer.
Make sure this is by email.
The results are likely to be eye-opening.
I had one like this once so I sent it to my lawyer....
He said go ahead and sign it as it is not enforceable or legal here.
And the BTW since it such I don't have to pay any attention to the non-compete part also.
Cost me $100 for 15 minutes but put my mind at ease and I knew where I stood.
Bob