I’m similarly frustrated by this limitation of Fantastical.
It’s a sufficiently frequent & high-value usecase (for me) that, in addition to Fantastical, I pay for a separate service (SavvyCal) to enable third parties to schedule meetings with me. Calendly would be another option, but SavvyCal includes extra functionality which I find useful, which is missing from Calendly.