If it’s any consolation I think many of us prefer being the driver to the passenger.
When my car was off the road, kind friends picked me up to go for coffee, but I was shocked at the standard of driving and even basic lack of the Highway Code shown by a couple of them - I wouldn’t have liked a long drive with them. I was grateful of course, but even more so, to get my car back . I’m fine being a passenger with my BF and both DDs.
Do you actually have to go with other people (other than DH) so maybe turn down the invitations? However, in your position I wouldn’t let it take over - maybe a mild medication could help, like those given out to nervous fliers or dentist phonics. Easy to say, but don’t let it get the better of you.