You've muddied the water a bit by saying you would go another time with them, unless that's what you genuinely feel. I'm not absolutely clear what your DH feels. Does he enjoy going with them, or was just being polite?
Either way, if he is prepared to accept its really important for you for it to be just the two of you, it's time to say something clear. "we've decided from now on we want to holiday alone", sorry to have upset you, but we've discussed this at length". If they have done it all by texts, I don't see why you should do the same btw.
However, if you do enjoy meeting them just for meals etc then you can say that as part of the gentle laying down of boundaries and even arrange one.
Other people have pointed the obvious out, they have gone on holidays without you both, but I think to mention that is just getting attritional. Yes you do risk losing a friendship, but truly it has just made you dread what is an expensive holiday!
(Btw, I'm a little wary of using the word "mental" casually, for real mental heath problems are serious and its a bit offensive to fling that label around when it's a matter of someone behaving in an aggressive or bullying manner.)