These days, going on holiday with another couple would be my idea of a nightmare! I wouldn't mind a weekend break, but not a main holiday. We used to do it years ago, in our younger days, and we'd put up with doing things that we sometimes didn't want to do, just to please others. One couple (still very good friends of ours, even after 40+ years), always wanted to befriend every stranger that spoke to us, and we ended up spending time with people that we didn't know, and had nothing in common with! I believe these friends are still the same now whenever they go on holiday. The only thing that got us out of going away with them was when we all started having children!
We did sometimes go away with a large group of friends, and all the children, just for short breaks over New Year, which was always a lot of fun, and the kids always enjoyed it. But, nowadays, DH and I would much prefer our own company so that we can please ourselves when we're on holiday.
PinkCosmos, if I were you, I'd have to have a serious discussion with DH about this. Tell him how you feel, and find out what his true feelings are. Going on holiday should be a pleasurable experience for both of you, and you shouldn't have to tag along with others if that's not what you want to do. If you and your DH enjoyed holidays alone together before this other couple latched onto you, then I think you should make a stand and insist that this time your holidaying without them! When we get older, life's too short to be doing something we don't want to do, just to please others!