Hmmm... It sounds like they're aiming for a brief visit in a public place, anticipating/fearing drama. While I get that, given the bad history, I also agree that it's as if they are treating you two as an "afterthought."
Regardless, IMO, you are right to follow DH's lead in this. If he doesn't feel comfortable w/ this idea, then there's no point in meeting them.
I've got another 'keen'... Ouch!