I’ve not experienced this myself, but I imagine when you go into second and subsequent marriages, you have to take all the person’s life, not just the person. You’re bound to come with, for want of a better word, baggage. It’s only right you should both be accommodating the other’s. If one of you can’t do this, namely him, perhaps you shouldn’t be together. I wouldn’t normally advocate putting grown up children before yourselves per se, but there should be give and take, even if it’s your first marriages, and you share the children/ grandchildren. In your case, it’s different, because you both had separate lives before you met, and had already made your children. He’s being unreasonable by the sounds of it, and needs discussing. He knew the situation when he married you, now he’s got to deal with it.