Well, he's certainly been given a free rein in his nastiness and, sadly, his Wife - instead of working on forgiveness and maybe eventual reconciliation, with lots of changes to be made (!!) , begged and pleaded for him to come home, presumably all on his terms. After an affair with her friend? After all that betrayal? And now she's afraid that if she crosses him in any way, she'll lose him. Just what he shouldn't have now. All that weakness means that she won't be much of an ally for you. As others have said, keep a relationship with her but don't count on it as being helpful. What it will do hopefully is allow you to maintain contact with your GC's. Show them your love and offer them the stability which they probably don't get from someone behaving like him.
He certainly sounds like the worst arrogant bully. He could be very unhappy but he's not going to share that with you. Or, he could just be seeing how far he can push all of you, feeling sure that you'll take that and more because you're 'family', trying to punish the wrong people for his own mistakes or because he's made bad decisions. Or maybe he feels that he just cannot end his relationship with his Wife and he's trapped and hates himself for it, so in his anger and frustration is turning on you.
Let it all go, or it will consume you because it's toxic and you'll end up feeling that there is something wrong with you. People like that are always so surprised when we suddenly decide stop, no more, and we mean it. You must be so sad and there's no one who really knows him, presumably, who will be strong with you or for you. But you seriously have to look after yourself, and not give him any more opportunity to abuse or bully you. No need to row, just don't respond and walk away or end the conversation. Try not to let him see you upset, just look puzzled, shrug your shoulders and leave him wondering what you feel. Once you stop playing his game, maybe he'll change, maybe he'll be different, lead a different life, maybe he'll get help, but that's down to him to work out. Think about yourself now and don't accept any responsibility for his attitude towards you Stay safe and be strong.