I am getting very mixed messages here. This blatant affair has been going on for four years at least. He has said it is all over numerous times and gone straight back to his mistress.
And now you are asking us for advice about what you should do, Should you tell the other woman's husband. Isn't that blindingly obvious?
I think this appeal to us is another stalling exercise, you are half hoping we will recommend, or someone will recommend that you do nothing, say nothing accept being humiliated daily by this unpleasant couple, instead of just packing your bag and walking out.
I think the first thing you need to do is sit down with yourself and have a good talk. Ask yourself, why you are so determined to accept this humiliation. Did you grow up in a home, where your mother/grandmother/aunts, meekly acquiesced with flagrantly unfaithful spouses, perhaps it was a deeply religious home where the breakdown of a marriage for any reason was totally abhorrent. Was someone crushed by emotional blackmail when they tried to leave. Is your husband's behaviour, typical of all your past relationships.
Whatever the reason, you need to find out what you want, what you really really want. Chuck out the unrealistic - your H repents his wicked ways, returns to you repentant and is faithful ever after. That is so definitely not going to happen.
Do you want to keep suffering? Are you terrified of being on your own for any reason, so stay for that reason? Whatever it is you need to sit down and talk honestly to yur self. Perhaps seek councillingto help you through.
As for telling the woman's husband. You do not need to tell him face-to-face. Write to him, email him, text him.
Currently, I think there is something stopping you leave - and it isn't that you are worried about the welfare of the woman's family. Find out what that is and your next step will not need advice from us.