I'm trying to see this from a slightly different perspective.
In principle I agree with all those who say you should just invite your DD and GC round and tell your DH that he can stay in and be polite or go out for the day.
But is this advice really something you can do? My DH is not a controlling person, and neither am I, but if I said something like that to him, or he to me, it would lead to an almighty row, which certainly would not improve matters.
So I understand why you would not want to follow the advice given.
However, from the little you have told us, I certainly find it hard to know what is really going on, but as you are still living with your husband, I assume you love him and are between a rock and a hard place.
Has your GD paid back the entire loan that you cautioned for?
If she has, perhaps you could ask your husband whether this being the case he is prepared to invite them into YOUR ( I mean your and his) home again? Explain to him how much you dislike not being able to see your children and grandchildren at home and that the entire situation upsets you.
Surely three years is time enough for him to forgive you for using your own money without consulting him?
If he is still adamant that he will not have them in the house, you will either have to go on seeing your family in their homes, or give your husband an ultimatum , always assuming that you are willing to consider leaving him, that is.