If you are only seeing your children once a week or a fortnight, I don't think that you are being unreasonable at all.
What would you be doing with your husband if you cut yourself off from your family altogether? The two of you can't do everything together, or you would feel suffocated, and may well find that you have nothing to talk about.
There is clearly a problem in the fact that your husband hasn't settled in your new area. This is a difficult one, as you both have a right to be happy, and unless there is a compromise (eg finding somewhere less busy but still near your family) then either one of you will be unhappy or you will have to consider whether it is worth staying together.
Have you discussed this properly? Not as an argument, but in the spirit of trying to find a way to move forward so that you are both happy?
I know we are only seeing your side of things, but (to me) your husband sounds as though he is being unreasonable by objecting to you doing something without him once a week or so, particularly as he was invited but chose not to go with you. Can he not amuse himself for that length of time? Is he jealous of the fact that you have family and he doesn't?
Maybe the way forward is to find him something to do on the days you spend with your family? To be honest, though, I would be surprised if he were prepared even to think about it, as I think the objection is a red herring. He seems very controlling (or possibly very needy), and if you move miles from your family you are going to have nobody but him, which could bring problems down the line - particularly if he puts the kibosh on your spending time with friends as well as family.
My advice (which is easier to give than to follow, I know) would be to insist on your right to see whomever you see fit, whilst at the same time genuinely looking for a solution that will let him see that you are taking his objection to the area seriously. If nothing else, it might concentrate his mind on what is the real issue (it is possible that he doesn't realise), and then you can look at that together, whatever it turns out to be.