If I had got something like that, I would have sent back something saying that I have done my best to involve them (i.e., inviting them to visit, even though they chose not to, etc). I'd point out they had never invited me, although I would have accepted invitations. I would let them know that they are welcome to come and spend time with the family in order to build a relationship with their GC, but that they would not be having visitation with my young children until the children knew them and were comfortable and wanting that themselves. I would say that if they wanted to visit, maybe we could organise dinner/meet up in the park. Please let me know.
That would be what is appropriate in my own situation anyway, so just some thoughts.