QRF, I hope you dont mind me shortening your name. If matters are as you say it seems to me that there may be a problem relationship within the paternal GP's household or even mental health issues.
Why have the GPs allowed their son to hoard stuff in the house to the extent that it can make it an unsafe environment for your child? Most parents would at some time say something like, 'Whats in your room and in the shed is your business, but the rest of the house and garden are to be kept clear and tidy' What kind of things is he hoarding? Hoarding in itself can be a sign of mental health problems. Sometimes those closely involved with someone like that get drawn into and complicit with the hoarder, usually from the best of reasons, like protecting someone they care about who they know is vulnerable. Occasionally because they are intimidated by the vulnerable adult. Often things get beyond their capacity to cope and they just become frozen into inactivity. Refusing to see a GC unles he visits their house could be a way of pretending everything at their home is OK because GC comes round regularly. Your decision not to visit is another pointer to them that their home is not 'normal'
I think you need first to think about the GPs household and how it works, taking my comments above into consideration. It seems what ever the problem is, it is not with you. Be open to the GPs keep them up to date with their GC's progress, telephone, email and send photos and make it clear that you would love to see them at your home or elsewhere, although not at their home and, if you can safely do so, explain gently why you do not visit.
You do not mention your child's father does he have any input or views on the subject?