Is it fair that you want him to stay overnight in a house where there is something that could cause him physical distress, and potentially have a serious anaphylactic reaction? If you have never seen anyone in one of those reactions, maybe you don't know how serious it is - their nose and throat swell up so that they cannot breathe, and they can die without instant medical attention. Do you really want that to happen to your grandson?
An allergy is a very real condition, it isn't just a fad of his mother's. Does he have an epipen? Can he use it himself? Do you know how to use it?
Does he react to traces of the pets on your clothes and hair if you meet him outside, after you have taken a shower and washed your hair, and put on clean clothes? If he is Ok with that, and you keep to meeting him away from the actual pets, you can see him and have fun without putting him into danger. Work with his mother on this, not against her - it is your grandson's health and life you are dealing with.