Scarlett1 it is very hard not being able to see your grandson in your own home, but I'm inclined to agree with those who have pointed out the dangers of small children being subjected to the toxic chemicals left by smokers.
Ex SIL left DD3 when she was pregnant but after their son was born insisted on frequently taking him (newborn) to his mother's house where his father and brother smoked.....heavily. DD asked him not to do this for obvious reasons but he went ahead anyway. When DGS returned, instead of that lovely new baby smell he stank of cigarette smoke. It began to alienate DD from the baby, as well as her breast feeding being disrupted (ex SIL's mother insisted on giving him formula). He and his brother's hair and clothing still smell of cigarette smoke after every visit.
DD's concern was brushed off as her being too fussy, but for 2 or 3 years the little boy had the most horrendous chest infections causing him to be admitted to hospital on one occasion when he stopped breathing.
While I sympathise with your sadness, I really do think that you and your DH need to respect your son's concern for the baby's welfare, and make other arrangement to see the little one