Tired husband

Hi, I have been looking after our grandchild once a week since he was a baby. We also have had him the odd weekend, and for almost a week when his mum had surgery. I do all if the care. His grandad does play with him and they have a good relationship, but he always moans after about the mess and upheaval. He is 68 and I am 66. My husband spends a lot of time in his workshop or the garden even on days that we don't have our grandson. However, he feels he wants peace and quiet and wants me to stop having our grandson, saying it is too much for him. I love having our grandson and he loves to come to us. He starts school in Sept. Am I inconsiderate if I carry in having him?