What was your husband's manner towards your children?
Did he leave them entirely to you?
If so, you could hardly expect him to want a little grandson around.
What exactly does he suggest you occupy yourself with, instead of looking after your grandson?
IMO he should be able to "put up with" you doing what you want for ONE DAY A WEEK!
I might have felt more sympathy for your husband, and so might you, if you had been providing day-care five days a week. So no, I do not think you are being unreasonable or inconsiderate in wanting to continue doing what you are doing,
However, if this tiredness, grumpiness , or whatever you want to call it, is a new trait in your husband, I do feel you should try to find out if there is anything else behind it than not wanting a little child around the place.
My husband has been "poorly" in the sense of being lethargic sleeping badly, having back-ache, eating poorly and catching every cold or tummy-upset that was going the rounds.
Finally, he made, or rather I made, with his consent, an appointment with a chiropracter, as we both knew this would at least relieve his back-ache, which it has.
It has also made an astonishing improvent in all the other issues.
So, don't just put up with a man of 68 feeling old and tired.
Pack him off to the doctor for a health check to start with, or sit him down to a frank discussion of what is really worrrying him.
You don't mention whether retirement is new for him - lots of men find the transition to retirement hard, nor what you do while your husband is in the garden or workshop on the days where you are not looking after the child.
On the face of it he sounds unreasonable, but if you are engrossed all the other days with hobbies, voluntary work or the like, he could be feeling that you have no tme for him any more.