We looked after our grandson every day from him being 6 months old. I had just retired and our son's inlaws were younger than me so still working. We looked after him from 7 am to 7 pm when our son would pick him up, then we were asked to have him at weekends as well so they could go out but my husband who had started being ill had decided to take early retirement at 60 so we told them it was too much now as I was also looking after my husband as well as our grandson. We still looked after him during the week but because we had refused to have him at the weekend as well our son stopped talking to us for two years, but still dropped his son off at our house every single day without speaking to us. We did not want to lose our grandson so put up with it.
He only started to speak to us when he knew his dad was now very ill and by this time our grandson had started school but we (I) still looked after him during school holidays. Shortly after my husband died.
My advice would be, and I wish we had followed this advice, live your own lives, do what you want when you want, your son and daughter in law decided to have children yet, like our son, wanted someone else to look after them. Go on your holiday, if they don't like it, that is their problem, I know you say they both have medical issues but they should have thought this through when they decided to have children and not expect elderly parents to look after them.
If I had my time over again I would have said no, our retirement (to be honest we did not have a retirement as my husband was so ill) was taken over by childminding duties, don't let your lives be the same. Now our grandson is nearly 20, our son, now divorced, never does anything for me and has moved as far away as he possibly could, telling me that children should not be living on their parents doorstep.
To be honest, I could not care if I ever see him again.