I wouldn’t do it. As you feel anxious at the idea, you need to say no. They are being unreasonable.
When mine were little, I would only have one grandchild at a time, when she was 2, 3, 4. I had them both just the once, aged 5 and 2 1/2, and it was fine, but they were older than yours are. I was the one who fed them, played with them, took them out, but DH was there as back up, and he would keep an eye on them whilst I cooked a meal. It was tiring, but they were old enough to understand and not feel abandoned by their parents. I don’t think it is a good idea at this young age.