I'd say give it a go, perhaps just one day a week at first, and see how you go on. Try it out at your house then at theirs and just see what works best for you.
I look after my DGS2 two days a week, 0700h to 1745h, and it's a joy. Of course there's only one of him and as of last month there are two of us as DH has just retired and is, for now at least, coming with me. Would you be on your own with the children? It's so much simpler to nip up two flights of stairs for a forgotten muslin, etc, when you don't have to carry a child with you. DGS2 adores his grandad and they have their own fun sessions out in the garden etc while I have a little nap rest.
We usually look after him at his home, though occasionally he comes to ours. There are different advantages to each. We are told very firmly that we are not expected to tidy up, etc, at DD2's, though of course we do a bit of it on the sly. That means we can just wave goodbye at the end of the day and go home to our tidy house. She also cooks for all three of us, bless her, so our lunch and tea are in the fridge ready for us just to heat up.
It's good to have him at ours sometimes though - we have a lot of toys here for him and can take him out to our local child friendly attractions for a change. And he can torment play with our cat, which he adores.
Good luck with your decision! It sounds as though you have a lovely family, so I'm sure things will work out whatever happens.