I feel for you, but I can see the other side so easily. When my mother was alive I used to travel with my children, a fair distance, to stay with her for a week or two, a couple of times a year. We would have a lovely time, go out together shopping, or just visiting places. We would visit the rest of the family and they would come to Mum's too.
But sometimes, my siblings and I just wanted to have a night out - or in with a takeaway - on our own. My mother recognised this and happily stayed in to look after my children.
So please don't be too upset by not being included - though I do think it wouldn't have hurt them to actually ask you if you minded staying in with the children. To simply assume you were nothing more than the resident babysitter was a little