Paternal grandmother decided she wants grand son on weekends

The problem is my daughter is autistic, she was put on the spot , by her sons grandmother to have him at weekends, normally that would not be a problem, but she only comes round for his birthday and Christmas, and that is in the same month , then she expects him to go with her 2 months later , she did this last year , he went and I had to go with him as I am his main carer , he is struggling at the moment, he is being assessed for autism just like his mum , he said to me after her last visit , can you come with me , I said do you want to go , he said no , I was just wondering if he should go , he has nothing to do with his dad , he walks past him in the street , he held him once after he was born that was it .