amma it's hard to work this one out without being a fly on the wall. It could be what you fear or it could simply be the age he's at. My eldest grandson is now 6. He went though a stage of not wanting to come to our house and then it reversed to not wanting to go back to his own house.
He was about the same age as your grandson when it started. He would cling to his mother or if going home time, cling to me and break his heart. I even began to wonder what was going on at home and doubtless his parents wondered what was happening at my house. It was just a form of separation anxiety and he grew out if it.
Think some mediation is called for here. His mother might be just as worried by his reaction to your son. Please try to get help and mediation as both patents need to put the child before themselves.
To think that London, or anywhere else for that matter, does not belong to any one demographic