I can appreciate how upsetting this is for you but I also think that it is just a stage he's going through.
I would continue visiting as much as you can do, without "breaking" the bank and while there , I would not approach him or even look his way.
Just be there! If you see what I mean.
When my Grandson was one, I cared for him 3 full days a week when my Daughter went back to work.
Very often when my gorgeous Son in Law, who worked long hours, walked in, if he was on my knee, he would turn away from him and if he was playing, he would run back to me.
My Son in Law, who absolutely adores him, was heartbroken.
I would try to "cover" by saying, he'd just woke up, or something along those lines.
My lovely Daughter told me that, he also did the same when with her and she had seen her Husband walk into the kitchen with tears in his eyes.
It probably lasted about six months, then I got the idea of being very excited when his Dad's car pulled up, saying, "Hurray, Daddy's here," and eventually , he was jumping off my knee, running to the door, shouting Daddy, Daddy.
Perhaps you could ask your son to do something similar, the next time you go to visit.
I'm sure that it will work out though as he gets older.
Try not to worry about it too much.