Oh, that must be very hurtful.
I think this a stage a lot of children go through. They worship Mum and Dad, and heaven help anyone who they think is trying to take their place in even the smallest way.
He will come out of this when he gets older. My older grandson was a bit like this. He is 9 now and, although I know he has always loved me, he has behaved in just the way you describe. He has just, in the last few months, turned the corner and now is much more open and loving.
I don't think there is anything, really, you can do. Just carry on letting him know that you love him, without stressing it too much. Don't insist on a kiss goodbye or at bedtime. Just be casually nice to him. And wait for him to grow out of it.
Of course, so far as holding your hand when you are, that is quite a different matter. And he musn't be allowed to be rude to you. Both those things call for firm talking to by Mum and Dad.
HTH.