It's easy to get things out of proportion when we are feeling a bit down, isn't it, and you are obviously depressed about this . Certainly the not holding hands thing seems ridiculous, but otherwise I think you are quite lucky actually. As other posters have pointed out, you see him far more than some grandparents see theirs.
My opinion will probably not be popular, but I honestly think some grandparents expect too much. W have raised our own children, and I doubt very much if most of us allowed our parents to dictate how we raised them.
It is wonderful to be a grandparent, an absolute bonus in life, but they are not our children. They should never be the be all and end all of our lives. Far better surely to make a life of our own with friends and activities and then regard our grandchildren as the icing on the cake, to be enjoyed as and when it suits the parents. I know there are exceptions to this, and some parents need child minding and baby sitting from the grandparents,- and that should never be taken for granted either,- but generally speaking just take a little step back and live your own life first. There is a whole world out there.
But I do sympathise, Isabelle, and hope it all works out satisfactorily so that you feel better. As some have suggested, maybe a frank and friendy discussion with your daughter is needed.