It seems to me that generally one set of grandparents always has more input into grandchildren than the other. I have 3 children and in our case we're the less contact GPs for all of them. Only one DD lives close and her MiL is besotted with our DGD (her only grandchild) and looks after her on a regular basis. I don't begrudge her this remembering how things were with my grandparents. I lived next door to one set. Took them for granted, they were always there. But hardly ever saw the others (my DM didn't like them). To me though they were special, seeing them was a treat. I notice my DGD is the same. She gets sooo excited when she comes to us (4/5 times a year) but the other GPs, although she loves them, aren't as special.
Our children are grown ups now and lead busy lives and need their independence. They need their time with their children too. I know families who have Sunday lunch with alternate parents every week. They can't go to a village fete or the beach as it would cause upset. I'd hate my children to have that tie and feel that resentment.
My DMiL advised me to 'hold the reins lightly'. Enjoy the time you do get with your family, don't make them feel obliged, and they'll relax and come to see you willingly and happily (although maybe not so often!). It'll give you time to keep your independence too so you're a more interesting person to visit.