Silver 14, you are not alone. There are many paternal grandmothers who have had the same feelings at some point, myself included, so I completely understand how you feel.
Maternal grandmothers, who only have daughters, have no real idea what it's like being the grandparent who sometimes doesn't get to spend so much time with our GC, and feeling like you're missing out. And, as we know, some DiLs can be quite unfeeling when it comes to their in laws. I'm sure if things were the other way around, many of them would feel the differently.
I think we just have to accept it, and make the most of the time that we do get to spend with our lovely GC.