Once a week is a lot and more than you saw them before when they were a lot less busy than they are now.
Also I assume your DIL is on maternity leave, her mother is not just seeing her grandson, she may be primarily spending time with her daughter, and of course that entails spending time with the baby too but it's not just her visiting the baby.
Definitely don't drop in and try not to guilt your son with this. You really aren't hard done to and he doesn't need the stress. Guilt is not a pleasant feeling, I'm sure you want your son and DIL to enjoy your visits not dread them. Go, enjoy your time with them, don't overstay because that can be tricky with a baby when you are sleep-deprived, and don't make comments about "appointments" or quiz them about when the other grandparents have visited.
I think a lot of in-laws don't get that one reason many women are a lot more comfortable with their mothers in the precious but tiring baby months is that the relationship is solid enough for discord; by which I mean that they can be a bit snappy; tell them they are sick to death of hearing how they did x or y, - without it jeopardising the relationship. Even when we have a great relationship with our in-laws, it tends to be a bit more polite and it might be unwise to tell your MIL "FFS! Give it a rest will you?!" I mean my mother would have taken the hump at that too but we'd definitely get over it.
I could go for an afternoon sleep if my mum was with me but it would have felt rude when my in-laws were there. We tend to feel with have to put our best foot forward when we are with all but our very closest friends and family.
I'm sure you will develop a close and loving relationship with your grandson but at 4 months old he's going to be all about his mum anyway, with dad in seconds place. It will get better and better as long as you don't push it and lead to a strained relationship.