I may have a slightly different view point, depending on how you were in your younger lives.
If it were me, I would definitely take his feelings on board, because of the life I’ve had. So...I was a stay at home mum, who didn’t miss one minute of anything my children did. I was with them all the time until nursery/ school etc.
My husband worked, and sometimes went away. We had great family times.
Now I’m a grandparent to eight. I love them all dearly, but wouldn’t put them before my husband, unless it was some kind of emergency.
On the other hand, you may have to have worked, and not been able to spend the time you wanted with your children, so are now taking the chance to do it through a grandchild, which is understandable.
All I can say is, try to talk it through with your husband. Many people say time with grandchildren is precious....and it is, but in my opinion, not at the expense of each other.
All the best.