In your place I would try to sort out the relationship with the mother of your older grandchildren who are missing you so badly.
Initially, you will have to mention that you realise that you have hurt her by seeing the other grandchild, but once that apology has been made and accepted, please do not refer to him again. In your place, I would seriously consider seeing him, if his mother is agreeable that is, as discretely as possible and never speaking of him to your other grandchildren or their parents.
This probably sounds two-faced and dishonest, but I cannot see that you have any other alternative unless you choose between your grandchildren, which I quite understand you do not want to do, or feel would be right.
However, if you irrevocably offend the mother of your elder grandchildren, you will not be able to see them at all, will you? So do please try to sort out your relationship with her for your own, the grandchildren and your son and her sakes.