I guess it's a case of "love me, love my family". Your partner is already a loving dad and grandad from his previous relationship. That is not going to stop suddenly because he has found a new relationship with you.
His son needs, and is entitled to his father's help and support at a difficult time. The little girl needs somewhere happy to be with her family at the weekends when she spends time with her dad. It just happens to be with you, at your house, that this happens.
This situation is almost bound to arise in later life new relationships. Your partner cannot be expected to cut his previous family ties. Neither should you expect him to. He has to be available whenever he is needed by them.
You are a grownup. You choose. How much do you want to continue with this partnership?
Estranged Son and Future Granddaughter
To think that London, or anywhere else for that matter, does not belong to any one demographic