Getting cold feet

Long story - will try to give as much info.
Been with DP around 8 years but have lived apart. 5 children between us. My two DD’s have own properties. His 3 are a bit younger (early 20’s) and were either working abroad or planning to.
Then Covid hit.
His 3 went back to live with their mum local to us. One has since returned to work abroad.
Just before Covid my DD2 sold her property and I agreed her, her DP and my little GS would move in with me so that they could save to get a better property. It’s taken so long for everything to progress but they move in with me next week.
Before Xmas, DP’s son fell out with his mum and moved in with my DP.
All ok so far because we were both happy to help the children out.
Then, (with hindsight, probably not our best idea) after Xmas we viewed a property, and both fell in love with it. We have looked at properties before and find it hard to agree (more to do with location - this is perfect), and had our offer accepted.
Discussed the situation regarding the children and agreed that as their stays were temporary, we’d progress with the sale in the knowledge that we’d be alone by (hopefully!) the end of this year.
I went round last night (he’s my bubble as I am a lone household until next week!) and he said he had received a text from his ex saying that she was selling up and that middle daughter would have to come and live with us too
I didn’t feel I could say I was unhappy beings as my DD will be moving with us, but it kind of tipped me over the edge!
I’m quite perceptive and I know this DD would prefer I wasn’t on the scene. She’s said a few underhand things in the past which have upset me despite reassurance from DP. I also don’t think she has any plans to move back abroad and no means of financially supporting herself to live alone.
I didn’t sleep and had to tackle the issue with DP this morning
Naturally he was defensive of his children especially with the situation with my DD, and I totally get that. I know we will both have to make compromises and adjustments. But I don’t see mine being there long (if at all as there’s a slight chance they might move in with her in-laws after mine)
I see all three of his coming and going for years yet. It’s made me feel horrible because he’s such a sweet guy and never questioned the fact that my DD (etc) will be part of our household temporarily.
He said he was going to have to compromise by having a little one around with all that that entails which he knows he won’t find easy but has accepted that it’s part of the plan. Him and my GS get on fine but I guess living with a toddler will be a challenge.
The conversation didn’t turn into an argument and I left on good terms this morning, but I think he’s probably feeling quite hurt, as am I.
Do we pull out now and reconsider finding somewhere when the kids have all left home (we agreed we might have to house anyone of them temporarily in a future crisis - but hopefully not all at once!) and lose our dream house? Or plod on living in hope that they will all only be with us very temporarily?
My stomach has been churning all day. I don’t want to upset him as he’s such a nice, kind man and I love him loads. But equally, my feelings are valid aren’t they?
Sorry for the epic post