Arguments with son and I don't know what to do?

Hi, I'm a mum to 3 children and and grandma to 3 and married for 42 years. Im in my early 60s.
I thought i was a decent mum and as a sahm, I spent most of my time looking after my family. But now im questioning myself that I have damaged my kids.
My son is the eldest and was the 1st grandchild too so he was given plenty of attention by all the family. He also was given everything and more, he was spoilt. He is very clever but never worked hard at school and has no qualifications. He worked steadily up to 2013 when the company folded. He has not had a proper job since then, mainly working when is suits him or needs money. He got married to his long term gf and they had a daughter but when she was 18months old they split, mainly due to money issues and his argumentative ways. He had no-where to live so came back to us.
This was the start of my relationship with him deteriorating badly, over 5 years and after many arguments and disagreements and him not contributing to the bills, I moved him into a house I had inherited rather than it be empty.
4 years on and husband and myself have decided to move into this property ready for our retirement( financial and downsizing) son has had plenty of notice that we intend to do this but still didn't sort his own living arrangements out. He very often borrows money from me and I get it back only for it to go again days later, he drinks and smokes often and yes I'm an idiot for being soft.
Work is now being done on this property and he was told he had 8 weeks to move out, again he's still there. There is very little by the way of comfort at this house while the work is being done so he stays at ours while he has his daughter for visits. Tonight he turned up and after about 10 mins launched into me, that i interfere in all my kids lives and that they are sick of me, this was after I told him not to untidy his sisters room as she is away on holiday and id straightened it for valuation for selling. I mentioned that as he was staying over so he could spend time with his daughter, he should have some respect. That didn't go down well. He waited till my gd had been dropped off then decided to go to the other property and before he left told me that I was to stop being this person and that I had to back off from gd as the damage I had caused him was not going to happen to her....i asked him what he meant and then I could not do it and he said just said I had to stop. Only thing I can think of is that sometimes when im looking after gd she asks me questions and i answer her age appropriate, or that I squeezed her shoulders recently and she got upset, her mother was there and I said id never seen her react like that before but obviously wouldn't do it again. I understand her boundaries and think her mum has mentioned this to my son and this has caused tonight's issue. Did I do similar to him as a child and didn't see him upset, im mortified that I could have caused him issues.
My husband is sick of him behaving this way with the houses and the way he speaks to me and his lack of providing for himself and his daughter but he knows I can't cope with my children being in need. Hes told me numerous times that I'm at fault for being a pushover, Im stupid I know!
My daughters are doing OK and managing their lives, and they disagree with how their brother behaves.
What do i do, he's going to be homeless and I'm going to feel dreadful, how do I stop being so pathetic. 😢