The father does have rights. He needs to make the decision himself as to how far he wants to take this - custody?/greater access?
Your statement that " they seem happy and definitely aren't abused" is a critical one. If that truly is the case, then SSD is unlikely to take any steps that might remove them from a parent's care - they prioritise keeping children with their natural parents wherever this might be possible. And they can offer help, as when has pointed out. Children in this situation benefit enormously from good nursery placements, and these have the great advantage that another eye is on the situation and the children's well-being.
I think it is important to make an assessment of how far the neglect is endangering their health - enough to take steps via the authorities?
One very important factor here is that you are still in contact with the children and are clearly a lynch pin in their lives, where they receive good food, clean clothes and above all else love and attention. If they are happy and not abused, there is something to be said for retaining the status quo, because, however much your son may have rights, a mother who wishes to make waves can cause a lot of disruption in a child's life by just being awkward and uncooperative over access.
It is interesting that she still goes along with you having the children every weekend, and it would be tempting to assume that this is because it leaves her free to pursue her own life free of the children. So she has an incentive to keep this going - and it needs to be maintained as a stable feature in their lives.
The key question is how far they are at risk and exactly what is it that they are at risk of? Enough to aim towards their removal from their mother's care? In my experience children love the worst of parents and can be hugely damaged by being received into care or having their parental contact radically changed. Better that some sort of support should be found to help mother make a better go of parenting.
It is a difficult dilemma for you all but if they seem happy then that does need to enter into your deliberations even if you dislike their lack of physical cleanliness. You do not suggest that they are undernourished, which is also a key question to be asking.
It may possibly be that at the moment they have a reasonable balance - the presence of their mother and their father in different amounts and different places, and the positive influence that you are able to offer to redress mother's shortcomings.
But the involvement of a health visitor could be a good way forward - they are often very experienced, tactful and supportive - so a word in the ear of the local HV could be a good move. She could perhaps drop in while the children are with you and casually run an eye over them.
It must all be a big worry for you, and feelings run very high in these situations. But it is seldom completely black and white: mum bad, dad good, although it is inevitable that it will be easier for you to see your son's side of things.
Kate Garroway-Care at home costs