@Gran16 - most of my immediate thoughts have already been said by others. It does sound as though you have reached a point in your life where you need to break away from being unhappy. Nobody has a right to make you feel constantly bad about yourself, to put you down, manipulate you, make you doubt yourself (especially planting a big doubt around your memory and mental health). If there is a high degree of emotional abuse and possible overtones of sexual abuse, I do wonder whether physical and financial abuse will follow if they are not already present. He should not be implying you are developing amnesia/dementia as that is cruel. Has he always been like this or is this sort of behaviour getting worse as time goes on? Is there any possibility he could be ill, suffering mental health problems? Or is he just being a selfish, controlling git? Knowing how these bad traits can become habit, can escalate, if your instincts and gut-feeling are to get out before he gets even worse, before you are unable to make the changes, are more trapped etc, then start making future plans for yourself. I'm curious as to why his previous relationship ended with 3, presumably quite young, children - was he a bad husband/father there too?
You say you own your home - I would advise you to ensure your home and any finances, investments, savings etc are protected and out of his reach. If he has not made provable financial contributions, any claim he tries to make on you shouldn't be very strong.
You have been together a long time. If friends have reassured you that your memory is fine maybe you could have a heart-to-heart type chat with them if you trust them to keep your confidence. Sometimes people outside of the relationship notice behaviours that they refrain from mentioning as feel it is none of their business, they don't want to interfere or make their friend feel bad in any way. Perhaps someone in your circle of friends will echo the types of problems you have highlighted e.g. notice you are wary of challenging him, hear him put you down, think he manipulates you or the situation to his own advantage, plays on your good nature and treats you like a personal servant, butler, housekeeper and taxi driver. (I don't drive and am reliant on my better half for lifts if I need to go anywhere, but I would never take it for granted, and always check with him before confirming anything like hospital appointments).
Gransnet can be a good place to find support while you are going through tough times or a period of change. I would say have some quiet time to think everything through and get it clearer in your own mind. Then, make some decisions that are right for you, that offer you the chance of happiness and peace of mind for your future. Most couples have the odd things they disagree on, or have to compromise on, but any relationship where one person constantly has their own way and doesn't care how much the other sacrifices to ensure their happiness becomes unhealthy. Please don't allow him to literally drive you insane - take advice if you need it, continue with counselling if it is helping, Google any terms you would like further clarification on and search the Internet for any helpful information sites, support agencies, local groups, online forums etc. Maybe you could post every now and then with an update, whatever you decide to do about your current situation. I hope one way or another that your future will be happier, that you will feel safer, more confident. It can be a lonely place living in an unhappy relationship/family situation - sometimes even living alone is less lonely! Maybe it is time to put yourself first, consider your needs, your head and heart need a place of contentment, you need to feel safe in your own life. I wish you all the best