Is the upkeep of the house too much in physical terms or financially, Butweam?
And do you really want a garden at all? Would an apartment with access to a communal garden to sit in, be OK? If you don't enjoy gardening, it can be a real downer, feeling that you have to get out and deal with things. Of course, if there is a small courtyard and you can plant containers, you may enjoy that. People enjoy watching birds and insects. Do you?
I think you need to decide first where (geographically) you want to live. Will you stay close to your present home, and friends? If so, what is available in the area, realistically?
You are young and if your health is good, this doesn't have to be planned as your last move. That allows you more freedom to experiment. You could, for instance, sell up and rent somewhere in a new area, or try a different size.
If you are finding the financial upkeep a challenge, now you only have one income, consider a new home which will be comfortable for you, but not have space for the family to visit. You could pay for them to stay somewhere when they do visit. That is much cheaper than having two extra bedrooms to buy and pay Council Tax on, for only a small amount of occupancy each year.
I have been steadily downsizing for 40 years, from 10 bedrooms to 1. Never regretted a downward move. I would say, don't rush.
Keep up the clearing out, that has to be done and gives you an idea of what you can't live without and need to have room for. Some people have masses of kitchen equipment, I have books.
Well done for making a start.
Lots of people on here have experience of this and there are threads which are worth reading to see what their experience has been.
If you are still working, and intend to keep on, that will influence what you do. If you are about to retire, then this will be your time to experiment and follow your dreams. Do work out the finances carefully, though. It would be heartbreaking to find that you couldn't afford to stay where you have just chosen to be.
Good luck!