Hi dollie, I completely understand where you are coming from as I was in the same position 18 months ago.
But I took the leap of faith and went for it, and can honestly say it is the best thing I have done in a long time!
I went from a large 3 bed house to a 1 bed flat (converted semi, so have upstairs and next door neighbours). To cap it all, I went from a town to a village, although only 10 min drive from anywhere. It was daunting, and a bit scary, but I am SO glad I did it.
And I actually found the 'downsizing' bit quite cathartic. I was amazed at how much I had accumulated over 15 years in the house, some was great to rediscover, but on the whole, I couldn't even tell you why I had it in the first place.
My neighbours are lovely, and although I can sometimes hear their telly (upstairs peeps) or them talking, it doesn't bother me, and, to be honest, I like it, because it means I'm not totally alone!
It sounds like you want to do this deep down, but are slightly worried by it too!
I'd say 'go for it', it sounds ideal as it's somewhere you know well already!
Good luck with the bidding, and I wish you well in your new home, whether it be this one or somewhere else. 