I was the same before I took the plunge. I don't regret it, though.
I am living on savings and a very small pension, so it's a big drop from what I was used to, but I don't notice very often, as everything changes at once. I don't have commuting charges, and things like cups of coffee, lunches and work clothes really add up. One thing that I both miss and save money on is going out after work. I used to do that a lot, and in some ways I do miss it, as I live in a small town and worked in a city. I could make the effort to go into the city for nights out, but there rarely seems much point. On the other hand, it saves a lot of money, so it's swings and roundabouts.
What is interesting is that I am not in touch with many ex-colleagues. I thought I would have stayed close to a few of them, but we have drifted apart (largely because I don't go for nights out these days), which shows that without the common bond of the workplace there wasn't as much substance to the friendships as I'd thought. The other side of that, though, is that I am closer to friends from home.
I have never got bored easily, and have built on old interests and picked up a couple of new ones. My husband retired before I did, and we do some things together, but still keep our own separate interests, so we always have plenty to talk about.
The only nagging feeling I have is that if things took a turn for the worse financially, I wouldn't be able to earn money, and not having the security of a decent salary is a bit unsettling sometimes. I don't think about it often, but the Covid situation and talk of cuts in pensions and hikes in taxes makes me worry a bit, as for the first time in my life I am at the mercy of fate and politicians, which is not a reassuring thought, and low interest rates mean that my savings are diminishing faster than I'd hoped. If you have a better pension than me (which is likely!) though, that may not be an issue for you.
I'm pleased I left work when I was young enough to enjoy it (I was 58), as I have had a few health issues since then, and those, plus the Covid situation have made me realise that we never know what's round the corner.
If you want to do it, my advice is to do it now. It will never be the perfect time.