Going from a house to a flat is a very big transition, so I do think you need to think this over and discuss it without resentment with your wife,
Please do work out how you visualise your retirement: what are you going to do? A one bedroom flat does not only mean that your children and grandchildren cannot visit, it also means only one sitting room I should think, so if you want to watch TV and your wife doesn't or the other way around what happens? Do you see yourself trotting out to the (very small) kitchen to sit and read there, or what?
I do understand the financial aspect and I agree with you that paying off your mortgage is a good idea, but can you? What is the property market like where you live? Have you any hope of getting the kind of money in you need?
Talk to your bank and building society, and ask your wife what her dreams for this new part of your life are.
When my husband calmly asked whether we shouldn't sell our flat and buy a boat and go sailing round Europe for four years, I looked at him in horror and started to cry!
So I understand your wife as well as you. Now six years later, I am happy I agreed to what I thought a hare-brained scheme, so good luck to both of you!