A few years ago I was complaining about having to go back to work after a lovely long holiday. I thought it was quite a normal thing to do. I did like my job.
My ex asked why I didn't 'pack it in,' I would have been in my late fifties then and had worked all my life.
I then asked when he was going to retire, as he was in his early-sixties by then and finding running his business very stressful. He didn't reply- not for years.
Our relationship wasn't in a good place and due to his unwillingness to even go part-time, I kept my job and left him, only to find out he retired not long afterwards.
He just didn't want to be with me and said so in the end.
I think you need to communicate your anxieties with your DH, find more things to do yourself, even carry on working if you have that choice. Let him make the decisions about what he wants from his own life, and you do the same.