It’s not going to work, is it? You would be resentful and bored and I suspect that your DH would soon become bored and possibly start to vegetate if he has no interests other than that one hobby that gets him out of the house for a day here - is it a hobby he could easily do in Portugal - would language be a possible barrier?
As OP’s say, and you realise yourself, there’s a lot more involved when moving to another country to live than to just visit for a holiday : will your combined pensions and the sale from your house be enough to satisfy the requirements of the country your DH wishes to go to, and will it be enough to live on and cover all the necessary costs of being a resident there (tax, insurance, health bills, food, rental accommodation, hobbies, etc).
You need to make a list of all the pros and all the cons, then sit down with your DH and explain your reservations and think carefully through all the practicalities
If neither of you are fluent in Portuguese, then dealing with all the legal necessities to move there permanently are going to be difficult. Yes, there are interpreters, but it will cost you.
Would he be happy living there on his own, because my feeling from your post is that you don’t want to go and so won’t go, especially as it would be down to you to deal with all the practicalities.
Stick to your guns and refuse to go.