I often holiday with my girlfriends or siblings. Several times a year we take short breaks without OH. He knew I was independent and took off with friends when I met him. he has his days out too, frequently, mostly involving sport. Can you encourage your husband to take up interests that might get him out of the house, or socialise with others, without you being there?
I really feel in this day and age we should not have men stifling our lives or being resentful if we want to do our own thing, occasionally.
Rambling Rosie, do tell him you are going, asap. Stand firm, but maybe offer to book a break with him, later on? See if you can get him to enthuse about joining things, on his own?
I like being close to OH, but I couldn't live either in his pocket, or being joined at the hip - or afraid of his reaction if I wanted to do something that didn't involve him.
Please go on your walking holiday.