It's a difficult one! You're not joined at the hip, but I think, because of my experience, I personally feel that he has put his needs before yours. You are of course, both entitled to have time apart, but you say he also uses up some of his holiday playing golf, cricket and going on an annual walking holiday with his friends, so he already persues other interests besides the sking holiday. Perhaps he feels entitled, as he's worked all his life and while that's true, he's also overlooking your part in his life and should also be considering you.
I was/am in a similar, but not exactly the same situation. My DH suggested we move away from the area we lived for many years, where I grew up, had friends and family close by, to free up some cash (we downsized), so he could retire a bit earlier. We did this, moving to a small village, but he then had a bit of a mid life crisis, buying a motorbike and going out with friends most days, including a 'lads' holiday, leaving me on my own in a new area. It caused a lot of upset. He played it down, saying I was suspicious, but I knew it was totally out of character! After 18 months the novelty wore off and he stopped doing all that, but he's still out a lot of the time persuing other, more age appropriate interests! I have tried to find my own, but it's not always that easy and wish I'd never agreed to move. 
Sorry if I've put a spanner in the works, as most other replies are not to worry and allow him to have his time away, but you know your DH best and must know, deep down, whether he is taking advantage of your easy going nature. How would he react, do you think, if the boot was on the other foot? Try and do your own thing, if you can and see what his reaction is!