Crumbs - my kids didn't come on holiday with us any more at that age. They went off on their various camps and activities and we went our own way. No way they would come with us. At the most, we would join up for a few days' camping together in the area they were staying.
Whether he's sporty or not, I would agree that an activity holiday is the best idea. And YHA are perfect. You can all stay, possibly do different activites (e.g. he could go climbing or canoeing and you could go on a leisurely hike), and he gets to hang out with people of his own age. You don't have to worry about getting food, and the prices are reasonable.
But without any idea of your or his interests or the budget or knowing how far you are prepared to travel, it's hard to make any further suggestions.
Have a lovely time, whatever you decide.