Practical solutions from someone is the UK who has visitors from the US for longer than I’d like visits:
Plan your own days. What do you want to do whilst you are there - are there museums, national parks, tbeatres, galleries, etc that you’d like to go to? Are there places further afield you’d like to spend a few days? Tell them before you go next time that you plan to get out and about on X number of days, doing such and such, and would anyone care to join you for some/all of it? If you need a car for some of the activities, politely tell them you’ll either need to borrow theirs, or can someone give you rides. If not, budget for a shorter trip with cat hire (cheap in the US).
Speak to your son about what his and his wife’s expectations of your visits are, how they mesh with yours and what would work best for all of you. Maybe, if they have the money to pay for it, they and you would love to take the kids away for a couple of days? Or would it work better if you came for shorter trips, or maybe you should be alternating? I’m not suggesting these are the right discussion points - you will know that better than I can what you’d like to ask and suggest.