I think if they invite themselves, well that's what's commonly known on MN as cheeky f*****y. Many an entertaining thread over there along the lines of "we live in a beautiful part of the country, we have people who invite themselves".....and some more alarming ones of "they've come to stay and now we're worried they'll never leave" ..........no! you don't need it, we have a few friends where it's become a sort of set in stone yearly fixture on top of family, you can't say no to the latter! but I'm trying to cut back on some it can be hard work, preparing the house, the guest room, cooking etc.
You need to be up front, things don't stay the same, what might have been ok for you 20 years, 10 years, 5 years ago is now taking it's toll........just say "I don't want to do this anymore, for all the reasons you've given..... I'm too tired, get a hotel nearby and come for lunch"
You're not selfish, do they reciprocate in any way I'm wondering? meals out, gifts or reciprocal invites to stay with them.