The OP wasn't just referring to going for a coffee. She said "dinner, brunch, coffee etc."
We always pay when it is our suggestion that we go out for a meal. Even when it is our son's or daughter's suggestion, we usually (though not always) pay because we want to and because we have fewer financial commitments than they do. However, it is never taken for granted that we will bear the cost. Both always offer to pay all or a share of the cost of the meal.
I think to say "if you can afford it, just pay it" misses the point. It is rude to assume that anyone - be they friend or relative- is going to pay, even if they usually do and even if they are much better off than you are.
Emma4 Since it is your son and his wife who suggest these meals, it really is quite cheeky to assume you will be paying. I'd be inclined to say I'll make something at home. I know it's extra work but they might realise that you don't want to keep paying out all the time
Is it perhaps possible that your son and daughter in law have got used to being over-indulged by you and by her family also? You say your son has NEVER bought you so much as a cup of coffee, which I find astonishing. You also then seem to contradict yourself by saying you'd feel guilty if , as another poster suggested, you "forgot" your purse. Surely even if your son and daughter in law are hard up they could occasionally run to the cost of a hot drink and a piece of cake?