I, too would just keep it simple and say thanks for the invitation but we don’t think we’re up to it at the moment, but hope that you all have a lovely time.
We have to do something very similar sometimes. It gets easier, and have found that people do accept it and don’t take offence.
Waiting for memory clinic referral
Daughter not allowed to know new baby’s name until she visits him
