Just let them know in plenty of time if you decide not to go.
Two years ago we were invited to a lovely summer wedding, family but not close, early afternoon, with afternoon tea, but with hot food and dancing a lot later.
Since there were 2 other close family couples like dh and me, i.e. fairly oldies who weren’t bothered about dancing with probably very loud music, and particularly since we were all staying in the same hotel, we apologised in advance for the fact that we wouldn’t be staying for the later events (so they didn’t have to pay for us) and enjoyed a very nice restaurant meal together. Also it happened to be our sapphire anniversary. ?
The young couple were lovely, but I honestly don’t think they cared a hoot whether we stayed on or not - there were plenty of their much younger friends there.