Ellastone, I think you did the right thing. I have often found it difficult to jump in and stop someone saying something offensive! And, as you have described it, it sounds specifically unpleasant, not voicing a general fear about travel and attacks.
You have obviously got lovely neighbours, and no, it is not your fault.
I think how to proceed depends on the nature of the club and Mr. Bigot's place in it. Is there a meeting where you could ask for an agenda item about behaviour at gatherings? Could you or the chair write a letter? I do think that you have a right to express both your displeasure and embarrassment.
What I would advise is swift action or no action. It is fine to say that you kept quiet at the party, not wanting to inflame a situation, but now I think you do need to raise it.
Difficult if say, next year, they ask if you can host again, and you have to say ' not if HE 's there!
Something that has helped me in similar situations, is to think ' why should I feel awkward? It is he who was out of line!
Good luck