Maybe we're missing the core point of this issue. I think someone's motive for using a particular word is actually the key, rather than the word itself.
The OP used the word "ugly" as she was looking for a quick way to describe the actor and some aspect of his appearance brought that word to mind. She could not have been using it to offend/hurt the actor as he would never hear about it.
I would suggest some possible motives, however, for the guest who called the OP ugly: he did not think he was being included in the conversation as much as he would like or thought others found him boring, so made a misguided attempt at humour to address this; he was jealous of the OP in some way, such as she has a good relationship with his wife or is generally more popular with others than he is; he has been attracted to the OP in the past and is trying to convince himself/his wife that this is no longer the case....
Misguided, low self-esteem, jealous, just a bit "ugly" on the inside... Maybe he is more to be pitied?
I would not resort to insulting/ignoring him in an unpleasant way unless he deliberately does something else to hurt you. It wouldn't make you feel any better and may be witnessed by others who could then brand you as unkind, which is probably more hurtful than being called ugly by that man at the dinner.
I know it sounds cliched but I have known people who looked attractive, but suddenly physically appeared much less so if they showed themselves to be selfish and unkind. I also worked with a lady who did not have particularly pretty features if you stopped to analyse her looks, but everyone perceived her as being attractive because she was so kind, encouraging and never spoke badly of others.
If I were you, I would give myself a wee treat of some sort to cheer me up and then focus on being as lovely ( inside and out) as I'm sure you've always been.