My opinion, the OP was in a social setting as a solo with three couples. Would the git (my opinion) have made that comment if the OP had a big burly bloke with her? Probably not. He's a coward who bloody got away with his snotty little comment.
Thank the heavens, OP, that you don't have to live with him. I really hope I'd be able to say something if I saw him again, a simple touch on his arm and a gentle "you hurt my feelings, you know, when you called me ugly". Practice it at home, just those words so you are prepared. His response doesn't matter, it's about you "marking his card". Don't try to avoid him, don't cut off the friends, you did nothing wrong.
Calling an actor ugly, it does not signify. The response is the issue.