Just go Gal. From what you already said you have no real social life with him.
Just tell him you are out at a party, you don't have to go into anymore detail unless he asks.
You say he is a bit of bully which might be why he is not asked. Is he actually likely to ask you for details?
Good luck!
A new definition of what constitutes British


