My OH also loves trifle at Xmas, and it seems mean spirited to deprive him, although he is the only one who eats it! He doesn't help make the trifle, but he heroically peels and chops veges for huge numbers!
I use his mother's method - no jelly as described above - and pipe the cream on as decoration.I make a smallish trifle (I'm told it's even better the next day!) and he loves it.
The kids love Panettone, and we make custard to serve with it. If I am in the mood, I buy a small individual pudding, otherwise I enjoy the Panettone (which I make or buy as I feel like it)
And then, later in the day, I eat my absolute favourite - Xmas cake with a slice of Wensleydale - neither of which OH likes - that I think is one of the lovely parts of Xmas - saying to everyone 'what do you want?' Only a problem if money is very short, or people don't pull their weight.