Very insensitive of him. I bet there was a delicious smell coming from the kitchen. He must have a tin nose.
If he does it again, serve up your meal and apologise for getting on with it while it is still hot. If there is not enough to include him, and he wouldn't want it anyway, apologise again and offer him some buttered toast. You may find it difficult, but you are doing it as a demonstration that it is your mealtime, not a time to devote to him. If he wants something, he can talk to you over the plates. He is the one who turned up at an inconvenient time, not you.
Say what you said to us - that you expected him in the afternoon - and that you had decided he wasn't coming. Say it matter-of-factly, not as an accusation, but say it . Some people are just not aware that others believe them, that when they say the afternoon they mean just that - the afternoon - not mealtime. They need it spelt out.