It's an indubitable fact that the older you were the more likely you were to vote Leave. Of course this doesn't mean that every person over 65 voted Leave, nor does it mean that every young person voted Remain; it's just more likely that they did, Also, tory voters were more likely to vote Leave than Labour voters. So if you put the two together, over 65 and vote tory then it's very likely that a certain amount of harking back to pre- EU days could be involved. In fact, I've had a number of Leave voting contemporaries tell me that we were fine before the EU and we'll be fine when we've left. It might not exactly be nostalgia for Empire but it's certainly not being forward looking and realistic about the modern world.
I do think it's time, though, that people in the public eye stopped saying things like this because it isn't at all helpful and, as mostlyharmless says, it just upsets lots of people who seem to take every similar pronouncement as a personal insult.