My own feeling is that many people who voted to leave the EU had an anti-immigrant agenda, but certainly not all. My husband was a Brexiter and he is most definitely not racist.
Some people feel that the EU itself could be accused of a degree of racism in that it is, in the main, a rich European man's club which left other nations out in the cold and which didn't support its own members when they hit troubled times (eg Greece).
I did vote to remain because, on balance, I did not believe that, at least with the current government, it would be wise to cut ourselves adrift from Europe and possibly have to rely on countries like the US, Saudi, and China for trade. I was also very worried about the remarks being made about getting rid of "red tape" - which I believe can have some very dangerous consequences.
I think it is unhelpful when either "side" attempts to ridicule or demonise the other. We need to try to find some common ground and understand the other's point of view.