I don't remember anyone "blaming immigrants for all the ills of society".
That is YOUR biased and rather nasty interpretation varian and a slur on those who voted to leave. It's kind of handy for Remainers to imply racism was at the heart of the leave vote. More superior virtue signalling!
Open borders was part of the concern but most sensible people, leavers and remainers, know that uncontrolled immigration, and mass unskilled immigration to the UK is a foolish, and given the times in which we live, a dangerous policy.
Immigration became part of every political party's agenda once the topic was out in the open, instead of being swept under the carpet as it had been for years and it was a discussion we needed to have and which was long overdue. That is not racist.