No need to accuse me of not understanding! I do understand that people are fearful, but I honestly think there's a great deal of paranoia. Many of the places which voted Leave, which people have said are full of 'fearful' people, have very low numbers of immigrants. Essex is one of those places. South Essex has been a hotbed of right-wing populism for decades, even voting for BNP councillors at one stage, but there aren't that many immigrants in Essex, especially in the coastal towns. There is no rational reason to be fearful.
I do understand people in old industrial towns, where small enclaves have been taken over by (mainly Asian) immigrants. I expect there's a certain amount of racism, but I can understand how people (especially the elderly) feel when a whole area no longer seems like home. I can also understand places like Boston in Lincolnshire, where the number of low-skilled agricultural jobs has meant a huge influx of Eastern European immigrants. The same thing happened in the past in towns such as Thetford, Hoddesdon and parts of Bedford (all in the East). Immigrants aren't actually taking jobs, because the locals weren't taking the jobs.
I'm going to very non-PC here and suggest it's a working class thing. Areas where there are much higher numbers of immigrants, but educated/professional immigrants (such as Cambridge and London), don't seem to have the same attitude. My suspicion is that immigrants are scapegoated for the woes of the working classes, who have been neglected by successive governments.