I do not think people are objecting on any racist basis, but at the disproportion between the number of people being housed in the facility, which is many times the number of people in the local community.
I look back at the village community I lived in until a year ago, that had a past military site on the edge of the valley. If some government body announced that they were going to use the facilities to house say, 2,000 people, whether those people were 2,000, men or women or university students, or homeless people or nurses and that these people would be free to go where they wanted in a rural area with limited transport, then I would object because, however unobjectionable each individual may be. So many people of the same group with no money and nowhere to go. As the saying goes, 'the devil makes work for idle hands'