Staying with religion for a moment, there are many who see the good face of religion and its bad face, and who can see that many of those who have gone overboard either for or against it see only the face they choose. Evangelists for either side push hard to get conversions, and assume that the "uncommitted" are either irrevocably ranged against them (so are "the enemy" to be fought) or are ignorant heathen barbarians who will be won over if they repeat their slogans at them often enough.
As the evangelists are unlikely to change their allegiance, it would be a waste of effort for the middle-of-the-road to argue with either, and their ears get bruised from the sermons or rants of believers sceptics - so they stay away in droves.
In case the analogy is unclear, that happens on the political threads too. The enthusiasts can say that it shouldn't, but that is the reality.