I believe in live and let live - but people who live lives that infringe on the happiness and peace of others need to be taken to task. If they are breaking the law as well, then it is even more important.
Green belt land is there for a purpose - to make life better for us all - to provide green space for aesthetic and practical reasons. Building on it is just not on.Wanting to get rid of an illegal, noisy eyesore is fully understandable.
Where I live there is no traveller problem, but sometimes there is a noise problem when people play music with a lot of bass. My husband always goes and politely asks them to turn it down, especially the bass. Most people comply. One lot got nasty, so he simply came back with our two big, soldier sons standing with him.
They complied. Sometimes old fashioned ways work best.
Those who make life unpleasant for others around them, clearly don't believe in 'live and let live', so they should not be surprised they get hostility.
I also remember the old gypsies. Apart from scary dogs, I don't remember any trouble at all. They came, they did some work, sold pegs and lucky heather, and moved on.