I believe you BluebellBlue- if only because I have been in a similar position myself but for very different reasons.
Lets just say I find it very difficult to trust anyone, even counsellors, GPs etc as they don't always listen. It is like being trapped in a world where everyone is lying and you feel like you are going mad, because you can still see where the truth is...
You are using an inclusive pronoun 'we' so I am taking it there is someone else involved- maybe DH for example?
What is this persons take on the situation- do they want to move too, or is it just you that is being targeted?
Do they listen and agree with you, or just brush it off.
I think at least having someone else to talk these things through with should be some help? I do have family and friends to offload on but I sometimes wonder if they truly believe me.
I can imagine the sort of community where this may happen, so moving may be one solution but it does seem extreme to me.
They have no right to 'drive' you out. I would work on your self-esteem- it shouldnt really matter what other people say about you. There will be someone who will understand, somewhere, you just have to find some allies, even outside the community.
I wish you well.