Abbey, the OP has a well off neighbour who is deliberately not taking responsibility for his own property and is thus causing damage and expense to the OP's property by his delinquency
How do you know the neighbour is well off? Because the OP says so? How does the OP know - have they seen their neighbours bank balance or something?
Money isn't always a primary issue anyway.
I may well appear wealthy to my neighbours - nice house from the outside, recently painted as it happens ( a friend of my husbands did it for me - my husband was a tradesman in his day, and the friend was a tradesman too) , nice large plot of land, not used anymore...…
In reality I am struggling now. Not necessarily financially but finding a contractor, getting them in and being there is difficult. My husband lost his sight recently, quite suddenly with wet macular disease and he is still in mourning for that, is having having his other eye monitored and treated and that means regular trips to the hospital. he is having difficulties and he wont allow anyone about the house when I am not there ( or sometimes at all) because he cannot deal with them . He is vulnerable. He doesn't drive anymore. he doesn't go out. He cant do the hedges as he would have once.
I am struggling because my husband did these things for me. I am pretty much alone too now.
My husbands condition is not widely known, I don't publicise it in case I am taken for a ride by some unscrupulous person.
My neighbour probably thinks I am being nasty too.
You do not know the circumstances of others.
I would say the fact that the smallholder has stopped running his own stock and now rents the field out suggests that there is something more going on there.
Just because you have a boundary with someone else and they have always had a fence, doesn't mean they necessarily have to keep it there. Whose boundary is it anyway? Do the deeds speak or are they silent ( quite often a case in rural properties I know). Has the neighbour always kept the fence up - it was mentioned he hasn't done anything for 15 years. What agreement does he have with the person using the land for his/her horses? Do you actually know?
In my own case , my neighbour would find it easier to cut my hedge from his own side rather than demand me do it. I am six feet below his level on that side and the hedge is held by a Cornish brick wall ( that's dirt with bricks stuck in it) with a five hundred year old hedgerow on top. If he wanted to take the hedge and put up a fence, as I know he does, he would have to raise the fence on his own property as my side wouldn't hold it.
( Although I am not sure about legal matters as it is an ancient hedgerow)
But , it is a simple matter to fence your own garden in surely if you want peace of mind?