OK it's your fence but unfortunately it's now leaning his way! I think that if you asked if he'd like to contribute towards the cost of a new fence, he might be a bit put out (and I bet the person who instructs him which trousers to wear would not be very keen on that plan either).
Any offer would really need to come from him, I think. I wouldn't mention money. However, it might be a good time, since you've been doing lots of work in your garden (I seem to recall), for you to have a nice friendly chat over the leaning fencing and tell him that you have lovely shrubs etc., planned for planting close to the boundary and you'd really like to sort out your leaning fence before you press on with your landscaping. Is there any chance he could give you a hand to get it right? Blasted gales etc., He may be totally o.k. with that, or at least understand that it'll be difficult for you to sort it out on your own. And that's very different from asking him to put his hand on his cheque book!
He might just surprise you and come out with a suggestion of sharing the cost of a new one - but I'd think that's very unlikely as it's not his responsibility and it would need to be his own suggestion so he doesn't feel put out by your asking for ££s.
If you say nothing about it, though, it's only going to get worse and it is your fence, so legally - unless he cares about the boundary - he could just wait for you to decide one day to replace it.
If it were me, I honestly wouldn't spend the money, unless it's rotten. I'd prefer to spend a bit of time sorting it out and save any spare money for plants - far more entertaining. I don't think that the quote for a replacement fence is really excessive, but I Judy wouldn't want to spend a lump of money like that - if there was any way around it.
I'd hope he'd help you fix it. If he does, don't be shy with the spirit level.....
Don't forget to let us know what he says.