Neighbour dilemma

Some good suggestions offered already, but just a thought - are any of your other neighbours troubled by his actions at all? It might be that, collectively, you could approach him, but I do understand that you don’t want to aggravate him any more than he already feels he has been, so might not want to do that.
Probably one of the best ideas to cut down the sound would be, as suggested by NotSpaghetti, to put up some trellis in front of the fence and place pots with trailing plants in them. This might deaden the sound a bit, and also place a water feature on your patio or close to it as also suggested.
I hope you can find a solution that helps.

The noise cancelling headphones sound like a very good idea.

Thank you, but actually our patio, pond etc is the other end of the garden but I can actually hear him when I'm in the kitchen so doesn't really help

I thought of this but do love the sound of the blackbirds and bees. Whole point of a garden really. Also seems unfair that we have to compromise when we do nothing to disturb him

I would be planting stuff and putting up trellis. Every bit of greenery will be a baffle to some extent.
Can you put your own patio near your house so you aren't right next to him (given he's at the bottom of your garden)?
Maybe include a babbling water feature and/ or wind chimes?

It's a height issue I think you'll find Astitchintime.
We recently replaced a wooden shed with a brick one. It had to be below a certain height if within a metre of the fence in order to not need planning permission.

If he has physically attached anything to your fence you have a right to ask for that to be changed. Also, do check your local council website as I did read on ours that a neighbour cannot erect a permanent structure within a certain distance from a boundary fence but this might not apply locally to your situation.

One side of me says he is arranging his garden features and whistling just to annoy you. Noise is a stressor whether it's factory level or whistling.
You don't mention a partner or children. If he is on his own then maybe he isn't very tuned in to consider others.
You have a tricky situation here because as you say he can erect sheds etc wherever he wants to on his property within reason and whistling may be seen to those who don't have to listen to it constantly as a sign of being happy. To complain about any of this may make you look like a misery.
However, it is clear that your quality of life has been affected and that isn't fair. Are you able to rearrange your garden to put maximum distance between you and the fence?
As he has been hostile to you before I doubt he will comply with any request to stop whistling.
Headphones may be some sort of compromise but it will hinder conversation.
My elderly next door neighbour who I share a fence with has gone to live with his sister and sold his house to his nephew. Said nephew is very pleasant but he has converted his garage which forms part of our boundary into a man cave complete with flashing lights and quite loud music.
Both of us will be looking forward to a nippy autumn where our respective thorns will retire to their homes.

I had this problem a couple of years ago although mine was young children screaming in the garden from 8am until evening on any day that was dry.
It was incessant and sometimes including visiting children .
I used to go indoors until I had the bright idea of buying noise cancelling headphones.
A drastic solution I know but I could sit out in my garden to just read or enjoy the weather.
It had lessened now the children are older and not so loud.
You also have the advantage of winter coming when presumably neither if you will be out so much.
Next year he may have stopped his whistling and taken up something else, I hope for your sake it isn’t a musical instrument.
Seriously though I do understand, I was driven inside by the noise until the headphones arrived.

This is difficult without seeing perhaps even a small drawing of your garden to advise on what to do in your t block or deaden some of the noise. I'm you won't stop the whistling by asking so just wait a while and he will probably get tired trying to annoy and find something else to do. My daughter lives next to a woman who sings along to the radio in a very high pitched voice. She just put on her own music, loud!