Yes, I am concerned and no, you are not just a worrier!
In the 1987 hurricane, 100ft of our then large Surrey garden was completely obliterated by 3 enormous ash trees that were blown over diagonally onto us. Fortunately, only the shed and greenhouse suffered. DH went out and bought a chainsaw, as owners of the trees didn’t offer any help or assistance! They were part of the field belt left when the estate was built in the 30s.
Where we are now, we have one enormous ash tree in our rear garden, part of the original field belt of trees that extends to 6 adjoining rear gardens. Many have been felled over the years, but all remaining are known to be infected with honey fungus and other diseases, but are covered by TPOs put on when the houses were built in the 60s. We are not allowed to have these trees removed unless they are actually dead. If any of the trees fall, they would definitely damage our homes, but the TPO can’t be lifted, although the PC are in agreement, as it would cost money to do so. We did manage to get one tree on our property removed in 2018, the contractor issued a 7 day order, as it was so dangerous, but we’re instructed to replant with a similar tree. DH researched all about the suggested replacements, forest trees which would eventually be larger and taller than the diseased tree - along the lines of recommendations to plant at least 200 metres from buildings, when the planting site is 20 metres from our bungalow!
If your tree isn’t protected by a TPO or in a conservation area, I am pretty sure you can have it removed, although it might make the remaining tree more vulnerable, and the cost might be a consideration. What do your neighbours think about their tree. Could it be topped and thinned maybe, has it been examined by a tree expert to ensure it is not diseased.
Beautiful and vital as trees are, some have grown far too large for their surroundings and sensible options have to be considered. I am apprehensive now when walking in some wooded areas in high winds.