I had never had wasps nests until I moved into a rental cottage with a garage where I was storing all my possessions. The garage had a tin roof, and evidence of many abandoned wasps' nests, I used to spray it regularly to avoid new nests being built after finding a 'golf ball' sized one.
I would never remove honey or bumble bee nests, but I did have a wasps nest killed in the roof of the rental cottage just above the bathroom window. I felt very badly doing it, wasps are part of the circle of life and very useful to gardeners, but I was frightened they'd come into the house. I used to work at the RNIB in Notting Hill Gate, the college was housed in two beautiful old buildings. One day I went into the toilet and the toilet seat was covered in wasps - luckily I found it, not a student.