Buddlia

Buddleias can be pruned hard, very hard, and always recover.

Strangely enough, it is on the list of forbidden plants to bring into Switzerland - and the two I brought from our UK garden (1 white and 1 dark purple) were confiscated at the border.

It does sound like caterpillars, but as someone else has said it won't harm the plant. Taking off the leaves may make it look better but is unlikely to cure the problem.

Thanks folks, I am wondering if I should cut off the damaged leaves, going into the garden with DGS soon and will take another look.

I love buddleias when in flower, but for the rest of the year I do think they look very plain and uninteresting, not doing anything in spring or autumn. OH, (head gardener) cuts them back hard every year and we enjoy the butterflies and bees hovering round them when they're in flower. Maw your 'invasive' explanation makes sense to me now. They are weed-like. Hypericum is the same - it seeds itself and grows like a weed, everywhere, in paths, etc once planted in the garden.

I planted a lovely deep purple one this year - and I had quite forgotten about it until I read this, so thank you! It is rather swamped by foxgloves at the moment, but will come into its own, I hope. Haven’t looked at the leaves though. When (if) the rain stops......

I noticed some caterpillars on mine about a week ago, the leaves are now holey too. Pretty sure this is your 'problem'.
In addition there was a wiggly grub of some sort in one of the buds.
I've left the caterpillars to do their own thing with the leaves but ousted the grub to enable (hopefully) the flowers to bloom and encourage more butterflys.

I do like the buddlia's especially for their attraction to bees but in parts near where I live they can grow like a weed and end up straggly and unkempt. Last year I sadly had one cut down last year as after growing out of nowhere it grew and grew and the flowers were stunning but it blocked out the light in my back kitchen as well as it was getting out of hand ( thinking of pipework beneath the ground and the damage that the roots could cause )

Caterpillars /grubs! There is a reason they are called the Butterfly Bush!
I have always loved Buddleias especially the deep purple Black Knight.
At Chatsworth Flower Show last year, one of the show gardens was meant to focus on butterflies and I commented to DD’s friend who was head show garden judge, that I was surprised not to see any Buddleias. She replied that they are no longer encouraged as they are regarded as “invasive” (posh for “weeds”?) and it is true that they can grow anywhere - brick walls, railway embankments, between paving stones.
However, I like “invasive” because it means a plant has a chance surviving my inept attempts at gardening

Bet it’s a bug. Shouldn’t worry too much the birds will pick them off and buddleja is tough as old boots.

It sounds like something is eating the leaves. I know that there is a caterpillar that likes them, it's the caterpillar of a moth. If that is the case you should be able to spot them.
Otherwise I'm stumped.