I previously suggested being friendly (not over friendly) with children if they are within speaking distance. Definitely didn't suggest that OP should lighten up.
Of course I feel sorry for the OP, to feel so anxious all of the time must be so upsetting - but it's what to do about it.
Telling them to be quiet or to go away isn't working.
And so I would see the alternative as trying from a different angle.
There are always parents who do not exercise control and are happy for their children to be out of their hair, but they are unlikely to come around threatening if you say to a child - are you enjoying your school holidays !
I have been there with children who congregate, make noise etc. There is a wall about 15 yards from my garden which they find very useful for meeting, kicking a ball, doing cartwheels. To be critical and tell them to go away wouldn't have worked, especially when they are in numbers and feeling brave.
I went over to them, said hello, asked if they had any little brothers or sisters, and what's it like when they can't get to sleep...... My little dgd is trying to sleep in that bedroom up there and when you kick the ball it keeps her awake, she is very tired and grumpy. Could you help me by playing very quietly or moving down the road a bit (yes I know that would be outside another house) and then I said thank you and said - does anyone want a sweet - of course they did. Bye now.... see you another time.
They now say hello to me, sometimes they still too noisy.