You say “obviously “ you have them during school holidays. Do the other grandparents do their share?
Could you separate them so that 1 comes to you and the other to his other GP’s? Alternating of course. I always used to have my oldest 3 GC together when they were little but I have really enjoyed getting to know them better as individuals when they started coming alone. It also made them feel”special”.
The other thing is Granny’s house-granny’s rules and I found it easier to have them here where, despite less space than at theirs I felt comfortable setting out a few basic ground rules in familiar surroundings and they generally behaved much better than at home!
A nearly-only child (big age-gap with my sister) I have always found the fighting hard to take- they could be like puppies in a sack - but rough and tumble is also perfectly normal with children of their age!
Not the stones, obviously, but it occurs to me that it’s just the level of rough and tumble you find hard to take.
Again, in your own home, you can set your own ground rules.
I hope you can find a middle way through this so that time with the boys is a pleasure and not a chore - it will soon fly past.