Haha, having just come from the pool where exactly that was happening, I must admit it was just a wee bit annoying, especially as there was heaps of room in the shallow end; they didn't need to gather in the way of swimmers.
Trouble is, although I can see the light on the water and if someone is directly in my path they block out the light and give me a clue they are there, I have to wear dark glasses and when I'm on a roll, so to speak I don't always see them in time so have ploughed into them on occasion.
It won't stop me swimming though.
I learned in a bitterly cold Yorkshire river when I was six, was a member, for years of a swimming club whose name I am no longer allowed to say and is picked up by most websites and refused - but if you are curious....
www.thetelegraphandargus.co.uk/news/8073476.jim-appleby-past-times
And I've swum many a marathon for charity. So a few gossipy people won't put me off.