I certainly wouldn't worry if it feels too stressful and you don't enjoy it. But even 'wasted ' swims probably means that a child isn't afraid of the water when doing lessons later.
For me, it is part of our family story, my mum was an excellent swimmer and took me from very young. I am very clumsy and cannot really do any fun physical sports with GCs, except swimming. I am also now better able to look after a child in a pool than a playground where I can't keep up!
The King's Speech To Announce 'All But The End Of Leasehold System' System'



