Travelling solo in a group and avoiding people who want to be your new best friend

I love to travel and sometime find it easier to go an organised group trip. Not a solos holiday, just a group but I often find myself sitting next to another singleton who then want to hang on to me and go everywhere I am going. How do I discourage this without being rude? I am fit and a fast walker and have often planned my days but find someone with limited mobility wants to tag along. I am happy to do some things or go out together for dinner but I love doing my own thing!