I do feel for you but the whole tone of your post smacks of a bitterness which is harmful to your mental and physical health. You are making assumptions that the GP will not understand, your friend wants a pound of flesh and being resentful of anybody who happens to have more money than you. This will lead to you becoming alienated from your friends too which is going to be really unhelpful to your wellbeing. Incidentally, overpayment of benefits is repayable now, thanks to a landmark case in the courts. You are not alone by a long shot in being victim to the state's incompetence. It isn't even a rare occurrence these days with a complicated benefit and tax credit system with untrained staff calculating them. The information which you would think would be shared between the different departments doesn't happen and one hand doesn't just be unaware of what the other is doing, they are on different planets!
I know you want to save, which is very sensible, but you aren't saying that you are getting into bigger debt, just that things are tighter than you would like. No fun, I agree but you are talking about about a limited time for this before the payments become a thing of the past so there is a small light at the end of the tunnel. I would hold on to that whilst I explored the CAB or debt advice people. If things are tighter than they seem to be from your original post, then I suggest you get help in getting the repayments reduced because it is illegal to leave you poverty but you will have to pay more in interest which the HMRC are allowed to charge.
Try to reframe how you look at this by doing some mindfulness exercises. It won't change your situation but it will make you more able to cope for it. Negativity, even justified, is so toxic. It is unlikely that you would be able to do much for a while yet with the Pandemic restrictions in force so you aren't actually missing anything in the travel/holiday stuff. For all their money, your friends are as restricted as you.
You will get through this, just try to do it in a way that doesn't leave you feeling so resentful. No experience is ever a waste of time, it is a learning curve.