I very much doubt that all but a small number of wealthy people regularly use their bus passes.
We are reasonably comfortably off but, given the amount of travel I do, I would think twice about using public transport if I didn't have my travel pass. For people who are struggling but who would be just outside a means tested limit they would probably only use public transport when absolutely essential. In the long run, I believe this would have an impact on older people's mental and physical health, which would also have cost implications.
When we have multi-millionaires and huge organisations evading or avoiding taxes, the wealthiest people paying ridiculously low council and inheritance tax, councillors like the one on our London News being wined and dined and given dozens of expensive gifts by property developers and the like, I refuse to feel guilty about benefiting from free travel.