I think it depends on the relationship between you and the person offering. If it's an acquaintance I would just say that I don't drink coffee, and hope to be offered tea. I don't think it's ok to assume that being offered one thing means you can ask for another, unless you are with close friends and family. It's no different from being offered water and asking for gin instead, really
.
I often have groups of people here (not friends, but group members) and when I offer drinks I usually say 'can I get anyone tea or coffee?' or similar, but every time someone will say 'do you have decaffeinated coffee with oat milk, or strawberry leaf tea, or sugar-free unicorn-horn matcha?' I'm not running a cafe, and if I had fancy stuff in I might want to save it for actual friends.
When those actual friends are here, I ask 'what would you like to drink?' and people are free to suggest anything - they know what I'm likely to have, and that they are welcome to it.
I was always taught that good manners mean not putting people on the spot, so you wouldn't push someone into giving you something that's not spare, or making them admit to not having what you've asked for. I know it's just a teabag in the case the OP mentions, but the principle stands.