I normally tumble dry my towels too, but I mainly use small ones which dry much more quickly. If I've had a bath or a shower I use a towelling dressing gown and wash it less often than the towels.
I do have a couple of bath sheets but would much rather use a smaller towel to dry myself off - I keep a pile of them folded in the bathroom so there is always a dry one waiting.
I've always understood that it is the calcium from hard water which gets into the fabric of towels and eventually makes them rougher, presumably that is why vinegar is often recommended? Others use washing soda which has a similar action. We have a water softener so don't have this problem.