I also do not like driving with people chatting to me apart from the GC as I can respond or ignore. DH is absolutely the worst back seat driver and I have to ignore him or we would be dead or other people would be dead! I zone out if giving a lift to others - my mother never shuts up but she has never learned to drive, moans about all the cars yet expects a lift everywhere. have on occasion stopped the car when she says "so and so doesn't go this way" and asked who is driving! I also don't like talking on the hands free phone so if anyone calls it is very brief.
I think if I were you, Peonyrose, I would make the break by getting the bus - maybe say you fancy a glass of wine or some other reason ( sore ankle, neck etc) so can't drive, then the cycle has been changed and your friends will have to seek alternative ways home.
Things you find stressful that other people don't notice.
New house and a sloping garden