I dunno - I'm a hand washer after the loo. But I do something think "why"? What ARE these germs that we need to banish? And where are they coming from? Urine, as I understand it, is sterile. Even if it stops being sterile when it exits the body, it's only "touching" air once it leaves our bodies.
I don't know about anyone else, but personally, I use toilet paper as a barrier between my hand and whichever "part" needs wiping. I've never wiped with bare hands and I don't know of anyone else who does.
As has been pointed out, germs are everywhere. Not touching a handle of a door when exiting a toilet seems over the top to me. If we are so worried about non-hand-washers, shouldn't this concern filter out to every other aspect of life. How far do you take it? Would you avoid touching, or wash hands after touching, everything you come into contact with?
We'd surely be caught up in a maelstrom of hand washing and unable to get on with day to day living.
I will continue to wash hands after toileting, but I won't get too caught up in worrying if others don't.