I just use a microfibre cloth to clean the loo (after it has been flushed), rinse it with flush water, squeeze out over the loo bowl, then wash the cloth. I don't wear rubber gloves to do this but I do wash my hands afterwards.
As for the fridge possibly probably being 'dirtier' than the loo, maybe so, but so what? I put the food in containers. It is not touching anything dirty.
And we don't suffer from tummy bugs as a result of either of these methods.
I think a lot of people are really hung up about germs.
janerowena, I find your method perfectly sensible. Don't have a problem with it at all except that I don't have a dishwasher and rarely use a loo brush.