For me, the problem is that, with so many items of clothing, both men and women don't seem to have a clue how they look.
For example, if I'm looking at cellulite oozing out of the rips, or nipples and belly buttons that seem to be set on stun, it detracts from the overall experience, unless it's an absolute dive with great food, like an oyster bar on the ocean.
I don't enjoy being served by someone who is heavily tattooed and or has piercings in their nose, lips, and especially tongues. Also, don't like flip flops or Tevas with gross unkempt looking feet.
It's like looking at a train wreck, and I think it detracts from the overall atmosphere and the food. The reason it detracts from the food is that often times it gives the person sort of an unclean look that is unappetizing.