I'm glad to hear what you say in your last sentence, lucky. That is how it should be. Would that it were always. I also agree that quite often a professional judgment of a person's capability, based on their limitations (of whatever cause; we all have limitations) is not unreasonable, but it should be and can be done properly and without being insulting.
Sadly some people will feel insulted however careful others are. That's just how people are.
soon, I think you are right about people's feelings. My point is the same as janea's: one doesn't have to let the insulter know one feels that momentary stab of their insult, and after that has passed one realises that the insulter is the one with the problem.
I did once feel hurt by an assumption that someone who should have known better (did! know better; he was a fellow scout leader) than to make about me. I challenged him in due course and then realised that he would feel ashamed because he had lost some of my respect, at least temporarily. He was the loser in that instance, not me; I did not lose any self-respect because someone had done me an injustice.