I think so-called 'political correctness' is better described as being considerate of others.
It just means replacing an offensive term with another, more neutral term. It really doesn't matter whether the speaker would or wouldn't be offended - it is the receiver who counts. Manners are about making others feel comfortable, after all.
I think that those who insist on using terms that others find offensive are rude, and often narcissistic, as in refusing to expand their vocabulary they are only thinking of their own 'comfort', and refusing to make an effort.
I think that most people would understand if someone accidentally caused offence by using an outdated term (within reason, however), so long as when it was pointed out, the user changed their ways.
Speaking as a redhead, I don't think that 'ginger' is particularly high on the hierarchy of offensive terms; but any personal remark can cause hurt, and there is no need to point out aspects of someone's appearance, as we don't know one another's insecurities.