Hidden agendas
I wonder about three aspects of derogatory words, in addition to the issue of whether somebody deserves condemnation.
First, it is logically offensive to criticize a person by criticizing his group--unless the group truly deserves insults (like Nazis or thieves), and the person shares their shame or guilt. It would be wrong to call a politician a Nazi for things that are not Nazi-like--but appropriate for things that are.
Second, it is logically offensive to avoid truthful criticism in general just to avoid hurting people's feelings. If a politician does Nazi stuff, we should call him or her on it, even if it hurts feelings. This does not mean that we should always police people's faults--only that we are not logically forbidden from calling a spade a spade if we think it is called for.
This point is sometimes lost in law. Sexual harassment, for example, is one of the few laws that is defined by what the victim thinks (if a woman feels harassed by an action, she is). We would never determine assault and battery by what the victim thinks or feels--there must be overt visible action that everybody of sound judgment would call assault and battery.
So must we ban words that offend people, whether they ought to be offended by them or not? In most company, we know what words are offensive to the crowd, but in public we do not always know that. That is why signs might be posted to refrain from profanity or vulgarity because of the crowd that might be there (children are allowed, for instance). This is a courtesy issue--not law or morals--but because in some cases it affects a place of business, it is also a financial issue. We don't want people running off our good customers. And the law gives some support for this. The only support from morality on this is that morality calls upon us to show respect for others to the best of our ability, and insults (if not deserved) do not show respect.
And the third question is whether it is immoral to use derogatory language at all (when it is not justified). The hinge of reasoning here falls upon the word "justified." I am justified in calling somebody a murderer if he or she is one (though that does not entail that I must bring it up). Am I justified in calling someone who killed a deer a murderer? I would say no. Or someone who killed by accident? Again I would say no.
Consider the N-word, or the others listed above for Jews, or for Puerto Ricans, etc. I consider it morally relevant that there is never "justification" for these words, because they do not attribute anything morally interesting to the victims. They are meant only as an insult (though why they are insulting is cloaked deep in the history of the words). If the n-word, for instance, only meant "black-person!" why would that be an insult? If it meant, "low-down, no good black person," then it would be an understandable insult--but nobody wants to ban that language, because sometimes it is true.
Is it that the n-word is never sometimes true? As African Americans use the term, I think that sometimes it is true (as the word "murderer!" can sometimes be true). As white people traditionally used it (low class southern white people), it was a term that carried no crimes or immorality with it, yet was meant as an insult because it associated a belief in inherited inferiority with the group. So the insulting false-accusation (that makes it unfair and therefore not moral) is that it carries these hidden false assumptions.
(Note--I grew up in Memphis before integration, and we were taught not to use the n-word because it was low-class language, and that was moral reason enough.)
p.s.
This conversation does not address the weird logical problem that referring to a word should never be confused with using the word. If I "use the n-word" on the witness stand to say what the accused really said, I am not in the logical sense "using the word" myself--I am merely referring to it. But we have got used to being confused about the logic of it, and came up with "the n-word" as a way of referring to it without using it. This assumes that people know what "n-" stands for (but logically we can never tell them).

My question is why we have forbidden words in the first place. Yes, it's obvious that common civility and a desire for harmony tend to preclude the use of such things as the N-word, the K-word, the C-word, and so on. They are almost universally considered insults. But why is "fuck" forbidden, or "shit"? I can say "sexual intercourse" or "excrement" or any of a number of other synonyms, but why are these simple words - mostly of Saxon origin, I think - so widely thought of as unacceptable? My own theory, not based on much evidence, is that it is a class thing. Using these words indicates a lower class origin and they "must" therefore not be used.
The real question, maybe, is why we need forbidden words.What social purpose does it serve to forbid them, in other words, and who gets to decide that they are and that they should remain forbidden? I say, "Screw these opinionated self-appointed moral arbiters!" Or is that forbidden, too?