Negro describes a set of physical characteristics that apply to a certain race - it is not just skin colour. You can be black and not a negro. It is a scientific term that refers to a specific racial grouping within which are shared physical characteristics.
Equally the word caucasian does not refer simply to skin colour, but also to other characteristics (skull shape, facial features, hair colour and texture etc.)
The idea that the word negro is derogatory implies that this group of people are in some way inferior, a view that I do not share.