Yes, he is making assumptions. But his gumption is right, I think. Obviously it's a very generalised statement he's making but I think, like many 'soundbites', it contains the essence of truth about labels, about why people use them and what they do to others' perceptions of a 'labelled' person, whether the label is applied by the person themselves or by somebody else.
I think he's saying that the fewer labels a person has, the fewer assumptions others make about them before they really know them.