It is important sometimes. If you want to talk about racism it would be impossible without being able to refer to different racial groups. How would you explain that if you are black you are more likely to be stopped and searched than if you are white, for instance?
The difference between 'coloured' and 'person of colour' is that 'coloured' puts white as the default, and suggests that anyone not white is tainted in some way. 'Person of colour' puts the human aspect first with 'person' or 'people'. That's how I see it, anyway. Also, 'coloured' was a term in South African apartheid, which gives it those connotations.