Well Jamaica was also populated by slaves originally, from around the same time period.
*I think pragmatically that is far back enough to establish rights to live where forefathers and mothers were living".
Ironically, Jamaica was given its independence in 1962, the year after the people of the Chagos Islands were forcibly moved.
As to who first brought the slaves there, in the West Indies as well as the Chagos Islands, there were continual skirmishes between France and Britain as to "ownership" of the islands, several changed hands more than once, the original slave population being imported by both at different points.
Again, pragmatically, we have to deal with whoever held/holds the remainder of Empires at the time of issues of independence or conflict, which is what is happening now.
Whats fascinated me today is the sudden shock horror press along the lines of "this means the Falklands are in danger".
No comparison.