I don't think it is just gang masters - although they are a major problem.
Apparently, some wealthy overseas people bring staff with them and keep their passports so that if they are being ill treated, or not paid, they can't do anything about it.
I really don't understand why people with so much money are not prepared to pay people properly and give them proper conditions of work.
One of the solicitors I worked at in central London dealt primarily with clients from the Middle East. One of the paralegals had to visit a client because there had been complaints that there were too many people staying in she and her husband's holiday residence - against the terms of their lease - something which they vehemently denied. While he was having tea there, various members of staff trooped in and out with refreshments. He asked whether they were staying in the house, to which the client said "Yes". He told her that this was why there had been a complaint - they weren't allowed to have that many people sleeping in the house. "Oh", the client said "These aren't people, they're staff"!!