I've read interviews of people coming back year after year for a decade or more. The seriously wouldn't do that, if conditions were so bad.
More importantly, it's pie in the sky to think that British people are going to step in to do the work.
We no longer have armies of housewives, willing to spend their summer fruit picking. Most women with children already have jobs. Students' holidays don't last long enough for the whole season. Unemployed steel workers from the North East aren't going to move to Kent to pick fruit, especially as there is better paid work available on wind farm installations, etc.
One way or another, foreigners will still be picking the UK's fruit. The immigration proposals are already beginning to fall apart. There will need to be exemptions for the NHS, some teachers, care and domestic staff, au pairs, seasonal hotel staff, entry level scientists and researchers, ete etc etc.