Why not stop farming in some areas, where it's not profitable and the land could be put to another purpose?
Probably some of it could be sold off for housing, yes, but my view is that we should be producing more home-grown food rather than importing more as the population grows. Farmers should be paid a price (perhaps subsidised as foreign farmers are) which enables them to make a profit, not a loss, employ workers who could rent this low-cost housing which has been built on a field which is not good for agriculture and we become more self-sufficient.
Crops can be grown on set-aside land at present as long as they are not food crops.
Pause for thought........ Can you think of examples in other industries in the UK where owners of a resource are paid either not to use the resource, or to manage it in a specific manner?
Rather than import as we now do we may need to produce more food in the UK.