A free trade agreement could be not based on free movement of people and same rules and regs for both sides- I suppose. But NOT with the EU- that has always been clear as a bell. Because that would unleash a whole series of 'pick and mix' in the EU- and this just cannot happen. Not within EU countries, and not with those with trade agreements, like Norway or Switzerland- for same reasons.
In 2014, the Swiss had a referendum to limit immigration (it is a tiny country, with limited space for living and growing crops- and has a much larger % of immigrants than UK). One difference being that Referendums are binding in Switzerland, unlike in the UK. The result mirrored the one in the UK, eg very small majority in favour, mainly german speaking Cantons. The Government had 2 years to implement- but then negotiations showed that the effect on the country would be very negative, making our reciprocal agreements on Trade, etc, with the EU to be void - as one of the major tenets is free movement within the EU and those with Trade Agreements with it. The Swiss Government, for the first time ever- was unable to put the results of a Referendum in practice, and it was quietly 'dropped'. Even the majority of those who had voted for it let it happen, bar a tiny minority of hard xenophobic UKIP types (called here SVP or UDC on French side). The EU made it clear that it could NOT in any way, shape or form- give way on that issue- as it would lead to other requests for same from the UK- and perhaps others later.