I think you have been seriously taken advantage of here, the young man should have been required to move his car as soon as you asked him to.
I think it’s really lovely that you are happy to let your neighbours park on your drive, long may it continue, but people also have to respect that they are parking on your land. They should all be asking your permission and checking when/if you need to go out yourself.
To be honest, if I’d found a completely strange car parked on my land, with no forewarning, I would probably have phoned the Police and reported it as an abandoned vehicle, possibly stolen? That might have given the owner something to think about!
Equally, it’s quite tempting to suggest you start charging, if people want to use your land as a car park, causing wear and tear to your property, maybe they should start paying for the privilege!
You are certainly not being unreasonable to expect to be able to get on and off your property as and when you need to. The unreasonable bit is assuming it’s ok to invade someone else’s property and dump your car there!