Many good points made by other posters but as I read I kept feeling this is not and should not be your problem.
The commitment you have chosen to make to your partner is one that means your house is still yours and quite a bit of your time is your time. The fact that the son is abusing these arrangements is not your problem it is his dad's.
It will be difficult but, as far as I can see, the alternative may be you and his father no longer having a relationship. I feel you have to firstly speak to your partner and ask him to speak to his son or, at least to understand your position and then you have to start to say no. Do not make excuses, do not blame yourself, they are not your responsibility. I think caller ID on your phone would be useful as, once you have said no a few times you can make sure you just don't answer his calls for a while.
He seems to be very thick-skinned and selfish and he needs training so first "no" and then no reply would, I hope, get through to him in the end. Should he just turn up it would be a pain but until he gets the message say you are just going out and it's not convenient. Actually get your coat on and go if necessary until you brake the habit he has, very conveniently for him, got in to.