Of course "fair" should come into it. Surely it would in any relationship, otherwise there would be a complete imbalance of power.
Your son borrowed your car for 18 months. Was there any reason for this - were you or your husband incapacitated or unable to drive during that time? If not, you have already done them a huge favour, limiting your own movements for such a long time. Have they their own car now?
Somebody commented something to the effect that you should be grateful for the honour of looking after your grandson - and, of course, it is lovely to be able to do so BUT at the same time you are doing his parents a favour and shouldn't be treated like an employee.
If you were retired and doing a day a week, it might be seen as a relatively small favour - although even one full day with a young child can be very tiring. However, you are working from Monday to Thursday and your husband rarely has Fridays off. It seems to me perfectly reasonable for you to want to care for your grandson in your own home, where you naturally feel more comfortable and have access to more amenities. It seems totally unreasonable that you are expected to drive to their home and be marooned there for a whole day because you can't take your grandson in the car - and especially mean of them to expect this when they know you are both still working and have limited time together.
By the way, we looked after both our grandchildren for extended periods and our son and his partner NEVER gave us a set of rules by which we should abide. They were both very grateful that we were available and willing to help out.
This idea that parents can dictate on every issue that arises rather than discuss concerns in a respectful way and hopefully come to some sort of compromise, is I think quite wrong. And it is especially annoying to hear that you are, in effect, being pressured to accede to their demands by denying you access to your grandson. I agree with those who say they are ungrateful and entitled.