How to change my gardener?

I've been paying a gardener to come twice a month to cut grass & do a little tidying, but I'd like to stop using him. He's expensive, cuts his visits short at both ends, and has just been very grumpy with me about a "Push to Open" sign on my back gate: "Wouldn't have hurt you to use the P word, would it?" I did try to find a sign with please on it, but they don't seem to exist, and it wasn't meant as an insult to him anyway. I know it sounds feeble-minded but I'm a little autistic and don't deal with difficult people very well. Just wondering if someone with a better brainbox than mine has any tips on how to tell him to stop coming?