Savvy
They're not using it as a shortcut, my garden is self contained, but they are sending any random workman straight into my garden without asking me. I'm quite happy to allow access if I know who they are and what they are doing there, but my upstairs neighbours are usually at work and without expecting them, I've no idea if these people are genuine workmen or not.
Their argument is that they don't need my permission to engage workmen, but surely they should be asking my permission to access my property, especially when its not an emergency.
We went through this some years ago when we split one property into two. You should probably talk to someone who really knows but my memory is that you have to give them reasonable access, not unrestricted access, and that the interpretation of this is that - if they need access for works - they discuss it with you and you came to an agreement on a mutually acceptable solution on a case-by-case basis.