What bothers some people, I think, is not so much the actual diminutive but the over-familiarity of complete strangers in using it.
My name does not have a diminutive. My late ex re-configured its letters to devise a 'pet' name for me; I did object when an individual I'd only just met overheard him and started using it instead of the proper name everyone else used. It seemed a tad presumptuous. Like Mr Darcy, I do not have the talent of conversing easily with people I have never met before.
However, I did - as the current culture now demands - manage to "get over myself" and accept that some people are naturally gregarious.