I wonder if those who don't mind it have generally been supposed to be younger than they were and so were assumed to be actual girls well into their adulthood. I was still called a lass or a girl when I had a baby on my back and a toddler in a buggy. I think they'd stopped assuming I'd only need to pay a half fare on the bus by then.
Then there was the one time in Edinburgh (also after two kids had arrived in my mid to late twenties), when I was cycling home from an evening meeting. As I came down onto Princes Street, well wrapped up against the cold (balaclava with a helmet on top, etc), some "young lad" in a group yelled "Are you a laddy or a lassie?" I smiled to myself and cycled on. Why would I care what someone unknown to me and who was not being aggressive thought about anything?
? If one has a healthy level of self-respect I'm not sure anyone can belittle one. Any belittling they try is about them, not you.