I think it depends on where you are from. I am in Canada and here in the Maritimes specifically Newfoundland it is a common phrase used for everyone not just older people. Another one that is used is Mother which is an assumption of your being a mother I guess. If they don't like you though there are a few other phrases they might use that you would wish for the "my dear". I do know what you are talking about though as the last few years working at a school before retiring I used to hear people say isn't she cute! I knew that was an ageism thing but I guess decided that it was better than being called an old battle ax so I used it to my advantage! 
By special request, let’s discuss our favourite Classic Music and why?
Women are a minority view so should be disregarded

