Yes, it is impolite, this is from the Age UK website:
Dan Holden of the International Longevity Centre, summarised:
“Elderly is problematic because of its association with dependency and frailty, which typecasts whole generations. Although it is natural that we refer to people in groups by their age, chronological age is a poor guide to understand any given person; their health, their personality and their experience. It’s not perfect, but a more neutral term like “older people” is more neutral and goes some way to addressing the stereotyping of this age group.”
I'm in my 60's and still working but I don't mind being an "older person" the word "elderly" makes me think of that hunched up old couple on the road signs.