I used to teach tai chi, mainly to older people, and it is true that learning the form is a challenge. Not every movement class will suit everyone but if you want to memorise the form then you should certainly go every week and practise what you are shown in the week. It takes about two years of diligence to learn a short form.
That said, not every teacher suits everyone. Teaching with the back turned is traditional Chinese practice and it is so that people do not become confused and mirror (get the wrong way round) the movements of the teacher.
For Westerners, the challenge is to relax and be in their bodies, it is thus incredibly good for us - tai chi lowers blood pressure, teaches mindfulness, brings relaxation and excellent balance etc etc - but the biggest benefit is to learn to focus, to not 'end gain' but instead to learn to manage frustration and to be kind to yourself. It really is worth it but you have to persevere. I used to think that in a way, it was like learning professional dance, yes every person will move slightly differently but you have to learn what is 'good enough' and that will change!