Axxliz, you're not alone! I am guessing that you usually watch on your own, and it's harder to keep track when you don't have anyone to turn to in the advert break and say, "Hang on, who's the guy with the beard again?" Even when my husband was alive, we sometimes used to record the first episode of a drama and watch it twice, just to get all the characters and relationships sorted out in our heads. When it's just a single head, it's that much harder - you lose concentration for minute and suddenly the police are breaking down a door and you think "What??"
I think they should have special action subtitles for - well, I won't say the elderly, just those who can't always keep up. The subtitles could say things like "This is the woman we saw in the opening shot but she's wearing a different coat now" or "Two of those are her children but the youngest is her stepson", or perhaps even "Yes you saw right, the car rolled right over and we cannot explain why all the occupants and the car itself are undamaged."