I used to handle/settle motor injury insurance claims.
In my experience, most of the injuries caused in motor accidents were not caused by older drivers - they tend to have scrapes in car parks and minor accidents (there is always the exception of course!!) They also tend to drive older, slower cars and not go far out of their own locality which they know well, so unless there is a medical reason - physical or mental, many older people should be able to continue to drive as safely as the next person imo.
I do think however, that doctors should "grow some" and if they think a person should not drive they should say so, not make airy fairy statements, that leave the matter open. It just makes it more difficult for children and relatives to have "that conversation" later!