If he is at Nursery, they will be aware of his speech ability and if they think he has a problem, they will ask if the SENCO can be involved. This should be a red flag for the parents that there is more of a problem than they realise if that is the case. What the nursery won't do is tell the parents outright that they think there is a problem.
It may well be that there is nothing wrong and his language skills are still in development. It may well be that the people closest to him can't see the wood for the trees. I'm afraid it is just a sit back and wait approach from you because they are obviously defensive and pushing about this would blow a massive hole in your relationship.
As the Gran of an autistic child, I can remember the agony I went through with noticing problems that nobody else seemed to see. However, once he was referred for diagnosis, I talked to my son about the worries I'd had and he was glad that I hadn't said anything because he was so sure that when I was hinting at things, I was making a mountain out of mole hill. He felt that I would have just made his wife dig her heels in and she would have been reluctant to let me keep on looking after my Grandson. Sometimes, sadly, it is better to keep quiet and just be supportive when they need it.