I think because, like other things in life it is a spontaneous comment when surprised by someone or something.
We recently ate at a restaurant where the waiter who came to serve us was at least 6ft 6in and the low ceiling and his thinness made his height more obvious. I confess DH did, look at him with surprise and say, 'Gosh, you are tall'. I thought the same, but didn't say anything. DH wasn't telling him anything, it was more surprise reaction. I did say to him afterwards, that he really shouldn't have said anything and he said, he was aware of that but had been so taken by surprise at the time.
I have shrunk several inches since the menopause and as I am long in the leg, I look minute when sitting down. It takes me by surprise, let alone anyone else! Today I went to the optician and she was commenting on my small head because the sides for my new glasses frame will need to be specially made for them to be short enough.
If you are slightly out of the main stream, people will comment, I cannot say it bothers me even the smallest possible amount. i am as I am.