Trisher, do you honestly think that in most cases people can't tell someone's sex - regardless of how they are dressed?
You don't have to be actively scrutinising or looking for 'anomalies', but when you see transpeople on University Challenge, or on the bus, in the pub, or anywhere else, you can tell.
In most cases all that happens is a fleeting thought - possibly instinctive - that something doesn't look quite right, then you realise and carry on watching TV, drinking your drink, reading on the bus or whatever else you were doing. It doesn't matter, and it's not important, but people can tell, no matter how much you protest otherwise.
As for the idea that everyone finds transpeople frightening - it's just not true. As has been said over and over. It is not 'ordinary' transpeople who give any pause at all - it is the ones who try to silence women, who force their way into things, who want to change the language, to cheat at sport, and so on. They are the ones people complain about. They spoil things for the transpeople who just go about their lives, as well as for female athletes, women from sex-segregated cultures, female prisoners, vulnerable hospital patients and so on.
I don't know why that is so difficult to understand.