First of all, "muslim" isn't a race. That said, the comments by O'Leary were, I think, deliberately provocative.
I've been trying to think how (or if) one actually could tell of 'terrorist intent' or the risk thereof say, hypothetically, by the glint in a person's eye, whether it would be ethical to target people with the glint (i.e. pull out and check over more thoroughly than those without that glint). I think it would be ethical in the same way as we target disease risks with vaccinations, or putting people with high risks of other medical complications (heart disease, stroke, etc) on medication to reduce their risk.
What I'm arguing is that approaching terrorism risk using mathematical probability, based on up-to-date information, would not, I think, be unethical—always providing such a method could be devised.
I think O'Leary's point was that, as he understands it, the greatest risk of terrorist offences at the moment comes from extreme Islamists and so, statistically speaking, from his point of view, it would make sense to check more in that demographic than others.
Admittedly, such an approach doesn't look good and it's hard to see how, even if it were probabilistically targetted, most people could be persuaded of its 'fairness'.