I agree with Notanan2.
I have known people deliberately have conversations about topics that exclude some (or only include others) and it is definitely very rude.
Sometimes it just happens, however, with no malice aforethought. If people are talking about what happened on Eastenders and someone else arrives who doesn't watch it, they will feel left out. Ideally, people should be able to read the room and notice if someone is looking a bit left out, but I suppose it depends on how long the conversation is expected to last. 5 minutes before a meeting is very different from a whole lunch hour.
Sexist? Not at all, unless it is intended to be. In the old days when golf courses were for men only, it was often said that the best deals were done on the golf course. That sort of thing was very definitely discriminatory, as is deliberately having meetings on licensed premises (which excludes members of some religions) and regularly holding meetings out of school hours to exclude parents.
I once had a manager who did the latter. She resented working parents (well, working mothers, really - the one single parent male in the team was treated like a saint), and refused to have end-timed meetings so that I could make arrangements for my children if they were going to run late. She did it on purpose, but I knew that it would be playing into her hands if I made a fuss, as she would have used it as 'proof' that mothers were a burden in the workplace.