I'm so sorry to read that, Nanato3.
A good friend of mine was attacked and sexually assaulted in the Ladies years ago when we were students. She had left the crowded bar to use the loo, which was through a door and along a corridor at the back, and a man followed her and attacked her.
Luckily, a member of staff saw him go after her and intervened, so the assault stopped short of rape or worse, but it was horrible. Had the barman been used to seeing males in women's spaces that intervention may very well not have happened, and I dread to think what the result might have been.
The only way to stop this sort of thing is to have a strict 'women only' policy in female spaces, and that has to include males who think they can be women. I realise that there will be entirely harmless transwomen who are upset by this; but I'm afraid their feelings have to come behind the risk to women in the spaces that were intended for them.