Single sex toilets should be compulsory in buildings that stay open at night (eg in universities, hospitals and anywhere that can be accessed by the public - which includes universities and hospitals).
I've said this before, but in my old workplace the Ladies was near the entrance, and the building was on a public right of way which was used as a shortcut at all hours of day and night (the building was open until 9.00pm and accessible with swipe access 24/7.)
The Gents was deep inside the building, and it is unlikely that anyone not familiar with it would know where to look, but it wasn't uncommon for members of the public to use the Ladies. This wasn't a big problem during the day when the place was busy (and I only saw women in there), but I would have felt very uncomfortable at night to come across a man, whether he was staff, student or a member of the public, and a young student could be very vulnerable.
Obviously unless there is good security anyone can gain access whatever the signage, but as with pubs, as soon as it is acceptable for men to use the Ladies, it is much more difficult for people to challenge someone they think might pose a risk.