I see the public loo issue as symbolic of the other 'female spaces' - a metonym if you like. They are called 'The Ladies', and even have a picture of a woman to differentiate them from 'The Gents'. They have Sanpro machines next to the condom ones, and the 'nicer' ones have tights on sale and scented handcream for when you've washed your hands.
Every different issue has slightly different features and connotations, so we could argue all day over whether it matters more if men are allowed into changing rooms but not refuges, or whatever. Even then, a changing room in a busy store with cubicles and an attendant in the communal area is different from those in a swimming pool where it is noisy and people are in stages of nakedness.
I don't think anyone is saying that men don't need refuges, but many women escaping male violence definitely need somewhere they are guaranteed not to hear male voices, or where a man won't tower over their children, or be sleeping nearby. Even if that's only for the first couple of nights, it is important to a lot of women. Maybe not all, but it is the ones who do need this who should drive the agenda, not the occasional man who needs refuge.
To stick with refuges for a minute - they used to be called 'Women's Refuges', and were set up by groups of feminists who could decide for themselves who could enter. Like so many things, they were formed by women for women, and had to be fought for - whether for funding, for the right to be set up amongst those who didn't want them in the neighbourhood, for police protection and so on. I'm sure that even then there were men who needed refuge, but they didn't set them up (as far as I know, anyway). If they had, men (however they identify) could have used those, with women and transitioned transwomen in the women's ones. But they didn't. Instead, they rely on women to take them into ours.
It seems that any issue with transwomen has to be dealt with by women. We are the ones to take them into our areas, our research groups, and our sports teams and deal with any associated problems, whether we like it or not. Men (and even if you think that transwomen are women, they started out as men) just wash their hands of it, and leave women to move over and deal with their issues, with the support of women who also see any 'coping' as being 'women's work'.
For the millionth time (or for the benefit of anyone who is coming to this discussion for the first time) none of the above means that transwomen who want to live quietly as women shouldn't do so. As is always 'helpfully' pointed out, nobody checks their pants to see what is in there, so there is nothing stopping them. It is the men with their penises out and on display in changing rooms, the ones with booming voices in refuges and the ones insisting on being locked up (and locked into a cell) with women in prison who are the concern, as well as the possibility of men who are intent on getting intimate access to vulnerable women using the No Debate climate.
And as has been alluded to here, it's not just about safety. It's about women having nowhere to go to be amongst other women. Yes, there are many public loos where few people would want to hang about, but 'powder rooms' have always been where women could gather and find female support - from borrowing mascara or buying a tampon to asking for help. Arguably the most famous feminist novel of the 70s was called The Women's Room.