Of course they can refuse, but it doesn't mean someone won't go ahead. We tend to think of rape as involving one man and one woman and, maybe a difference of opinion about consent. There is no difference of opinion about a young man and woman gazing at the stars, cuddling and kissing romantically and then a dozen or more young men swarming over a hill, knocking out a boyfriend and raping the woman.
Similarly, you can say no because your partner is violent and you want him to go. He is stronger than you are and your child is asleep in the room next door. He is angry and wants to hurt you. Anal rape is not a bucket of laughs, even though his has been your "loving" live-in partner for some years. Then you are so pathetic and under his thumb that you pull you clothes on and meekly walk round to the all-night garage to buy another packet of cigarettes.