Does sexual fidelity matter?

If you have taken another human being (covers most things methinks) to be your significant other (perhaps even in some sort of ceremony) then I think the basic idea is that you are supposed to stay faithfull and certainly not cause any deliberate pain or upset. So yes, sexual fidelity is important. I also think - having seen the damage such things can cause - that just having some sort of private flirtation that is verbally intimate i.e. the sharing of secrets that exculdes your partner - is venturing on dangerous ground.
The old - can I tell my OH what we have been doing or talking about and know it won't upset him? - is a good guide line.

I suppose it depends on how it would affect your partner. If they would be deeply hurt then it matters.