In answer to the question, an emphatic Yes. Of course it matters, assuming there is love, respect, morals, ethics and loyalty between the two people concerned.
I think HildaW has the right idea and the litmus test she highlights is a good guideline. I guess if we do not believe that sexual fidelity is important then perhaps we should not marry or co-habit with anyone who does as that is just cruel and hurtful should the opportunity and decision ever be taken to have a 'fling'.
For my part I find the idea abhorrent and beneath contempt but each to their own. I do not expect everyone to have similar values to me but I would expect honesty. As a victim of infidelity I would not encourage the act.