My ex husband was told that I was unfaithful to him while we were married by a neighbour, on the basis of a man who was chasing me coming to our front door ONCE (not actually coming IN to the house mind...).
He never really believed nothing had happened although we did get past it. He was always a jealous type with me, and is now, with his current wife. He actually drove to check she was where she said she was a couple of months ago. This of a man in his 60's.
Ironically, my ex-husband considered (and may have been, who knows!) being unfaithful to me during our marriage. I caught him more than once with a girls phone number in his things (this was before mobile phones).
Have you considered that you contracted the virus from him? And that HE may have been unfaithful to you and hence is paranoid about YOU being unfaithful to him because he knows how easy it is to do?