You go out usually with him and his wife, presumably she thinks you are a friend to her because not many wives would happily tolerate the presence of someone else on outings. Even if nothing happens between you and her husband, yes, her husband, yOu are abusing her trust by continuing with this relationship.
My mother was in this position and it broke her heart when she found out the truth.
Please think about what you are doing. He has made it clear no kisses, take it that he has no intention of deceiving his wife. Dont put temptation in his way. Find other friends, join a group, go out , enjoy yourself. Distance yourself and this infatuation will fade and perhaps put yourself in her position, how would you feel?