In a thread recently someone quoted from the Bible, "He who has looked at a woman with lust has already committed adultery with her in his heart" It was quoted to justify condemning anyone who enjoys a bit of banter and joking, so nearly all of the responses were to say that a bit of banter is not what was condemned there.
BUT perhaps you should think a bit about it. In fantasising about this man and depending on him as your emotional outlet, you are skating dangerously near to "looking at him with lust" and the boundary between that and being drawn into a relationship that puts his marriage into potential disaster is not difficult to cross - accidentally, maybe, but crossed even so. It is most unfair of you to risk doing this to him - and to his wife, who doesn't seem to feature as high in your list of friends.
If you have any consideration for her, or a conscience, ease back a bit in your contact and look around actively for somewhere else to find the closeness that is lacking in your life. Can you volunteer to help vulnerable children or adults who need some love in their lives? You have love to give, there are people out there who have no-one of their own to provide it.