To me your husband's behaviour is a form of infidelity, and whether you can or should condone it, is not really something I feel able to advise you on.
In your place I would find it hard to decide too, and my list of pros and cons would look like this (I think)
Can we afford to divorce? In financial terms that is?
Do I want to divorce him? Emotionally speaking.
Would I miss him so much that life would be difficult, if I left?
Am I prepared to go on living without sexual fulfilment, because his impotence remains a factor?
Or if not, am I prepared to find a lover and remain with my husband? (Could we both accept this solution?)
Is his impotence really uncurable?
Or is he just not interested in me any more?
I don't know if these points are of any use to you, but if even one helps a little, then that's fine.