Dating someone with bipolar

I’m hoping for the experience of other people regarding being with someone with bipolar disorder. I have been married to a lovely man for 18 months. We have lots of interests in common and apart from this have a good marriage. He told me soon after we met that he has a diagnosis of bipolar disorder as his father had. I believe his mother left his father in her 60s when the children were all grown up because she found his mood changes intolerable. To start with I noticed he could sometimes be quiet and introspective and it gradually got worse. A few months in to the relationship I found that he often promised things and is very optimistic about our marriage and sharing the future with me and then he changes out of the blue with no reason I can see. He becomes very withdrawn needing time alone and he is very negative and depressed. He does go to his GP and has a therapist. When he is down any plans we have made such as events and holidays are cancelled. I have to leave him alone for a few days and wait for him to come out of it and start talking to me again. What really upsets me is when he is like this is says he feels numb and that he is unable to love another person and does not feel able to be in any relationship. Then he will get better and go back to his upbeat and normal self. I know that this is down to his bipolar disorder. To start with I was very accepting of this and patient but recently it is becoming very hard for me and it is making me very unhappy. My friends and family are worried about the impact this is having on me and even a close friend of his who he has known a long time has advised me to walk away, he knew him when he was with his first wife and they split up when their children left home for university. Does anyone else have experience of being with someone who has bipolar and does any of this sound familiar? I feel I have tried everything to make things better but I also wonder if there is something I could do to manage his ups and downs better. We have tried marriage counselling but it didn’t help.