Poor Flakesdayout ... it's a bit of a dilemma, isn't it? However I think you need to decide whether you are better off on your own or with this man. If you want to stay with him then you will just have to put up with the loneliness in your own home, abandonment every weekend, and insults (in front of your neighbour, quite unforgiveable IMO). If you prefer to live alone and be happy then that's fine but in that case you need to consider your course of action.
You say he quickly moved in with you but don't make it clear whether you are actually married or just living together. If the latter, it's easy. Put the house on the market and start looking for a smaller place for you to live on your own (maybe close to your sons/friends) ... once the sale goes through pay him back any money he may have invested and get on with a happy life, with the chance of meeting someone who shares YOUR interests. If the former, and the house is still in your name only, see a solicitor and make sure your finances are secured and safe from him before you start any proceedings.
I have to say that, in your place, I know exactly what I would do and it doesn't involve any motorbikes!