You are absolutely right to stick to your guns, joolsangel. We had a similar dilemma after my father died and my mother re-married. He seemed like a nice, caring man to begin with but after a year or so we realised his drinking was way out of hand and my mother, not a heavy drinker herself, chose to turn a blind eye.
One evening he picked our son up from Cubs (without our permission) and drove into a ditch. Fortunately DS was not hurt and was able to run home as the 'accident' happened just up the road from our house. From then on we refused to let them have the children to stay and if they visited us we picked them up and drove them home. I felt sorry for my mother because her husband was excluded from all family occasions other than at our house so I ended up hosting endless get-togethers so she could see the rest of the family.
I never doubted we were doing the right thing. It often caused a lot of bad feeling but you cannot put children at risk. You DH will have to realise it is better to be safe than sorry.