In view of his behaviour, I don't think the idea of staying overnight should come into it.
Do the DGC like spending time with him? If not, why persevere with it? If they do, then daytime visits only would seem like the best way.
Some good suggestions from Eloethan.
What age did you reach your current height?
and 

None of us believe he would be violent with the children but I take all the points above.