You asked how we feel about this man's behaviour and I think the posts above say it all...but the important thing is really where you DD goes from here.
From what you say there is no problem with child care so the children have no need to be left with him. I'd say there is a simple solution - only allow him very occasional access to his GC and then only when you and/or your DD is there.
No need to make an issue of it just sort of make that how it is.
๐ฆ Locked down no longer but still firm friends ๐ฆ