Oh dear, ooonana. Sounds like he's afraid to give you time to think. Well, he can go where he wants, but that doesn't mean you have to spend a lot of time w/ him. I would either give him a couple of dates when he can see me and that's that or I would wish him a good time but let him know I won't be spending time with him.
If you choose the former, imo, you should limit the amount of time, too. Decide beforehand if you want to discuss the situation and stick to your decision. If yes, then hold his feet to the fire and insist that he discuss his issues with you or the visit is over. If no, you're not ready, don't want to spoil your visit with dd, etc., then let him know you're not talking about it till you're home and end the conversation if he brings it up.
Really, imo, he has a nerve!