What mcem (and others) said - he can't be allowed to continue to call the tune in this way. Contact should be about the children, about meeting their needs. Children need routines, they benefit from knowing which day they'll see daddy. Only knowing the day before or minutes before leaves them in limbo.
Is this a pattern that was set in their relationship, that is he makes the decisions about what happens and when it happens. If so, it may help if they get some mediation to help them discuss his contact to the children in a neutral environment.
It's also totally unfair to expect you to dance to his tune. I'd suggest as mcem does, that you set a day and a time and stick to it. It can only be re-negotiated when its impossible to avoid.