Do you take the baby out or only look after him and his brother indoors? Depending on what time they are dropped off if the weather is reasonable perhaps you could take them out, soon after arriving. Little ones of that age often enjoy watching the world from their pushchairs and it can act as a good distraction and he might associate it more with enjoyment rather than being abandoned by mummy/daddy. By the time you get home again hopefully he would be happier.
Is it rude to not finish a book club choice that was selected by someone else?
