Bravo those last few posters for saying get your husband on your side, please don't fall out with him over this and damage your marriage.
Your baby will grow up all too quickly, so cut down on these visits - if your husband is arranging the behind your back and delivering the as a fait accompli tell him No, you're doing something else that day (work out a few believable outings such as a clinic check-up, visit from a friend, picking up a parcel from post office and keep them ready to use) unless you can dredge up the courage to simply point out you can't cope with a visitor today.
I have to admit it was something I was never strong enough to do. This will pass.
AIBU The problem I have with my husband exercising in our flat
Is there any reason why children should not repeat a school year?