I agree that you and your husband should set your boundaries and stick to them but I also feel that your comment about your children preferring them to you is a bit of a red flag. It suggests that you have low self esteem or don't feel confident in your parenting abilities. Maybe you need a little talking therapy to help you find a way to raise your perception of yourself.
As with all things there are lines in the sand that nobody should cross which you and your husband should be able to agree upon. One should be that it is your child, your rules. The other should be that you are never undermined to your children. Those two things encompass most of the problems parents are likely to experience. Perhaps when your in-laws are overstepping, your husband could very quietly say something like, "We have discussed this as parents of our child and we are agreed that this is the way we are going to do things." He should do this when you are there and when you are not. He can shut down any discussion by just telling them that it is not open to negotiation.
As a grandparent, I know that some of the things I have said that might have been misinterpreted as a criticism were observations and it seems to me that this is par for the course for relationships when a baby enters the equation. I've learned to keep my mouth shut but I have always said to my children that the best advice I can give is the sort that can be ignored if they want to and I won't take offence. It is a learning curve for Grandparents too, not just for new parents, so mistakes will be made and working through those mistakes will make future children so much easier. Framing things in a nice way will make life smoother too. A "we love your visits but would you mind phoning first?" can soften the blow. If they don't call first, a calm but firm refusal at the door (no crossing that boundary!) "Oh I hate to give you a wasted journey. If you had rung we could have told you, we are just off out. Could you ring tomorrow and we'll arrange a visit." You probably wouldn't have to do this often and they'll get the message. They won't like it at first but they'll get used to it and when you say other things, they'll be prepared for you mean it.
Good luck.