This thread has left me with more questions than answers tbh. First, if my husband absolutely did not want us to socialize with the new neighbors, then we would not socialize with the neighbors. He is a reasonable person, he'd do the same for me and he comes first.
But I didn't understand here if that was what he said or if he just said he does not want to interact with the man, but does not mind if you maintain contact with the woman. In that case, I don't see a problem with it if you like her, though maybe you wouldn't want to become besties with her (besides, as has been mentioned, it can be awkward when a friendship with a neighbor doesn't work out).
I too am curious about what the man said because as has also been mentioned here, some people are just a little sheltered and clueless and don't even realize they've said something racist. I might give some leeway there IF the husband would want to reconsider.
I absolutely do not blame the husband (or the man's wife) for not speaking up right then. Sometimes you can be too stunned by an unexpected zinger to react immediately and then it's usually far better imo to err on the side of keeping one's trap shut lol. It's also perfectly reasonable to make your decisions about someone privately as you get to know them. You are not obligated to discuss anything with them or try to work anything out with someone you just met; that would be just one option that you have.