Frankly, there is no way you can solve this problem without either resorting to lies or by being a good deal more forthright than you probably feel is polite.
Yes, you could leave it to your husband to deal with, if "putting his foot down" means that he will do so, and wasn't just telling you what he wanted YOU to do. which is what my DH would have done.
If this lady's mental health issues are something that she or her husband have talked freely to you about, I suggest next time they turn up that you leave your husband to talk to the lady, and you draw her husband aside and say that you quite honestly cannot do with these frequent visits.
They interupt your day, tire you and your husband, and as far as you can see do your visitors no good either.
If he protests that his wife needs to get out of their house, I suggest you say you are sorry, but that is not your problem and will he please find somewhere else to accompany her to.
You are feeling worn out and quite honestly inconsiderately treated by them never taking a hint that their visits are inconvenient.
If he is offended, so much the better, as that will result in your not seeing them again, or as much at least.
If on the other hand they keep on coming, simply do not let them in, open the door, say, "Sorry, you have come at a bad time, ring or text next time." and close the door firmly.
If they just walk in, either consider keeping your outside doors locked, or put on you boots and coat and say, "I must ask you to leave, I am on my way out, as you see."
Escort them out and go for a short walk yourself.