It’s difficult. A man I once worked with, and got on well with, invited me and my husband to dinner to meet his wife and we subsequently became good friends with both of them. I was only part-time as I had young children, so I was at home quite a lot of the day. It was fine until the husband suddenly decided that as both his wife and my husband worked 9-5, it would be nice if he and I had coffee occasionally while they were at work. There truly was no ulterior motive as far as I could see - he behaved impeccably, and at first, it was quite nice to put the world to rights over a coffee. He adored the girls and they thought the world of him. The problem started when his (fortnightly or so) visits started getting more and more frequent. He was obviously bored. I wasn’t! I had the house to run, food to cook, and everything else to do with being a wife and Mum. It got to the stage where the girls and I would hide upstairs around the time he usually turned up, until he gave up. (It had to be upstairs because he would walk down the drive and come into the garden to see if we were there, and there was a tall side window he could look through into the dining room). When we moved to a nearby town (not because of him), I think he eventually got fed up with the journey as the visits became fewer and fewer, probably due to the 24 mile round trip if he wanted to see us.
We still saw him occasionally when we had dinners with him and his wife and that was fine, but he stopped his frequent visits, thankfully.