Lily I haven't been to a 'dinner party' for years but I do still go to formal dinners occasionally and I think it's only good manners to talk to both the people you're sitting next to!
As regards the ladies withdrawing, one of the biggest hurdles of my undergraduate training was going to the Professor of Oral Surgery's house for dinner, with the other 'dresser' as we were known. The Prof was ex-army and truly scary, and the evening consisted of a viva on the maxillary antrum, sherry, then dinner.
Students from previous years would regale us with details of the various faux pas to be avoided, and we were warned that after the meal was over, the students would be asked if they would like to 'go upstairs' ie use the loo.
When it was my turn I dutifully tried to leave the room, only to find the door knob had come off in my hand 

I couldn't help suspecting that it had been deliberately loosened by the Prof as a test of my social savoire faire