B****y cheek! No manners themselves, ignore the child, and condemn you for acknowledging a friendly greeting. And if you feel embarrassed and slink out then you risk looking guilty. If they accused you of stealing a purse, you could be indignant, but such a vague suspicion is difficult to counter.
If I were feeling confident (not always the case) and had been in that situation, I would like to think that I would have paused at their table and said - loud enough to be heard by others in the cafe -
"Excuse me interrupting your chat, let me introduce myself.
While I was sitting drinking my coffee and reading my paper, your child wandered by and said hello to me. I smiled back and said hello, then went back to the paper. I have three grandchildren of my own, and much as I like children, I don't need another.
Nor do I need to be treated as a potential kidnapper for politely returning a greeting. I can give you my name and address if you would like to get me officially checked out. If not, please don't label me to all the customers in this room as a danger to children as you are now doing"
To do that would need some gall, though! We should be more assertive. why put up with it?