Generally I don't think its abusive. If your child tells you that another child is mean to you, you tell them to stay away from them, its the same. If they cause you harm (physical / emotional) you can choose to stay away.
HOWEVER it can also be abusive in other ways. My great aunt, she is an amazing lady. Her son (with a learning disability) married a bit of a controlling lady. They demanded that my aunt buy them cars, pay their bills and stock their fridge. If she didn't, they would threaten that she wouldn't be able to see the grandkids. All of her money went to them while they wasted it away. It's better for her now as the kids are grown, but that is abuse, to withdraw grandkids to make them do what they want.