I don't have a medical background, but as a teacher I have often been aware of the kind of situation your family is in.
I think for the time being you will have to accept that the child's mother can dictate that he is only with you if she visits as well.
It sounds as if she either is jealous of the child's attachment to you, or insecure and feeling that her son prefers you to her. These feelings are difficult for both you and her.
The only way to cope is to try to make her feel more secure as a mother by trying to do things that include her when you are all together.
Obviously, we don't kno w the whole story - are you living in the UK, or somewhere else?
There has been a very extreme situation for the social authorities to remove a baby from h is mother and place him in care. It is good that the child was placed with his grandparents rather than with strangers, but even so, his mother is probably terrified that he will be taken away from her again.
You say the child's father was the problem and has now been deported, so obviously there was something very unpleasant going on, which I am sure must be difficult for both you and your daughter to get past.
Concentrate right now on improving your relationship with her as the child's mother by making it clear that you are not trying to usurp her position as his mother, nor are you a spy for the social authorities who decided originally to place him in care.
I hope you can reach an agreement so the child still is able to visit, but please, try to include his mother to start with.