I deal with this kind of thing on a daily basis at work. From experience, there is the mother's side of the story, and the father's side of the story, and somewhere in the middle, is what is actually happening, so forgive me summermary if I don't take everything you say at face value. The father isn't here to defend himself.
There is no such thing now as a Residence Order. Now there is a Child Arrangement Order, which states where the child lives and what contact the non resident parent will have with the child.
You do not need a solicitor to apply for one. You can go to the Court yourself and apply for one. I think it is a form C100, but there is a lot of information if you Google "Child Arrangement Order", including on the .gov website.
Does your grandson have an official diagnosis of his special needs?
I don't think you've said exactly what contact the father and son have had since the cruise work stopped, but I would think it needs to be built up to maybe every other weekend, with one evening/overnight a week, so that they can both get used to each other.
I don't think you've said how far apart the father and child live, so that will need to be taken into consideration and you haven't said if the father has parental responsibility (i.e. is his name on the child's birth certificate). If so, and in the event of your daughter's death, then the child would go and live with his father, so it would be in everyone's best interests to have as good a relationship in place as possible, and that's not going to be achieved if they don't see each other much.
This Cafcass scheme is one that we refer parents to a lot from work. I don't know if it got rolled out to the whole country, but it might be worth looking into.
www.cafcass.gov.uk/grown-ups/supporting-separating-parents-in-dispute-pilot.aspx
I think the most important advice is to not be so embroiled in your own animosity towards the father that it becomes more about point scoring, and not letting him have it all his own way, than about doing what is right for the child. Of course, you are going to say that all you want is what's right for the child, but in so many cases, that also means mother getting what she wants at the expense of father. I bet if the father was on this thread and having his say, his words would be much the same.
But never forget that there is a little boy in the middle of this. You haven't indicated that he has been upset when going to see his father and paternal grandparents - although you have used my most hated word "bullied", but not said what the paternal grandmother has done that is considered bullying. You seem to be basing the decision to limit contact as much as possible on your daughter's feelings towards her ex. It really isn't about her, and it isn't about him either. It is what is best for the little boy. And just because the mother doesn't like the father, or fears the father, or thinks he isn't doing things the way he should, that doesn't mean he doesn't care about or love the child, or that he should be kept away as much as possible.
You haven't said anything to make me think that contact should take place in a contact centre. There's no indication of the child being neglected or abused in father's care, so I would think hard about making them meet in a contact centre, where it is not comfortable familiar surroundings for either of them, especially when the child has been used to going to a home environment.
Have a look at www.childrenslegalcentre.com/ - there is a lot of information there and a free advice line if you can't find what you are looking for on their website.