Welcome to GN 1cr123. Your pain, anger and frustration are clearly visible in all your posts and as Mandrake has said "the question is whether you can let that go" and I would add, do you want to let it go.
Your son is 7 so for 7 years has had a relationship with his GP's. Have you considered how confusing and upsetting it will be for him if he has his GP's taken away?
With regard to the lost pencil, as has already been said, it would have been better if you'd played it down. Told your son not to worry, 'gran and grand dad are a bit fussy about this sort of thing but don't worry, we'll buy another pencil'.
I agree with Mandrake in that you are projecting your own feelings on anxiety onto your son who may not experience this type of behaviour the way you did when you were a child.
He doesn't live with them, he doesn't have a sibling who bullies him and isn't prevented from doing so by you, his parents. He has the love and security all children need on a daily basis at home.
I suggest you start thinking of some coping mechanisms. When your dad arrives to watch his GS play football for eg., rather than resent this, see it as a positive. He sees his GS, you can exchange a few pleasantries and when the match is over, he and your son can spend a few minutes together before you all go home.
A good counsellor will provide other suggestions for the best way to manage the relationship. Walking away/estranging IMO should be the last resort especially as there's an existing relationship between your son and your parents. As for the suggestion that you should think of your father as dead, is your son expected to do the same about his GF?
Apologies if I've got this wrong, but I have the impression that your parents see your son without you, taking him to see your brother who used to bully you, and who you had no contact with for 20 years.
As I've said, I may have got that wrong but if not, it does seem a very odd thing to do. If this is the brother who bullied you and who you have no contact with, how do you know he agrees with everything you say about your parents, and with your obvious concerns about your parents, why do you allow them to take your son to see his uncle?