Mamou, my first thought on reading your OP was "Does he have ADHD or similar?", but I think that you answered that question later in the thread. I sympathise with your situation (especially the getting up at 5 am) as I have ADHD (diagnosed bar one) grandkids and a niece, and they are/were totally exhausting, even for an afternoon. I have no qualifications in this field, so what I have put is from my observations and experience of them, along with what their parents have told me.
What their parents found was that lots of energetic activity made the kids even more energetic and restless, and that they needed downtime to help their brains relax. Fiddle toys proved useful as they always need to be doing something physical, however small, to aid concentration.
DGSs still enjoy "Granny-time", even though they are now 11 and 15 As 3-year olds, a trip out for a drink and a cake after a walk to the local cafe or a bus ride filled the need for exercise and downtime. In the case of one, it was a bus ride to the local museum which was interactive and child-friendly - lots of buttons and levers, and covers to look under - and an ice-cream afterwards. (We became great friends with the stuffed polar bear in the arctic display!)
A trip to the local play area (plus a ball) was always good. Yes, I got exhausted pushing the swing or coping with balls coming in my direction, so when I was approaching my limit of physical activity, I just let them know that Granny needed a rest, or, if it was nearly a meal time, "5 more pushes" and when the swing stops, it's time to go! Then we could have a bit of screen-time once back. I still miss Dora the Explorer and The Night Garden 
Showing me how (well) they could skip, hop, run etc meant that I didn't have to be very active myself! One ADHD DGD, now 8, spent some time recently showing me how well she can hula-hoop, so that still works even as they get older!
All have enjoyed drawing and painting, but one has needed a "fidget cushion" to sit on or he literally still can't sit at a table long enough.
I am not in any sense a maternal type, and have always had physical limitations, but I have enjoyed the company of my DGCs, even if not the energy required!
It sounds to me as if you are too tired/exhausted to enjoy your DGS, and perhaps you need to talk to his parents and say you can't manage it for so long. They need to look after you lest they find you can no longer manage it! My family has been very good about making sure visits and babysitting wouldn't be too much for me. Maybe they could suggest things you could do or easy meals. It also occurs to me that if you could stay at his house to do the babysitting, he might be more settled. On the other hand, they may be using a visit to you to recover from their exhaustion!
As far as possible, the important thing is to enjoy time with your DGS, so make sure you have control over activities and do things you both can enjoy. Make sure you get enough rest, and do as another Gransnetter put about what time is getting up time. A special Granny clock could work wonders! And would a slightly later bedtime be an option? We have the rule that "What happens at Granny's house stays at Granny's house"!! So a later bedtime could be a treat, and might mean a later start time!
Do take care of yourself, and my apologies for such a long post.