My eldest GC is reserved, was very shy as a young child and used to hold back from us and hide his face, even though they only lived round the corner and we saw him very frequently. He's always been... how can I put this? ... less than animated around adults, speaking in a monotone, mumbling, being very economical with his words. Asking him if he's excited about going on holiday used to elicit little more than a flat 'yes' spoken very quietly, or even just a "mmm", without making eye contact. I used to feel just like you Debido, that I couldn't break through this barrier and make a connection with him. And yet now he's older he's gained in confidence and he and I can sit and have long conversations about all sorts of things. I love being with him and really enjoy his company.
I just had to wait several years for that lovely close relationship which our other GC found so easy from a much younger age. So don't be too despondent - in time he may well change and become closer to you.