Across many situations, it seems that individual differences ( and possibly gender differences) account for a large part of potential misunderstandings and hurt. We all tend to view the actions and words of others as though the motivations and emotions behind them are identical to what ours would be had we acted or spoken that way. In many situations, that is miles off the truth.
There is literature around the five ( sometimes seven) languages of love, describing how physical contact may be one person's main way of expressing love and their partner's may be gifts or words of affirmation. Initially aimed mainly at couples in a romantic relationship, it has relevance for other relationships as well.
Definitely celebrate the fact you do have a relationship with your son, even if his expression of it differs from the ways ( and emphases you might give to each)in which you might express your love for him.
It's not just a gender issue, but I know that can definitely be part of it.