Nothing is wrong, it is just something you have to agree between yourselves,
The two people concerned have to sit down and each explain to each other why one has decided to not make the journey and the other is upset at this decision. Listening and understanding is essential, shouting and temper losing is forbidden.
For example, it may be something as basic as the older man having prostate problems he has never really admitted to, and he doesn't want to risk the journey and service, in case he has an accident. The younger man may be seeing this as rejection by his father, because of past history when he was a child.
Issues like this only come out and acceptable compromises are only brokered if the people concerned can sit down and calmly explain why they think like they do, listen to the other person and then reach an agreement satisfactory for both.
Right or wrong doesn't enter into it.