Your memories of your grandfather's house is wrapped up in your memories of him and how it was furnished, smelt and how your grandfather occupied it.
Living in the house yourself with your belongings and family will not be the same. The neighbours will be different, the shops, the amosphere, the life. More than anything memories are of people and then their settings, not the other way round.
When my children were young we spent a week every summer staying with my aunt and uncle, my children adored it there and adored their aunt and uncle and the special things they did together. They also loved the house, which was old and quirky.
When I was sorting out the estate after my DU's death my son was applying for a job in the area and had he got it wanted to buy the house and live there himself. Thankfully he didn't get the job. As we all admit now, we all wanted to keep the house in the family because of all the happy memories of its occupants.
This summer, DH and I revisited the area, some 10 years after DU's death. It was lovely revisiting an area we had got to know well over the 35 years our relations lived there. It was almost unchanged and we will go back again, drive past the house - and at the end of the holiday go home. The area is 100s of miles from family and friends, difficult to access, and a long way from amenities we want in our lives.
We are very glad now that DS didn't get the job, or buy the house, it was an unsuitable house for him. We have realised the truth of the saying; 'The past is another country, they do things differently there'.