My stepdaughter has a cat that takes holidays with neighbours. The first few times he went, she searched high and low for him, then she picked up on his game!
He'll happily stroll in like the king of Siam, expecting everything to be where he left it - though he did get a shock the first time as, fearing he'd died, she got rid of everything of his!
He may stop that now, though, as they have a kitten now, too, so he may decide to stay put!
But, he is a cat and cats do wander, and they happily stroll into other cat welcoming houses and eat there so, on this note, YABU!!!
Sorry, but if you didn't want him wandering you should have made him an indoor cat!