Talking of smells, I live in a block of flats with a communal entrance. You can always tell who was the last person to walk down the passage by the smell they leave behind them.
Maybe youre a "super smell recogniser" like the super facial recognisers the police use?
(Sorry for Tangent OP)
I love the whiffs of cooking smells coming from different homes in the early evening it makes me imagine cosy scenes
BUT if the level of draught between the 2 houses is that extreme that you feel like youre in their kitchen when they cook I would worry about 2 things: the speed in which a fire/smoke would spread between the houses. Being semi SHOULD buy next door time to get out but not if there are significant gaps.
And 2, less importantly, insulation
So I would get it seriously looked at. As another poster said the "weak link" is usually between ceiling and floor so some degree of disruption may be needed to get to the root of it and fix it properly