If the balance of 2nd home owners, vs people who live in small communities full time is too high, the infrastructure suffers; their children are not attending the village schools and so numbers drop and schools close (this has just happened in Abersoch where 40% of homes are owned by 2nd home owners). Local people, who usually earner lower wages than those in cities, can't afford to live near their workplace and so have to commute further and further out. If people aren't living in the houses 52 weeks of the year, the local shops, such as butchers, bakers, greengrocers only have a "season" to trade and over the winter months can't trade enough to survive. If the shops close, they're not paying any local business rates which, in turn, affects the budget of the town council and what social services it can provide. Close knit communities aren't sustainable when 40% of the people who live there are transient; they're not there to support parish councils, local charities etc.
I come from a small community in North Wales and I've seen first hand how, once the summer school holidays have ended, it's like a ghost town. And every April, the council tax goes up higher and higher for those locals who are grimly hanging onto their family homes by the skin of their teeth.