Highly desirable places also have many second-homes as well. In Hawaii, some neighborhoods are only 20% occupied because most of the owners actually live somewhere else.

I am sure second-homes also drive up prices (In Vancouver they even have a special tax for empty houses).