Step back and look at a primary residence as a line in your portfolio. It is a single asset that is concentrated (often the largest position you own), leveraged (bought with a mortgage many times your down payment), illiquid (sold in months, not minutes, and never in pieces), and undiversified (one building, one street, one city). No competent investor would deliberately build a position with all four of those properties at once. Most homeowners do — and call it prudence.
Worse, the position usually carries double exposure. Your home’s value is tethered to your local economy — and so, very often, is your paycheck. Suppose you work in the dominant industry of a one-industry town or region. Your salary, your unvested equity compensation, and your career prospects already ride on that industry’s fortunes. Buy a home there and your housing equity now rides on the same variables. A downturn in that industry can take your job and your home equity in the same quarter — precisely when you can least afford either loss, and precisely when you may be forced to sell into a falling market. That is the textbook definition of a fragile position: correlated risks stacked on top of leverage and illiquidity.
This does not forbid buying. It does mean that the more your livelihood depends on your local economy, the more cautious you should be about also betting your net worth on its real estate — and the more the rest of your portfolio should be diversified away from it. Your house and your human capital should not be the same wager. The portfolio-level treatment of this concentration appears in section “Housing’s Role in Optimal Portfolios”.