Although the U.S. has a very high level of socio-economic inequality, when surveyed about attitudes toward inequality in our society, respondents are not sharply critical. Many individuals express dissatisfaction with the high level of stratification in our society, but at the same time, they hold out the possibility that they themselves will one day rise to the top. Previous research has often focused on cultural explanations for American tolerance of inequality. While there is much support for this characterization of Americans' attitudes towards inequality as being rooted in a deep cultural ambivalence, we hypothesize that tolerance is also affected by structural conditions. Attitudes towards inequality are affected by local reference groups, and thus, individuals do not experience the full extent of inequality in society in their daily lives. Using the General Social Survey data, we investigate attitudes towards inequality within and between counties. We find that individuals' attitudes towards inequality are strongly affected by their relative, county-mean centered income, and that the relationship between relative income and tolerance for inequality is stronger in high-income counties. We infer that high levels of socio-economic segregation in the U.S. increase tolerance for inequality.