Men tend to report higher health-related quality of life (HRQOL) than women of the same age, despite higher mortality rates and lower life expectancy. Social support is one factor related to HRQOL that may contribute to the observed gender difference. The purpose of this study was to determine the factors associated with HRQOL, especially levels of social support, and variation by gender. Multivariable logistic regression was used to analyze data from 4,498 men and 6,948 women participating in an ongoing community-based cohort study in Washington County, Maryland (CLUE-II). The results showed that men reported significantly better HRQOL than women. Men reported having higher levels of social support than women, but the magnitude of the association with HRQOL was similar. Having two or less close friends was associated with a statistically significant increase in the odds of reporting poorer HRQOL compared to having 10 or more close friends among both genders (men: OR = 1.49; women: OR = 1.53). Differences in the level of social support did not explain the gender difference in HRQOL. Our findings show that social support is an important correlate of HRQOL for both men and women. However, the gender differences in HRQOL are not explained by social support or the other factors examined.