Emerging evidence suggests that Alzheimer’s Disease (AD) risk factors may differentially contribute to disease trajectory in women than men. Determining the effect of AD risk factors on brain aging in women, compared to men, is critical for understanding whether there are sex differences in the pathways towards AD in cognitively intact but at-risk adults. Brain Age Gap (BAG) is a concept used increasingly as a measure of brain health; BAG is defined as the difference between predicted age (based on structural MRI) and chronological age, with negative values reflecting preserved brain health with age. Using BAG, we investigated whether there were sex differences in the brain effects of AD risk factors (i.e., family history of AD, and carrying an apolipoprotein E ε4 allele [+APOE4]) in cognitively intact adults, and if this relationship was moderated by modifiable factors (i.e. body mass index [BMI], blood pressure and physical activity). We undertook a cross-sectional study of structural MRIs from 1067 cognitively normal adults across four neuroimaging datasets. An elastic net regression model found that women with a family history of AD and +APOE4 genotype had more advanced brain aging than their male counterparts. In a sub-cohort of women with those risk factors, higher BMI was associated with less brain aging whereas lower BMI was not. In a sub-cohort of women and men with +APOE4, engaging in physical activity was more beneficial to men’s brain aging than women’s. Our results demonstrate that AD risk factors are associated with greater brain aging in women than men, although there may be more unexplored modifiable factors that influence this relationship. These findings suggest that the complex interplay between unmodifiable and modifiable AD risk factors can potentially protect against brain aging in women and men.