While fractures of the distal femur are often considered as fragility fractures, detailed knowledge of the bone microarchitecture at this skeletal site is largely unavailable. Initial evaluation of a patient cohort with distal femur fractures showed a markedly increased occurrence in elderly women. The purpose of this study was to determine the extent to which demographic characteristics of distal femur fractures are reflected by general age- and sex-specific variations in local microarchitectural parameters. Fifty cadaveric femora were collected from 25 subjects (12 females, 13 males, age 25–97 years). A volume of interest within 3 cm proximal to the condyles was analyzed using high-resolution peripheral quantitative computed tomography (HR-pQCT), which revealed impaired trabecular and cortical bone microarchitecture in women compared to men as well as in osteoporotic compared to normal or osteopenic subjects, as classified by dual-energy X-ray absorptiometry (DXA) T-score. Linear regression analyzes showed negative associations between age and HR-pQCT parameters in women (e.g., cortical thickness −14 µm/year, 95% CI:​ −21 to −7 µm/year), but not in men (e.g., cortical thickness 1 µm/year, 95% CI:​ −12 to 14 µm/year). HR-pQCT parameters showed strong positive associations with areal bone mineral density (aBMD) determined by DXA at the hip in both sexes. Taken together, our findings suggest that female sex, advanced age, and low aBMD represent major risk factors for impaired microarchitecture at the distal femur. Both the diagnostic value of DXA for predicting distal femur fractures and the efficacy of bone-specific agents on fracture risk reduction should be investigated in the future.