Although projections from cross-sectional studies have shown that bone loss leading to osteoporosis begins around menopause in women and later in life in men, this has not been examined longitudinally in population-based studies using high-resolution technology capable of distinguishing cortical (Ct) and trabecular (Tb) bone microarchitecture. The aim of this 3-year prospective study was to investigate age- and sex-related changes in bone compartment–specific geometry, volumetric bone mineral density (vBMD), microarchitecture, and estimated strength. The distal radius and tibia were imaged at baseline and after 3 years (median 3.0;​ range, 2.7 to 3.9 years) using high-resolution peripheral computed tomography (HRpCT) in an age- and sex-stratified, population-based, random sample of white men and women (n = 260) aged 21 to 82 years. In general, at the radius and tibia there was a moderate annual increase in cortical thickness (Ct.Th) that seemed to offset the increase in cortical porosity (Ct.Po), resulting in net annual increase in cortical vBMD (Ct.vBMD) in premenopausal women and young men. With advancing age, postmenopausal women displayed significant bone loss with decreased trabecular vBMD (Tb.vBMD) (due to loss of entire trabeculae) and Ct.vBMD (manifested as increase in Ct.Po and decrease in Ct.Th) at the radius, and a decline in Ct.vBMD (with increasing Ct.Po) at the tibia, resulting in loss of estimated bone strength. In contrast, men had a lower rate of bone loss with advancing age with smaller increases in Ct.Po at both the skeletal sites. In summary, the pattern of bone loss in men and women was discrepant, with women losing more bone than men with aging, although with a dominance of cortical over trabecular bone loss at the peripheral sites in both sexes. This conforms to epidemiological evidence that most fractures occurring in old age are predominantly at cortical peripheral sites, with women having a higher incidence of fractures than men at any given age.