Bone metabolism in men is in part determined by sex steroid exposure. This is especially clear during puberty and senescence but it remains to be established whether declines in sex steroid levels during young and middle adulthood are associated with changes in bone mass and size. This study investigated changes in bone mineral content (BMC), areal bone mineral density (aBMD), volumetric BMD (vBMD), and bone size in relation to sex steroid levels in 999 young adult men (age 24-46 years) of whom 676 were re-evaluated after a mean period of 12 years. Sex hormone–binding globulin (SHBG) levels were measured using immunoassay, testosterone (T) and estradiol (E2) using liquid chromatography–tandem mass spectrometry (LC-MS/MS), and free fractions were calculated (cFT and cFE2, respectively). Areal bone parameters and BMC were measured at the hip and lumbar spine using dual-energy X-ray absorptiometry (DXA). Radial and tibial vBMD and bone size were determined using peripheral quantitative computed tomography (pQCT). Linear mixed models were used for statistical analyses. With aging, we observed decreases in almost all bone mass and density indices, whereas changes in bone geometry resulted in larger bones with thinner cortices. These changes in bone mass and size appeared related to sex steroid levels. Specifically, decreases in cFT (but not total T) levels were associated with larger decreases in lumbar spine BMC and especially with geometric changes in cortical bone at the tibia. Similarly, decreases in total E2 and cFE2 were associated with larger decreases in bone mass (all sites) and also with some geometric changes. Also increases in SHBG were independently associated with aging-related changes in bone mass and size in these men. In summary, even small changes in T, E2, and SHBG levels during young and middle adulthood in healthy men are associated with changes in bone mass and size.