To assess the association of height loss in old age with subsequent risk of hip and any clinical fracture in men late in life while accounting for the competing risk of mortality, we used data from 3491 community-dwelling men (mean age 79.2 years). Height loss between baseline and follow-up (mean 7.0 years between examinations) was categorized as <1 cm (referent group), ≥1 to <2 cm, ≥2 to <3 cm, and ≥3 cm. Men were contacted every 4 months after the follow-up examination to ask about fractures (confirmed by radiographic reports) and ascertain vital status (deaths verified by death certificates). Competing risk methods were used to estimate absolute probabilities of fracture outcomes by height loss category and calculate adjusted risks of fracture outcomes by height loss. During an average of 7.8 years, 158 (4.5%) men experienced a hip fracture and 1414 (40.5%) died before experiencing this event. The absolute 10-year probability of fracture events accounting for the competing risk of death increased with greater height loss. For example, the hip fracture probability was 2.7% (95% confidence interval [CI] 1.9–3.8%) among men with height loss <1 cm increasing to 11.6% (95% CI 8.0–16.0%) among men with height loss ≥3 cm. After adjustment for demographics, fall history, multimorbidity, baseline height, weight change, and femoral neck bone mineral density and considering competing mortality risk, men with height loss ≥3 cm versus <1 cm had a nearly twofold (subdistribution hazard ratio [HR] = 1.94, 95% CI 1.06–3.55) higher risk of hip fracture and a 1.4-fold (subdistribution HR = 1.42, 95% CI 1.05–1.91) increased risk of any clinical fracture. Height loss ≥3 cm in men during old age was associated with higher subsequent risk of clinical fractures, especially hip fractures, even after accounting for the competing risk of death and traditional skeletal and non-skeletal risk factors.