We aimed to comprehensively evaluate the association of body composition with fracture risk using longitudinal data from a Swedish cohort of 44,366 women and men (mean age of 70 years) and a subcohort of 5022 women. We estimated hazard ratios (HRs) of fracture for baseline body mass index (BMI), BMI change during the prior 12 and 18 years, baseline waist-to-height ratio, total and regional distribution of fat and lean mass, with and without areal bone mineral density (BMD) adjustment. During follow-up (median 8.7 years), 7290 individuals sustained a fracture, including 4279 fragility fractures, of which 1813 were hip fractures. Higher baseline BMI and prior gain in BMI were inversely associated with all types of fracture. Lower fracture rate with higher baseline BMI was seen within every category of prior BMI change, whereas higher prior BMI gain conferred a lower rate of fracture within those with normal baseline BMI. Each standard deviation (SD) higher baseline waist-to-height ratio, after adjustment for BMI, was associated with higher rates of hip fracture in both women and men (HR 1.12;​ 95% CI, 1.05–1.19). In the subcohort (median follow-up 10 years), higher baseline fat mass index (FMI) and appendicular lean mass index (LMI) showed fracture-protective effects. After BMD adjustment, higher baseline BMI, total LMI, FMI, and higher prior BMI gain were associated with higher fracture rate. Baseline fat distribution also was associated with fracture rate;​ a 1-SD higher android to gynoid fat mass ratio in prior BMI gainers was associated with BMD-adjusted HRs of 1.16 (95% CI, 1.05–1.28) for any fracture and 1.48 (95% CI, 1.16–1.89) for hip fracture. This pattern was not observed among prior BMI losers. These findings indicate that for optimal fracture prevention, low baseline BMI, prior BMI loss and high baseline central obesity should be avoided in both women and men.