Identification of risk factors for fractures is important for improving public health. We aimed to identify which factors related to physical activity and psychosocial situation were associated with incident fractures among 30,446 middle-aged women and men, followed from 1991–1996 to 2016, in a prospective population-based cohort study. The association between the baseline variables and first incident fracture was assessed by Cox regression models, and significant risk factors were summed into fracture risk scores. Any first incident fracture affecting spine, thoracic cage, arms, shoulders, hands, pelvis, hips, or legs was obtained from the National Patient Register, using the unique personal identity number of each citizen. A total of 8240 subjects (27%) had at least one fracture during the follow-up of median 20.7 years. Age, female sex, body mass index, previous fracture, reported family history of fracture >50 years (all p < .001), low leisure-time physical activity (p = .018), heavy work (p = .024), living alone (p = .002), smoking (p < .001), and no or high alcohol consumption (p = .005) were factors independently associated with incident fracture. The fracture risk score (0–9 points) was strongly associated with incident fracture (p for trend <.001). Among men without risk factors, the incidence rate was 5.3/1000 person-years compared with 23.2 in men with six or more risk factors (hazard ratio [HR] = 5.5;​ 95% confidence interval [CI] 3.7–8.2). Among women with no risk factors, the incidence rate was 10.7 compared with 28.4 in women with six or more risk factors (HR = 3.1;​ 95% CI 2.4–4.0). Even moderate levels of leisure-time physical activity in middle age are associated with lower risk of future fractures. In contrast, heavy work, living alone, smoking, and no or high alcohol consumption increase the risk of fracture. Our results emphasize the importance of these factors in public health initiatives for fracture prevention.