Enhanced knowledge regarding modifiable risk factors for hip fractures are warranted. We aimed to study the associations between two indicators of physical fitness (resting heart rate and level of physical activity) in middle-aged individuals, and the risk of hip fractures during the subsequent three decades. Data on objectively measured resting heart rate and self-reported leisure time physical activity from a national health survey (1985–1999) was linked to a database including all hip fractures treated in Norwegian hospitals from 1994 through 2018. We calculated hazard ratios (HR) with 95 % confidence intervals (95 % CI) for hip fractures according to categories of resting heart rate (mean of two repeated measures), and leisure time physical activity level in adjusted Cox proportional hazard models. In total, 367,386 persons (52 % women) aged 40 to 45 years were included, of whom 5482 persons sustained a hip fracture during a mean follow-up of 24.8 years. Higher resting heart rate was associated with higher hip fracture risk. Men with a resting heart rate above 80 bpm, who also reported low levels of physical activity, had a HR of 1.82 (95 % CI 1.49–2.22) for hip fracture compared to men with a resting heart rate below 70 bpm who reported high levels of physical activity. The same measure of association for women was 1.62 (95 % CI 1.28–2.06). Physical fitness measured by low resting heart rate in middle age, and a high physical activity level were associated with a lower long-term risk of hip fractures in both men and women.