Osteoporosis and fractures severely impact the elderly population. Polygenic risk scores for bone mineral density have demonstrated potential clinical utility. However, the value of rare genetic determinants in risk prediction has not been assessed. With whole-exome sequencing data from 436,824 UK Biobank participants, we assigned White British ancestry individuals into a training data set (n = 317,434) and a test data set (n = 74,825). In the training data set, we developed a common variant-based polygenic risk score for heel ultrasound speed of sound (SOS). Next, we performed burden testing to identify genes harboring rare determinants of bone mineral density, targeting influential rare variants with predicted high deleteriousness. We constructed a genetic risk score, called ggSOS, to incorporate influential rare variants in significant gene burden masks into the common variant-based polygenic risk score. We assessed the predictive performance of ggSOS in the White British test data set, as well as in populations of non-White British European (n = 18,885), African (n = 7165), East Asian (n = 2236), South Asian (n = 9829), and other admixed (n = 1481) ancestries. Twelve genes in pivotal regulatory pathways of bone homeostasis harbored influential rare variants associated with SOS (p < 5.5 × 10⁻⁷), including AHNAK, BMP5, CYP19A1, FAM20A, FBXW5, KDM5B, KREMEN1, LGR4, LRP5, SMAD6, SOST, and WNT1. Among 4013 (5.4%) individuals in the test data set carrying these variants, a one standard deviation decrease in ggSOS was associated with 1.35-fold (95% confidence interval [CI] 1.16–1.57) increased hazard of major osteoporotic fracture. However, compared with a common variant-based polygenic risk score (C-index = 0.641), ggSOS had only marginally improved prediction accuracy in identifying at-risk individuals (C-index = 0.644), with overlapping confidence intervals. Similarly, ggSOS did not demonstrate substantially improved predictive performance in non-European ancestry populations. In summary, modeling the effects of rare genetic determinants may assist polygenic prediction of fracture risk among carriers of influential rare variants. Nonetheless, improved clinical utility is not guaranteed for population-level risk screening.