Bone fragility is a major health concern, as the increased risk of bone fractures has devastating outcomes in terms of mortality, decreased autonomy, and healthcare costs. Efforts made to address this problem have considerably increased our knowledge about the mechanisms that regulate bone formation and resorption. In particular, we now have a much better understanding of the cellular events that are triggered when bones are mechanically stimulated and how these events can lead to improvements in bone mass. Despite these findings at the molecular level, most exercise intervention studies reveal either no effects or only minor benefits of exercise programs in improving bone mineral density (BMD) in osteoporotic patients. Nevertheless, and despite that BMD is the gold standard for diagnosing osteoporosis, this measure is only able to provide insights regarding the quantity of bone tissue. In this article, we review the complex structure of bone tissue and highlight the concept that its mechanical strength stems from the interaction of several different features. We revisited the available data showing that bone mineralization degree, hydroxyapatite crystal size and heterogeneity, collagen properties, osteocyte density, trabecular and cortical microarchitecture, as well as whole bone geometry, are determinants of bone strength and that each one of these properties may independently contribute to the increased or decreased risk of fracture, even without meaningful changes in aBMD. Based on these findings, we emphasize that while osteoporosis (almost) always causes bone fragility, bone fragility is not always caused just by osteoporosis, as other important variables also play a major role in this etiology. Furthermore, the results of several studies showing compelling data that physical exercise has the potential to improve bone quality and to decrease fracture risk by influencing each one of these determinants are also reviewed. These findings have meaningful clinical repercussions as they emphasize the fact that, even without leading to improvements in BMD, exercise interventions in patients with osteoporosis may be beneficial by improving other determinants of bone strength.