Bone strength is generally thought to decline with aging and prior work has compared traits between younger and older cohorts to identify the structural and compositional changes that contribute to fracture risk with age. However, for men, the majority of individuals do not fracture a bone in their lifetime. While fracture occurrence is multifactorial, the absence of fracture in the majority of males suggests that some individuals maintain bone strength or do not lose enough strength to fracture, whereas others do lose strength with aging. Consequently, not all structural and material changes observed with age may lead to strength-decline. We propose that consideration of different subgroups of older individuals will provide a more precise understanding of which structural and material changes directly contribute to strength-decline. We identified subgroups using latent profile analysis (LPA), which is a clustering-based algorithm that takes multiple continuous variables into account. Human cadaveric male femoral diaphyses (n = 33, 26–89 years) were subjected to whole bone and tissue-level mechanical tests. Morphological traits, porosity, and cortical tissue mineral density (Ct.TMD) were obtained, as were measures of enzymatic cross-links and the advanced glycation end product, pentosidine (PEN). A univariate analysis first identified a younger cohort (YNG, n = 11) and older cohort (n = 22). LPA was then conducted using three mechanical traits (whole bone strength, tissue-level strength, and tissue-level post-yield strain), resulting in a further stratification of the older group into two similarly aged groups (p = 0.558), but one with higher (OHM, n = 16) and another with lower mechanical properties (OLM, n = 6). The OLM group exhibited lower whole bone strength (p = 0.016), tissue-level strength (p < 0.001), and tissue-level post-yield strain (p < 0.001) compared to the YNG group. Meanwhile, the OHM only exhibited significantly lower tissue-level post-yield strain (p < 0.001), compared to the YNG group. Between the two older groups, the OHM group exhibited higher whole bone strength (p = 0.037), tissue-level strength (p = 0.006), and tissue-level post-yield strain (p = 0.012) than the OLM group. Probing the morphological and compositional relationships among the three groups, both OHM and OLM exhibited increased PEN content (p < 0.001, p = 0.008 respectively) and increased Log(cortical pore score) relative to YNG (p = 0.003, p < 0.001 respectively). Compared to the OHM group, the OLM also exhibited increased marrow area (p = 0.049), water content (p = 0.048), and decreased Ct.TMD (p = 0.005). The key traits that were unique to the OLM group compared to YNG were decreased Ct.TMD (p < 0.001) and increased Log(porosity) (p = 0.002). There were many properties that differed between the younger and older groups, but not all were associated with reduced mechanical properties, highlighting the relative importance of certain age-related traits such as porosity, Ct.TMD, water content, and marrow area that were unique to the OLM group. Overall, this work supports the hypothesis that there are subgroups of men showing different strength-decline trajectories with aging and establishes a basis for refining our understanding of which age-related changes are directly contributing to decreased strength.