People with type 2 diabetes mellitus (T2DM) have normal‐to‐high BMDs, but, counterintuitively, have greater fracture risks than people without T2DM, even after accounting for potential confounders like BMI and falls. Therefore, T2DM may alter aspects of bone quality, including material properties or microarchitecture, that increase fragility independently of bone mass. Our objective was to elucidate the factors that influence fragility in T2DM by comparing the material properties, microarchitecture, and mechanical performance of cancellous bone in a clinical population of men with and without T2DM. Cancellous specimens from the femoral neck were collected during total hip arthroplasty (T2DM:​ n = 31, age = 65 ± 8 years, HbA1c = 7.1 ± 0.9%;​ non‐DM:​ n = 34, age = 62 ± 9 years, HbA1c = 5.5 ± 0.4%). The T2DM specimens had greater concentrations of the advanced glycation endproduct pentosidine (+ 36%, P < 0.05) and sugars bound to the collagen matrix (+ 42%, P < 0.05) than the non‐DM specimens. The T2DM specimens trended toward a greater bone volume fraction (BV/TV) (+ 24%, NS, P = 0.13) and had greater mineral content (+ 7%, P < 0.05) than the non‐DM specimens. Regression modeling of the mechanical outcomes revealed competing effects of T2DM on bone mechanical behavior. The trend of higher BV/TV values and the greater mineral content observed in the T2DM specimens increased strength, whereas the greater values of pentosidine in the T2DM group decreased postyield strain and toughness. The long‐term medical management and presence of osteoarthritis in these patients may influence these outcomes. Nevertheless, our data indicate a beneficial effect of T2DM on cancellous microarchitecture, but a deleterious effect of T2DM on the collagen matrix. These data suggest that high concentrations of advanced glycation endproducts can increase fragility by reducing the ability of bone to absorb energy before failure, especially for the subset of T2DM patients with low BV/TV.