The ultrastructural response to applied loads governs the post-yield deformation and failure behavior of bone, and is correlated with bone fragility fractures. Combining a novel progressive loading protocol and synchrotron X-ray scattering techniques, this study investigated the correlation of the local deformation (i.e., internal strains of the mineral and collagen phases) with the bulk mechanical behavior of bone. The results indicated that the internal strains of the longitudinally oriented collagen fibrils and mineral crystals increased almost linearly with respect to the macroscopic strain prior to yielding, but markedly decreased first and then gradually leveled off after yielding. Similar changes were also observed in the applied stress before and after yielding of bone. However, the collagen to mineral strain ratio remained nearly constant throughout the loading process. In addition, the internal strains of longitudinal mineral and collagen phases did not exhibit a linear relationship with either the modulus loss or the plastic deformation of bulk bone tissue. Finally, the time-dependent response of local deformation in the mineral phase was observed after yielding. Based on the results, we speculate that the mineral crystals and collagen fibrils aligned with the loading axis only partially explain the post-yield deformation, suggesting that shear deformation involving obliquely oriented crystals and fibrils (off axis) is dominant mechanism of yielding for human cortical bone in compression.