Nanomechanical heterogeneity is expected to influence elasticity, damage, fracture and remodelling of bone. Here, the spatial distribution of nanomechanical properties of bone is quantified at the length scale of individual collagen fibrils. Our results show elaborate patterns of stiffness ranging from ∼2 to 30 GPa, which do not correlate directly with topographical features and hence are attributed to underlying local structural and compositional variations. We propose a new energy-dissipation mechanism arising from nanomechanical heterogeneity, which offers a means for ductility enhancement, damage evolution and toughening. This hypothesis is supported by computational simulations that incorporate the nanoscale experimental results. These simulations predict that non-uniform inelastic deformation over larger areas and increased energy dissipation arising from nanoscale heterogeneity lead to markedly different biomechanical properties compared with a uniform material. The fundamental concepts discovered here are applicable to a broad class of biological materials and may serve as a design consideration for biologically inspired materials technologies.