Heterogeneity of biological materials, such as bone, tooth, and mollusc shells, plays a key role in determining their mechanical performance (e.g. the strength, damage tolerance, etc.). Here, we quantify heterogeneities in elasticity and inelasticity of bovine cortical bone between 100 nm and a few microns and identify a characteristic length scale (λ_c) of approximately 200 nm. Below λ_c the mechanical heterogeneity of bone is pronounced and exhibits a strong nonlinear size-dependence, while above λ_c the heterogeneity is much less. Such size-dependent heterogeneity benefits the mechanical performance of bone since it not only promotes the energy dissipation at nanoscale, but also suppresses heterogeneity-induced stress concentration and strain localization at larger length scales. This is one of the possible mechanisms functioning at multiple length scales that make bone a well-designed tough natural material. Utilizing experimentally measured data, systematic computational simulations were carried out, showing that the heterogeneity in inelasticity, rather than elasticity, plays a dominant role in promoting energy dissipation during deformation. Possible parameters that determine the inelasticity heterogeneity (e.g. mean value and standard deviation of heterogeneous yield stress) and therefore affect energy dissipation are investigated under typical deformation modes of bone. The analysis presented suggests that there exists an optimum ratio of macroscopic strength to elastic modulus for improving energy dissipation under tension. All these findings are of great value to the design and synthesis of improved bio-inspired composites.