Bone materials are characterized by an astonishing variability and diversity. Still, because of ‘architectural constraints’ due to once chosen material constituents and their physical interaction, the fundamental hierarchical organization or basic building plans of bone materials remain largely unchanged during biological evolution. Such universal patterns of microstructural organization govern the mechanical interaction of the elementary components of bone (hydroxyapatite, collagen, water;​ with directly measurable tissue-independent elastic properties), which are here quantified through a multiscale homogenization scheme delivering effective elastic properties of bone materials:​ at a scale of 10 nm, long cylindrical collagen molecules, attached to each other at their ends by ~1.5 nm long crosslinks and hosting intermolecular water inbetween, form a contiguous matrix called wet collagen. At a scale of several hundred nanometers, wet collagen and mineral crystal agglomerations interpenetrate each other, forming the mineralized fibril. At a scale of 5-10 μm, the extracellular solid bone matrix is represented as collagen fibril inclusions embedded in a foam of largely disordered (extrafibrillar) mineral crystals. At a scale above the ultrastructure, where lacunae are embedded in extracellular bone matrix, the extravascular bone material is observed. Model estimates predicted from tissue-specific composition data gained from a multitude of chemical and physical tests agree remarkably well with corresponding acoustic stiffness experiments across a variety of cortical and trabecular, extracellular and extravascular materials. Besides from reconciling the well-documented, seemingly opposed concepts of ‘mineral-reinforced collagen matrix’ and ‘collagen-reinforced mineral matrix’ for bone ultrastructure, this approach opens new possibilities in the exploitation of computer tomographic data for nano-to-macro mechanics of bone organs.