We model the elastic properties of bone at the level of mineralized collagen fibrils via step-by-step homogenization from the staggered arrangement of collagen molecules up to an array of parallel mineralized fibrils. A new model for extrafibrillar mineralization is proposed, assuming that the extrafibrillar minerals are mechanically equivalent to reinforcing rings coating each individual fibril. Our modeling suggests that no more than 30% of the total mineral content is extrafibrillar and the fraction of extrafibrillar minerals grows linearly with the overall degree of mineralization. It is shown that the extrafibrillar mineralization considerably reinforces the fibrils’ mechanical properties in the transverse directions and the fibrils’ shear moduli. The model predictions for the elastic moduli and constants are found to be in a good agreement with the experimental data reported in the literature.