Previous experimental and computational studies have indicated that removing bound water in bone matrix makes bone stiffer, stronger, but more brittle at different length scales. However, a clear mechanistic explanation of the underlying mechanisms is lacking. Assuming that bound water mainly alters the mechanical behavior of collagen phase and the interfaces among bone constituents, this study investigated the effects of bound water on the mechanical properties of bone using a 2D cohesive finite element (FE) model representing the sub-lamellar hierarchy of the tissue. The model contained sufficient ultrastructural details of mineralized collagen fibrils (MCF), extrafibrillar matrix (EFM), and the interfaces among bone constituents. The mechanical behavior of the interfaces, and mineral/collagen phases, in the hydrated and dehydrated conditions was carefully selected based on the information available in the literature. The FE simulations indicated that hydration status induced changes at the interfaces played a key role in determining the mechanical behavior of bone. In tension, hydrated interfaces (weak but tough) in bone appeared to encourage multiple nanocrack formation, debonding between the MCF and EFM subunits, and crack bridging by MCFs. On the other hand, dehydrated (strong but brittle) interfaces made the tissue stiffer and stronger, but compromised the above energy dissipation mechanisms, thus leading to a brittle failure. In compression, hydrated interfaces resulted in sliding between the mineral crystals in EFM, debonding between EFM and MCF, and buckling of MCF, whereas dehydrated interfaces appeared to make the tissue stiffer and stronger and the energy dissipation mechanisms diminished. The outcome of this study provides new insights into the mechanisms underlying the effect of bound water on bone fragility at ultrastructural levels.