A novel energy-based mechanistic model of nanoscratch test was derived to estimate the in situ toughness of bone using cube corner tip. In addition, several methodologies were developed to estimate the unknowns required by the model (i.e., shear angle, shear plane area, and the friction forces on the tip flank faces). First, the friction co-efficient of bone with the diamond scratch tip was measured using a simple protocol during each individual scratch test. Then, the shear angle was estimated based on the assumption of the minimum cutting force during the scratch process and validated by the experimental results obtained from the scratch profiles. Moreover, the shear plane area was assessed based on the shear angle and the geometry of the scratch tip and the scratch groove. To verify the approach, finite element (FEA) simulation was performed to estimate the shear plane area. The results of FEA simulations were in good agreement with that derived from the model. Finally, to validate the test approach, bone specimens from osteogenesis imperfecta (OI) and wild type (control) mice were tested using the nanoscratch test and the scratch toughness was compared between the bone specimens of three test groups (i.e., wild type, mild and severe OI). Since OI bone is much brittle than that of normal wild type, the efficacy of the test methodology can be verified if the nanoscratch results can distinguish the differences in the toughness of the bone specimens. The experimental results indicate that the present nanoscratch test can be used to determine the in situ toughness of bone tissues at submicron levels.