This work consists of the validation of a novel approach to estimate the local anisotropic elastic constants of the bone extracellular matrix using nanoindentation. For this purpose, nanoindentation on two planes of material symmetry were analyzed and the resulting longitudinal elastic moduli compared to the moduli measured with a macroscopic tensile test. A combined lathe and tensile system was designed that allows machining and testing of cylindrical microspecimens of approximately 4 mm in length and 300 μm in diameter. Three bovine specimens were tested in tension and their outer geometry and porosity assessed by synchrotron radiation microtomography. Based on the results of the traction test and the precise outer geometry, an apparent longitudinal Young's modulus was calculated. Results between 20.3 and 27.6 GPa were found that match with previously reported values for bovine compact bone. The same specimens were then characterized by nanoindentation on a transverse and longitudinal plane. A longitudinal Young's modulus for the bone matrix was then derived using the numerical scheme proposed by Swadener and Pharr and the fabric–elasticity relationship by Zysset and Curnier. Based on the matrix modulus and a power law effective volume fraction, an apparent longitudinal Young's modulus was predicted for each microspecimen. This alternative approach provided values between 19.9 and 30.0 GPa, demonstrating differences between 2% and 13% to the values provided by the initial tensile test. This study therefore raises confidence in our nanoindentation protocol of the bone extracellular matrix and supports the underlying hypotheses used to extract the anisotropic elastic constants.