At the tissue level, the local material properties of human cancellous bone are heterogeneous due to constant remodelling. Since standard high-resolution computed tomography scanning methods are unable to capture this heterogeneity in detail, local differences in mineralisation are normally not incorporated in computational models. To investigate the effects of heterogeneous mineral distribution on the apparent elastic properties, 40 cancellous bone samples from the human femoral neck were scanned by means of synchrotron radiation microcomputed tomography (SRμCT). SRμCT-based micromechanical finite element models that accounted for mineral heterogeneity were compared with homogeneous models. Evaluation of the apparent stiffness tensor of both model types revealed that homogeneous models led to a minor but significant (p < 0.05) overestimation of the elastic properties of heterogeneous models by 2.18 ± 1.89%. Variation of modelling parameters did not affect the overestimation to a great extent. It was concluded that the heterogeneous mineralisation has only a minor influence on the apparent elastic properties of human cancellous bone.