The bone remodeling process takes place at the surface of trabeculae and results in a non-uniform mineral distribution. This will affect the mechanical properties of cancellous bone, because the properties of bone tissue depend on its mineral content. We investigated how large this effect is by simulating several non-uniform mineral distributions in 3D finite element models of human trabecular bone and calculating the apparent stiffness of these models. In the "linear model" we assumed a linear relation between mineral content and Young's modulus of the tissue. In the "exponential model" we included an empirical exponential relation in the model. When the linear model was used the mineral distribution slightly changed the apparent stiffness, the difference varied between an 8% decrease and a 4% increase compared to the uniform model with the same BMD. The exponential model resulted in up to 20% increased apparent stiffness in the main load-bearing direction. A thin less mineralized surface layer (28 μm) and highly mineralized interstitial bone (mimicking mineralization resulting from anti-resorptive treatment) resulted in the highest stiffness. This could explain large reductions in fracture risk resulting from small increases in BMD. The non-uniform mineral distribution could also explain why bone tissue stiffness determined using nano-indentation is usually higher than finite element (FE)-determined stiffness. We conclude that the non-uniform mineral distribution in trabeculae does affect the mechanical properties of cancellous bone and that the tissue stiffness determined using FE-modeling could be improved by including detailed information about mineral distribution in trabeculae in the models.