Ultrascalable implicit finite element analyses in solid mechanics with over a half a billion degrees of freedom

Adams, Mark F.<sup>1</sup>; Bayraktar, Harun H.<sup>2,3</sup>; Keaveny, Tony M.<sup>2,4,5</sup>; Papadopoulos, Panayiotis<sup>4,6</sup>

Affiliations

¹Sandia National Laboratories, PO Box 969, MS 9217, Livermore, CA 94551

²Orthopaedic Biomechanics Laboratory, University of California, Berkeley, CA

³ABAQUS, Inc., Pawtucket, RI

⁴Department of Mechanical Engineering, University of California, Berkeley, CA

⁵Department of Bioengineering, University of California, Berkeley, CA

⁶Computational Solid Mechanics Laboratory, University of California, Berkeley, CA

Abstract

The solution of elliptic diffusion operators is the computational bottleneck in many simulations in a wide range of engineering and scientific disciplines. We present a truly scalable–ultrascalable–algebraic multigrid (AMG) linear solver for the diffusion operator in unstructured elasticity problems. Scalability is demonstrated with speedup studies of a non-linear micro-finite element analyses of a human vertebral body with over a half of a billion degrees of freedom on up to 4088 processors on the ACSI White machine. This work is significant because in the domain of unstructured implicit finite element analysis in solid mechanics with complex geometry, this is the first demonstration of a highly parallel and efficient application of a mathematically optimal linear solution method on a common large scale computing platform—the IBM SP Power3.

Links

DOI: 10.1109/SC.2004.62

Cited Works (15)

Year	Entry
1999	Niebur GL, Yuen JC, Hsia AC, Keaveny TM. Convergence behavior of high-resolution finite element models of trabecular bone. J Biomech Eng. December 1999;121(6):629-635.
2002	Niebur GL, Feldstein MJ, Keaveny TM. Biaxial failure behavior of bovine tibial trabecular bone. J Biomech Eng. December 2002;124(6):699-705.
2004	Bayraktar HH, Gupta A, Kwon RY, Papadopoulos P, Keaveny TM. The modified super-ellipsoid yield criterion for human trabecular bone. J Biomech Eng. 2004;126(6):677-684.
2004	Morgan E, Bayraktar H, Yeh O, Majumdar S, Burghardt A, Keaveny T. Contribution of inter-site variations in architecture to trabecular bone apparent yield strains. J Biomech. 2004;37(9):1413-1420.
1996	Rüegsegger P, Koller B, Müller R. A microtomographic system for the nondestructive evaluation of bone architecture. Calcif Tiss Int. 1996;58(1):24-29.
2000	Niebur GL, Feldstein MJ, Yuen JC, Chen TJ, Keaveny TM. High-resolution finite element models with tissue strength asymmetry accurately predict failure of trabecular bone. J Biomech. December 2000;33(12):1575-1583.
2003	Van Rietbergen B, Huiskes R, Eckstein F, Rüegsegger P. Trabecular bone tissue strains in the healthy and osteoporotic human femur. J Bone Miner Res. October 2003;18(10):1781-1788.
2003	Stölken JS, Kinney JH. On the importance of geometric nonlinearity in finite-element simulations of trabecular bone failure. Bone. October 2003;33(4):494-504.
2004	Bayraktar HH, Morgan EF, Niebur GL, Morris GE, Wong EK, Keaveny TM. Comparison of the elastic and yield properties of human femoral trabecular and cortical bone tissue. J Biomech. January 2004;37(1):27-35.
2004	Homminga J, Van-Rietbergen B, Lochmüller EM, Weinans H, Eckstein F, Huiskes R. The osteoporotic vertebral structure is well adapted to the loads of daily life, but not to infrequent "error" loads. Bone. March 2004;34(3):510-516.
1995	van Rietbergen B, Weinans H, Huiskes R, Odgaard A. A new method to determine trabecular bone elastic properties and loading using micromechanical finite-element models. J Biomech. January 1995;28(1):69-81.
1998	Ulrich D, van Rietbergen B, Weinans H, Rüegsegger P. Finite element analysis of trabecular bone structure: a comparison of image-based meshing techniques. J Biomech. December 1998;31(12):1187-1192.
2001	Cowin SC, ed. Bone Mechanics Handbook. 2nd ed. Boca Raton, FL: CRC Press; 2001.
1999	Ulrich D, van Rietbergen B, Laib A, Rüegsegger P. The ability of three-dimensional structural indices to reflect mechanical aspects of trabecular bone. Bone. 1999;25(1):55-60.
1998	Guldberg RE, Hollister SJ, Charras GT. The accuracy of digital image-based finite element models. J Biomech Eng. 1998;120(2):289-295.

Cited By (40)

Year	Entry
2007	McDonnell P, McHugh PE, O’Mahoney D. Vertebral osteoporosis and trabecular bone quality. Ann Biomed Eng. February 2007;35(2):170-189.
2006	Bevill G, Eswaran SK, Gupta A, Papadopoulos P, Keaveny TM. Influence of bone volume fraction and architecture on computed large-deformation failure mechanisms in human trabecular bone. Bone. December 2006;39(6):1218-1225.
2007	Eswaran SK, Gupta A, Keaveny TM. Locations of bone tissue at high risk of initial failure during compressive loading of the human vertebral body. Bone. October 2007;41(4):733-739.
2009	Bevill G, Keaveny TM. Trabecular bone strength predictions using finite element analysis of micro-scale images at limited spatial resolution. Bone. April 2009;44(4):579-584.
2015	Wang J, Zhou B, Liu XS, Fields AJ, Sanyal A, Shi X, Adams M, Keaveny TM, Guo XE. Trabecular plates and rods determine elastic modulus and yield strength of human trabecular bone. Bone. March 2015;72:71-80.
2020	Sadoughi S, vom Scheidt A, Nawathe S, Zhu S, Moini A, Keaveny TM. Effect of variations in tissue-level ductility on human vertebral strength. Bone. August 2020;137:115445.
2020	Schenk D, Mathis A, Lippuner K, Zysset P. In vivo repeatability of homogenized finite element analysis based on multiple HR-pQCT sections for assessment of distal radius and tibia strength. Bone. December 2020;141:115575.
2007	Eswaran SK, Bayraktar HH, Adams MF, Gupta A, Hoffmann PF, Lee DC, Papadopoulos P, Keaveny TM. The micro-mechanics of cortical shell removal in the human vertebral body. Comput Meth Appl Mech Eng. June 15, 2007;196(31-32):3025-3032.
2008	Arbenz P, van Lenthe GH, Mennel U, Müller R, Sala M. A scalable multi‐level preconditioner for matrix‐free µ‐finite element analysis of human bone structures. Int J Num Meth Eng. February 12, 2008;73(7):927-947.
2006	Ün K, Bevill G, Keaveny TM. The effects of side-artifacts on the elastic modulus of trabecular bone. J Biomech. 2006;39(11):1955-1963.
2007	Bevill G, Easley SK, Keaveny TM. Side-artifact errors in yield strength and elastic modulus for human trabecular bone and their dependence on bone volume fraction and anatomic site. J Biomech. 2007;40(15):3381-3388.
2007	Eswaran SK, Allen MR, Burr DB, Keaveny TM. A computational assessment of the independent contribution of changes in canine trabecular bone volume fraction and microarchitecture to increased bone strength with suppression of bone turnover. J Biomech. 2007;40(15):3424-3431.
2009	Eswaran SK, Bevill G, Nagarathnam P, Allen MR, Burr DB, Keaveny TM. Effects of suppression of bone turnover on cortical and trabecular load sharing in the canine vertebral body. J Biomech. March 11, 2009;42(4):517-523.
2010	Fields AJ, Lee GL, Keaveny TM. Mechanisms of initial endplate failure in the human vertebral body. J Biomech. 2010;43(16):3126-3131.
2012	Sanyal A, Gupta A, Bayraktar HH, Kwon RY, Keaveny TM. Shear strength behavior of human trabecular bone. J Biomech. October 11, 2012;45(15):2513-2519.
2013	Nawathe S, Juillard F, Keaveny TM. Theoretical bounds for the influence of tissue-level ductility on the apparent-level strength of human trabecular bone. J Biomech. April 26, 2013;46(7):1293-1299.
2015	Nawathe S, Nguyen BP, Barzanian N, Akhlaghpour H, Bouxsein ML, Keaveny TM. Cortical and trabecular load sharing in the human femoral neck. J Biomech. March 18, 2015;48(5):816-822.
2015	van Rietbergen B, Ito K. A survey of micro-finite element analysis for clinical assessment of bone strength: the first decade. J Biomech. March 18, 2015;48(5):832-841.
2015	Nawathe S, Yang H, Fields AJ, Bouxsein ML, Keaveny TM. Theoretical effects of fully ductile versus fully brittle behaviors of bone tissue on the strength of the human proximal femur and vertebral body. J Biomech. May 1, 2015;48(7):1264-1269.
2015	Kaynia N, Soohoo E, Keaveny TM, Kazakia GJ. Effect of intraspecimen spatial variation in tissue mineral density on the apparent stiffness of trabecular bone. J Biomech Eng. January 2015;137(1):011010.
2006	Eswaran SK, Gupta A, Adams MF, Keaveny TM. Cortical and trabecular load sharing in the human vertebral body. J Bone Miner Res. February 2006;21(2):307-314.
2009	Fields AJ, Eswaran SK, Jekir MG, Keaveny TM. Role of trabecular microarchitecture in whole‐vertebral body biomechanical behavior. J Bone Miner Res. September 2009;24(9):1523-1530.
2010	Liu XS, Zhang XH, Sekhon KK, Adams MF, McMahon DJ, Bilezikian JP, Shane E, Guo XE. High-resolution peripheral quantitative computed tomography can assess microstructural and mechanical properties of human distal tibial bone. J Bone Miner Res. April 2010;25(4):746-756.
2014	Nawathe S, Akhlaghpour H, Bouxsein ML, Keaveny TM. Microstructural failure mechanisms in the human proximal femur for sideways fall loading. J Bone Miner Res. February 2014;29(2):507-515.
2023	Emerzian SR, Wu T, Vaidya R, Tang SY, Abergel RJ, Keaveny TM. Relative effects of radiation-induced changes in bone mass, structure, and tissue material on vertebral strength in a rat model. J Bone Miner Res. July 2023;38(7):1032-1042.
2010	Zhang XH. High Resolution Imaging Based Patient Specific Biomechanical Assessment of Bone Quality [PhD thesis]. New York, NY: Columbia University; 2010.
2016	Wang J. Plate-Rod Microstructural Modeling for Accurate and Fast Assessment of Bone Strength [PhD thesis]. Columbia University; 2016.
2015	Goff M. The Role of Micro and Ultra-Structure in Microdamage Accumulation in Cancellous Bone [PhD thesis]. Ithaca, NY: Cornell University; August 2015.
2017	Matheny JB. Interactions Between Bone Remodeling and Microdamage in Cancellous Bone [PhD thesis]. Ithaca, NY: Cornell University; August 2017.
2018	Torres AM. Fatigue Behavior of Cancellous Bone, Microdamage Accumulation, and Biologically Inspired Cellular Solids [PhD thesis]. Ithaca, NY: Cornell University; August 2018.
2012	Christen DB. Nonlinear Failure Prediction in Human Bone: A Clinical Approach Based on High Resolution Imaging [PhD thesis]. Swiss Federal Institute of Technology Zürich; 2012.
2014	Gross T. Development and Application of 3d CT Image-Based Micro and Macro Finite Element Models for Human Bones and Orthopedic Implant Systems [PhD thesis]. Vienna University of Technology; 2014.
2020	Stipsitz M. Development of a Nonlinear Micro Finite Element Framework for Image-Based Simulations in Bone Biomechanics [PhD thesis]. Vienna University of Technology; 2020.
2008	Bevill GR. Micromechanical Modeling of Failure in Trabecular Bone [PhD thesis]. Berkeley, CA: Berkeley, University of California; 2008.
2010	Fields AJ. Trabecular Microarchitecture, Endplate Failure, and the Biomechanics of Human Vertebral Fractures [PhD thesis]. Berkeley, CA: Berkeley, University of California; 2010.
2013	Sanyal A. Bone Strength Multi-Axial Behavior: Volume Fraction, Anisotropy and Microarchitecture [PhD thesis]. Berkeley, CA: Berkeley, University of California; 2013.
2014	Nawathe S. Micromechanics of the Human Proximal Femur: Role of Microstructure and Tissue-Level Ductility on Femoral Strength [PhD thesis]. Berkeley, CA: Berkeley, University of California; 2014.
2018	Pendleton MM. Effects of Spaceflight- and Clinically-Relevant Ionizing Radiation Exposure on Bone Biomechanics [PhD thesis]. Berkeley, CA: Berkeley, University of California; 2018.
2019	Sadoughi S. Micromechanics of Human Bone: Role of Architecture and Tissue Material Properties [PhD thesis]. Berkeley, CA: Berkeley, University of California; 2019.
2020	Bonnheim N. Fundamental Mechanisms of Load Transfer in the Human Vertebral Body Following Lumbar Total Disc Arthroplasty [PhD thesis]. Berkeley, CA: Berkeley, University of California; 2020.