High failure rates have been associated with nonlocking cancellous screws with a typical buttress thread in patients with osteoporotic bone. This study aimed to develop a novel thread design and compare its fixation stability with that of a typical buttress thread. Nonlocking cancellous screws with a novel thread design (proximal flank angle of 120 degrees, a flat crest feature, a tip-facing undercut feature) and nonlocking cancellous screws with a typical buttress thread were manufactured using stainless steel. Fixation stabilities were evaluated individually by the axial pullout and lateral migration tests, and they were evaluated in pairs together with a dynamic compression plate in an osteoporotic bone substitute (10 PCF polyurethane foam per ASTM F1839) under cyclic craniocaudal and torsional loadings. Pullout strength and lateral migration resistance for the individual screw test and the force, torque, and number of cycles required to achieve specific displacement and torsion for the multi-screw test were comparatively analyzed between both screw types. A finite element analysis model was constructed to analyze the stress distributions in the bone tissue adjacent to the threads. The biomechanical test revealed the novel undercut thread had superior axial pullout strength, lateral migration resistance, and superior fixation stability when applied to a dynamic compression plate under cyclic craniocaudal loading and torsional loading than those in the typical buttress thread. The finite element analysis simulation revealed that the novel thread can distribute stress more evenly without high-stress concentration at the adjacent bone tissue when compared to that of a typical buttress thread.