Ductile heterogeneous materials like cast aluminum alloys, undergo catastrophic ductile failure that initiates with particle fragmentation with evolves with void growth and coalescenece in localized bands of intense plastic deformation and strain softening. Conventional Voronoi Cell finite element model (VCFEM), based on the assumed stress hybrid formulation, is for small deofmration and is unable to account for plastic strain induced softening. To overcome this shortcoming of material softening due to plastic strain localization, this work introduces a locally enhanced Voronoi Cell finite element model (LE-VCFEM) for modeling the very complex phenomenon of ductile failure in heterogeneous metals and alloys. In LE-VCFEM, finite deformation displacement elements are adaptively added to regions of localization in the otherwise assumed stress based hybrid Voronoi celle finite element to locally enhance modeling capabilities for ductile fracture. Adaptive h-refinement is used for the displacement elements to improve accuracy. Damage initiation by particle cracking is trigerred by a Weibull model. The nonlocal Gurson-Tvergaard-Needleman model of porous plasticity is implemented in LE-VCFEM to model matrix cracking. An iterative strain update algorithm is used for the displacement elements. The LE-VCFEM code is validated by comparing with results in the literature and with conventional FE codes. Furthermore, LE-VCFEM simulations of real micrstructures are satisfactorily