Androgen deprivation therapy (ADT) given to men with prostate cancer causes rapid and severe sex steroid deficiency, leading to increased bone remodeling and accelerated bone loss. To examine the effects of a single dose of zoledronic acid on bone microarchitecture, we conducted a 2‐year randomized placebo controlled trial in 76 men, mean age (interquartile range [IQR]) 67.8 years (63.8 to 73.9) with non‐metastatic prostate cancer commencing adjuvant ADT;​ 39 were randomized to zoledronic acid and 37 to matching placebo. Bone microarchitecture was measured using high‐resolution peripheral quantitative computed tomography (HR‐pQCT). Using a mixed model, mean adjusted differences (MAD;​ 95% confidence interval [95% CI]) between the groups are reported as the treatment effect at several time points. Over 24 months, zoledronic acid showed no appreciable treatment effect on the primary outcomes for total volumetric bone mineral density (vBMD);​ radius (6.7 mg HA/cm³ [−2.0 to 15.4], p = 0.21) and tibia (1.9 mg HA/cm³ [−3.3 to 7.0], p = 0.87). Similarly, there were no between‐group differences in other measures of microarchitecture, with the exception of a modest effect of zoledronic acid over placebo in total cortical vBMD at the radius over 12 months (17.3 mgHA/cm³ [5.1 to 29.5]). In contrast, zoledronic acid showed a treatment effect over 24 months on areal bone mineral density (aBMD) by dual‐energy X‐ray absorptiometry (DXA) at all sites, including lumbar spine (0.10 g/cm2 [0.07 to 0.13]), p < 0.001), and total hip (0.04 g/cm2 [0.03 to 0.05], p < 0.001). Bone remodeling markers were initially suppressed in the treatment group then increased but remained lower relative to placebo (MADs at 24 months CTX −176 ng/L [−275 to –76], p < 0.001;​ P1NP –18 mg/L [−32 to –5], p < 0.001). These findings suggest that a single dose of zoledronic acid over 2 years is ineffective in preventing the unbalanced bone remodeling and severe microstructural deterioration associated with ADT therapy.