Poor vitamin D status and high parathyroid hormone (PTH) level are associated with impaired bone microarchitecture, but these data are mainly cross-sectional. We studied the association of the baseline PTH and 25-hydroxycholecalciferol (25OHD) levels with the prospectively assessed deterioration of bone microarchitecture and in estimated bone strength in older men. Distal radius and tibia bone microarchitecture was assessed by high-resolution peripheral quantitative computed tomography (HR-pQCT) at baseline, then after 4 and 8 years in 826 men aged 60–87 years. At distal radius, total bone mineral density (Tt.BMD), cortical thickness (Ct.Th^d), cortical area (Ct.Ar), cortical BMD (Ct.BMD), and trabecular BMD (Tb.BMD) decreased, whereas trabecular area (Tb.Ar) increased more rapidly in men with 25OHD ≤20 ng/mL versus the reference group (>30 ng/mL). Men with 25OHD ≤10 ng/mL had faster decrease in reaction force and failure load than men with 25OHD >30 ng/mL. At the distal tibia, Tt.BMD, Ct.Th^d, Ct.Ar, Ct.BMD, failure load, and reaction force decreased, whereas Tb.Ar increased more rapidly in men with 25OHD between 10 and 20 ng/mL versus the reference group. The results were similar when 12 ng/mL was used as a threshold of severe vitamin D deficiency. At distal radius, men with PTH levels above the median (>44 pg/mL) had more rapid decrease in Tt.BMD, Ct.Ar, Ct.BMD, Ct.Th^d, reaction force, and failure load, and more rapid increase in Tb.Ar versus the lowest quartile (≤34 pg/mL). At the distal tibia, men in the highest PTH quartile had faster decrease in Tt.BMD, Ct.Th^d, Ct.Ar, Ct.BMD, reaction force, and failure load and faster increase in Tb.Ar versus the lowest quartile. The results were similar in men with glomerular filtration rate >60 mL/min. The results were similar in men who took no vitamin D or calcium supplements for 8 years. In summary, vitamin D deficiency and secondary hyperparathyroidism are associated with more rapid prospectively assessed cortical and trabecular bone decline in older men.