Maintaining the correct proportions of different cell types in the bone marrow is critical for bone function. Hypertrophic chondrocytes (HCs) and osteoblasts are a lineage continuum with a minor contribution to adipocytes, but the regulatory network is unclear. Mutations in transcription factors, IRX3 and IRX5, result in skeletal patterning defects in humans and mice. We found coexpression of Irx3 and Irx5 in late‐stage HCs and osteoblasts in cortical and trabecular bone. Irx3 and Irx5 null mutants display severe bone deficiency in newborn and adult stages. Quantitative analyses of bone with different combinations of functional alleles of Irx3 and Irx5 suggest these two factors function in a dosage‐dependent manner. In Irx3 and Irx5 nulls, the amount of bone marrow adipocytes was increased. In Irx5 nulls, lineage tracing revealed that removal of Irx3 specifically in HCs exacerbated reduction of HC‐derived osteoblasts and increased the frequency of HC‐derived marrow adipocytes. β‐catenin loss of function and gain of function specifically in HCs affects the expression of Irx3 and Irx5, suggesting IRX3 and IRX5 function downstream of WNT signaling. Our study shows that IRX3 and IRX5 regulate fate decisions in the transition of HCs to osteoblasts and to marrow adipocytes, implicating their potential roles in human skeletal homeostasis and disorders.