We aimed to investigate cross-sectional associations between skeletal muscle density, a proxy measure for fatty infiltration into muscle, and cognition. Contributions from body fat mass, systemic inflammation and lifestyle were explored, as these factors have been identified in both muscle and cognitive deterioration. For 281 men (60–95 year) from the Geelong Osteoporosis Study, radial and tibial muscle density were measured using peripheral quantitative computed tomography. Body fat and appendicular lean mass were measured using dual-energy X-ray absorptiometry. Cognitive function was assessed for psychomotor function (DET), visual identification/attention (IDN), visual learning (OCL) and working memory (OBK) (CogState Brief Battery). Composite scores signified overall cognitive function (OCF). Higher scores represent poorer performance except for OCL and OCF. Regression analyses examined associations between muscle density and cognition;​ potential confounders included age, muscle cross-sectional area (CSA), body composition, lifestyle and serum markers of inflammation. Negative associations with age were evident for muscle density, all cognitive domains and OCF. Muscle density at both sites was positively associated with DET, OCL and OCF. After adjustment for age, the association persisted for DET (radius:​ B = − 0.006, p = 0.02;​ tibia:​ B = − 0.003, p = 0.04) and OCL (radius B = + 0.004, p = 0.02;​ tibia:​ B = + 0.005, p < 0.001). At the radius, further adjustment for serum TNF-α explained the association between muscle density (B = − 0.002, p = 0.66) and DET. Education and physical activity contributed to the model for radial muscle density and DET. There were no contributions from muscle CSA, appendicular lean mass, body fat mass, other markers of inflammation or other potential confounders. At the tibia, the association between muscle density and DET (B = − 0.003, p = 0.04) was independent of TNF-α. There was an age-adjusted association between muscle density and OCL at both sites (radius:​ B = + 0.004, p = 0.02;​ tibia:​ B = + 0.005, p < 0.001). None of the potential confounders contributed to the models. Muscle density was associated with cognitive function in the DET and OCL domains. However, there was little evidence that this was explained by inflammation or body fat mass. No associations were identified between muscle density and IDN or OBK.