This study investigates proteomic differences between knee osteoarthritis (OA) and rheumatoid arthritis (RA) to identify protein signatures with potential diagnostic and therapeutic relevance. Using SWATH mass spectrometry, synovial fluid proteome from OA patients across Kellgren–Lawrence OA grades 2, 3, and 4 was analyzed and compared with RA patients. Out of 333 quantified proteins, 45 were differentially expressed, including aggrecan, versican, and inflammation-related proteins (e.g., CRP, APCS, S100A8, and SAA2). ELISA validation confirmed aggrecan, versican, and cartilage oligomeric matrix protein (COMP) as significantly altered proteins in OA compared to RA, along with distinct trends with OA progression and mirrored patterns in paired serum samples. ROC curve analysis highlighted COMP's strong diagnostic potential, with an AUC of 96%, 87.2%, and 85.2% for OA grades 2, 3, and 4 versus RA, respectively. COMP differentiated OA from RA at a synovial fluid concentration of < 3136 ng/mL, AUC of 92.1%, 89% sensitivity, and 82% specificity. Versican also demonstrated diagnostic utility, particularly in later OA stages. Gender-specific analysis revealed no differences for aggrecan and versican, while COMP levels were significantly higher in males. Simultaneously, a lower aggrecan, versican, and COMP levels were observed in OA (females) as compared to RA, potentially linked to estrogen decline with age and cartilage degradation. However, gender variability underscores the need for a larger, sex-balanced cohort study. Future studies could aim to account for validating COMP's diagnostic potential with healthy controls, demonstrating its reliability to characterize different OA grades.