Vibration analysis has been widely used to detect gear tooth fault inside a planetary gearbox. However, the vibration characteristics of a planetary gearbox are very complicated. Inside a planetary gearbox, there are multiple vibration sources as several sun-planet gear pairs and several ring-planet gear pairs are meshing simultaneously. In addition, due to the rotation of the carrier, distance varies between vibration sources and a transducer installed on gearbox housing. This thesis aims to simulate and understand the vibration signals of a planetary gear set, and then propose a signal processing method to detect gear tooth fault more effectively. First, an analytical method derives the equations of a healthy planetary gear set’s time-varying gear mesh stiffness. Then, a gear tooth crack growth model is proposed and equations are derived to quantify the effect of gear tooth crack on the time-varying mesh stiffness. After that, a two-dimensional lumpedmass model is developed to simulate the vibration source signals of a planetary gear set;​ an analytical model is proposed to represent the effect of transmission path;​ and the resultant vibration signals of a planetary gear set at a sensor location are generated by considering multiple vibration sources and the effect of transmission path. Finally, a signal decomposition method is proposed to detect a single tooth crack in a single planet gear and experimental validation is performed. The methods proposed in this thesis help us understand the vibration properties of planetary gearboxes and give insights into developing new signal processing methods for gear tooth fault detection.