Each year, 1.7% of construction workers sustain an injury serious enough to take time off work. One of the major reasons for this, is the fatigue induced by the highly physically demanding nature of construction tasks. To monitor physical fatigue in real-time, wearable sensors such as heart rate monitors and inertial measurement units (IMUs) have been recognized in professional athletics and in the transportation industry. However, in the construction field, their use generally remains limited to either solely one task or to multitasking activity within laboratory settings. This study aims to quantify the physical fatigue during multitasking activities in a non-controlled environment by combining an IMU and a heart rate monitor. The proposed sensor system as well as the data processing method were tested on four case studies chronologically: 1) on data extracted from a repetitive task performed by fifteen subjects with laboratory settings, 2) on data extracted from two repetitive tasks performed by fifteen subjects with laboratory settings, 3) for a multitasking activity performed by three subjects in a non-controlled environment and 4) on three construction workers during three consecutive half-days of work. In all cases, Borg's Rating of Perceived Exertion was used as a subjective scale to obtain the fatigue level experienced by the participants. Three classifier models, namely the logistic regression, the random forest, and the bagged trees, were tested for physical fatigue quantification. The main results showed that the proposed method allows to quantify the physical fatigue during a half day of multitasking activities in a non-controlled environment with an accuracy of 78.6% using a random forest model using a maximum of 18 features. The results also showed that the method proposed in this study is not, at this stage, directly applicable for the quantification of the physical fatigue of construction workers during their working day. Indeed, the hostile climatic conditions of the construction sites resulted in the heart rate monitor malfunctioning and construction workers systematically rated their fatigue level as low, which created difficulties in labelling the data