Trellis coded modulation (TCM) schemes, due to their bandwidth efficiency and coding gain, have been proposed for multipath fading (MF) channels. The object of this research is to analyze the performance of TCM schemes in MF channels. While many excellent studies have already been reported, they share some of the following restrictions:​ (1) an assumption of ideal channel measurements and ideal interleaving, (2) use of Chernoff bounds, which are loose in this case, (3) analysis of only the Rayleigh channel, ignoring the Rician channel, and (4) reliance on computer simulation to get the actual performance. Extending analytical results without these restrictions is addressed in this work.

This thesis derives a saddle point approximation (SAP) method to compute the pairwise error probability (PEP) of TCM schemes transmitted over Rician fading channels. It can be applied under several conditions, including finite or ideal interleaving, and is derived for a pilot-tone model, encompassing ideal coherent detection, pilot-tone aided detection, pilot-symbol aided detection, and differential detection. Its accuracy is demonstrated by comparison to the results of numerical integration. Under ideal interleaving, the approximation can be further simplified to an expression, which is in a product form and is much tighter than the ordinary Chernoff bound on the PEP. Also, based on the SAP, the effect of finite interleaving depth on the error performance is studied.

The Canadian mobile satellite (MSAT) channel has been modelled as the sum of lognormal and Rayleigh components. Previously, the performance of TCM schemes in this channel has been obtained via computer simulation. In this thesis, new analytical expressions are derived for the PEP of TCM schemes transmitted over this channel employing ideal interleaving, and the results are substantiated by means of computer simulation. In addition, first order statistics of absolute and differential phases of a shadowed Rician process are derived.

This thesis presents new performance bounds of TCM schemes over nonindependent (i.e., finite interleaving) Rician fading channels. In addition, for Rayleigh fading channels with an exponential auto-covariance function, bounds resembling those for memoryless channels are derived. The bounds, being more accurate than Chernoff bounds, permit accurate estimation of system performance.

The performance of concatenated coding systems and automatic-repeal-request (ARQ) systems operating on fading channels is addressed. New error expressions,which sow asymptotic error behaviour, are derived for systems which use a modified Viterbi decoding algorithm. They allow useful evaluation of the coding gain and throughput.

Finally, the performance of convolutional codes in fading channels is analyzed. An upper bound on the bit error probability, the optimum power split ratio between the data and pilot signals, and the channel cut-off rate are derived.