With the rapid development of multimedia services, wireless communication engineers may face a major challenge to meet the demand of higher data-rate communication over error-prone mobile radio channels. As a promising solution, the concept of cooperative communication, where a so-called relay node is formed to assist the direct link, has recently been applied to alleviate the severe pathloss and shadowing effects in wireless systems. In addition, without spending extra spectrum and power resources, multiple-input multiple-output (MIMO) antenna systems have been shown to provide an immense improvement in system performance compared to its single-antenna counterpart. As such, cooperative MIMO communication is essential for wireless and mobile networks because of its remarkable increase in spectral efficiency and reliability. Although the utilization of cooperative communication in MIMO systems has gained great attention in the literature, most of the research works have assumed perfect conditions. Inspired by the aforementioned discussion, this thesis takes a step further to investigate the performance of cooperative communications with practical constraints. The thesis provides a general framework for performance analysis of cooperative communications subject to several practical constraints such as antenna correlation, rank-deficiency of the channel matrix, co-channel interference, and interference-limited constraint of cognitive radio networks based on an underlay spectrum-sharing approach.

The thesis is divided into six parts. The first part investigates the performance of orthogonal space-time block codes (OSTBCs) over MIMO relay networks in Nakagami-m fading channels under the antenna correlation effect. The second part extends the full-rank MIMO channel to the case of the MIMO channel matrix being of rank-deficiency. Several important findings on the impact of the single-keyhole effect (SKE) and double-keyhole effect (DKE) are observed for two types of amplifying mechanism at the relay, namely, linear and squaring approaches. An important observation corroborated by our studies is that for offering a tradeoff between performance and complexity, we should use the linear approach for SKE and the squaring approach for DKE. The third part generalizes the keyhole effect to multi-keyhole channels. The exact and asymptotic expressions for symbol error probability (SEP) are derived for some specific cases such as multi-keyhole MIMO/multiple-input single-output (MISO) channel. The fourth part proposes a distributed Alamouti space-time code for two-way fixed gain amplify-and-forward (AF) relaying. In particular, closed-form expressions for approximated ergodic sum-rate and exact pairwise error probability (PWEP) are derived for Nakagami-m fading channels. To reveal further insights into array and diversity gains, an asymptotic PWEP is also obtained. The fifth part analyzes the outage performance of a two-way fixed gain AF relay system with beamforming, arbitrary antenna correlation, and co-channel interference (CCI). Finally, the sixth part investigates the impact of interference power constraint on the performance of cognitive relay networks based on the spectrum-sharing approach.