Spatial modulation (SM) and quadrature SM (QSM) are promising versions of the single-user multiple-input multiple-output (MIMO) system that overcome the problem of inter-channel-interference which occurs in conventional MIMO systems. Both SM and QSM exploit the index of the activated antenna(s) to carry additional information to enhance the total spectral efficiency of the system transmission. In the detection, the SM and QSM systems jointly detect the index of the activated antenna(s) as well as the transmitted modulation symbol, which is an exhaustive process especially for higher order modulations and large system dimensions. This exhaustive process contradicts the demands of future wireless networks that require low-power consumption and low communication latency.

To fulfill the demand of low-complex decoders at the receiver side for future wireless networks, I propose three different low-complexity decoders for single-user SM and QSM MIMO systems. These algorithms are based on the concept of sphere decoding for the tree-search structure. The first proposed algorithm provides a significant reduction in the decoding complexity with optimal bit error rate (BER) performance. The second proposed algorithm provides an extra reduction in the decoding complexity without sacrificing the optimality of the BER performance. Finally, the third algorithm provides a flexible trade-off between complexity and BER performance to be suitable for most hardware implementations. The proposed algorithms are studied in terms of BER performance and expected decoding complexity for the single-user SM and QSM MIMO systems.

For multi-user SM-MIMO, a low-cost system is proposed using the sparse code multiple access (SCMA) technique. The proposed low-cost SM-SCMA system significantly reduces the required number of transmit antennas with almost no loss in terms of the BER performance and decoding complexity, compared with the conventional SM-SCMA. At the receiver, the message passing algorithm (MPA) is employed to detect the transmitted signals, which suffers from high decoding complexity in practical implementations. To address this issue, three low-complexity decoding algorithms are proposed for the SM-SCMA system. The first algorithm provides the benchmark for the decoding complexity at the expense of the BER performance. The second algorithm slightly increases the decoding complexity with a significant improvement in the BER performance. Finally, the third algorithm provides a near-optimum BER performance with a considerable decoding complexity reduction when compared to the MPA decoder. Moreover, it supports the parallel hardware implementation and strikes a trade-off between decoding complexity and BER performance.

More specifically, the three low-complexity receivers for the single-user SM and QSM MIMO systems are introduced in Chapters 2, 3, 4 and 5. In Chapter 2, the first low-complexity algorithm for single-user QSM-MIMO system is proposed. The second low-complexity algorithm for SM-MIMO system is introduced in Chapter 3, and is analyzed in Chapter 4. The reliable decoder for single-user SM-MIMO system is proposed in Chapter 5. for multi-user SM-SCMA, the low-cost system is proposed in Chapter 6;​ at the receiver side, the three low-complexity decoders for the SM-SCMA system are proposed and analyzed in Chapter 7.