Two of the most prominent techniques to meet the next generation wireless communication system's demands are cognitive radio and massive MIMO systems. Cognitive radio systems improve radio spectrum utilization either by spectrum sharing or by opportunistically utilizing the spectrum of the licensed users. Employing multiple antennas at the transmitter and/or the receiver of the radio can further improve the overall performance of the wireless systems. Massive MIMO systems, on the other hand, improve the spectral and energy efficiencies of currently deployed systems by reaping all the benefits of the multi-antenna systems at a very large scale. The price paid for employing a large number of antennas either at the transmitter or receiver is the high hardware cost. Judicious transmit or receive antenna selection can reduce this cost, while retaining most of the benefits offered by multiple antennas.

In my doctoral research, we have presented both upper and lower bounds on the capacity of a general selection diversity system. These novel bounds are simple to compute and can be used in a variety of different fading environments. We have also proposed and analyzed the performance of different antenna selection schemes for both an underlay cognitive radio and a massive MIMO system. Specifically, we have considered both receive and transmit antenna selection in an underlay cognitive radio based on the maximization of secondary link signal-to-interference plus noise ratio. Exact and asymptotic performance analyses of the secondary system with such selections are carried out, and numerical examples are presented to verify the correctness of the analytical results. Several sub-optimal antenna subset selection schemes for both a single-cell and a multi-cell multi-user massive MIMO system are also proposed. Numerical results on the sum rate of the system in different scenarios are presented to verify the superior performance of the proposed schemes over the existing sub-optimal antenna subset selection schemes. Lastly, we have also presented three novel hybrid analog/digital precoding schemes to reduce the hardware and software complexities of a sub-connected massive MIMO system.