OSTBC Transmission in MIMO AF Relay Systems with Keyhole and Spatial Correlation Effects

In this paper, we investigate the degenerative effects of antenna correlation and keyhole on the performance of multiple-input multiple-output (MIMO) amplify-and-forward (AF) relay networks. In particular, we considered a downlink MIMO AF relay system consisting of an $\nSx$-antenna base station $\mathsf{S}$, an $\nRx$-antenna relay station $\mathsf{R}$, and an $\nDx$-antenna mobile station $\mathsf{D}$, in which the signal propagation originated from the mobile station suffers from keyhole and spatial correlation effects. We have derived a closed-form expression for the exact moment generation function of the instantaneous signal-to-noise ratio which enables us to analyze the symbol error probability and outage probability of the considered system. We have shown that although the mobile station is in a poor scattering environment, i.e., keyhole, the relay channel ($\mathsf{S}\rightarrow\mathsf{R}\rightarrow\mathsf{D}$ link) still achieves a cooperative diversity order of $\min(\nRx,\nDx)$ provided that the channel from the source ($\mathsf{S}\rightarrow\mathsf{R}$ link) is keyhole-free. This result is important for the radio system designers since under such a severe scenario it is unnecessary to deploy a large number of antennas on the relay station $(\nRx)$ but only requires $\nRx=\nDx$ to obtain the maximum achievable diversity gain.