A Survey on Higher-Order QAM Constellations: Technical Challenges, Recent Advances, and Future Trends

Communication system’s performance is sensitive to bandwidth, power, cost etc. There have been various solutions to improve the performance, out of them, one of the fundamental solutions over the years is design of optimum modulation schemes. As the research on beyond 5G heats up, we survey and explore power and bandwidth efficient modulation schemes for the next generation communication systems in details. In the existing literature, initially square quadrature amplitude modulation (SQAM) was considered. However, only square constellations are not sufficient for varying channel conditions and rate requirements, thus, efficient odd power of 2 constellations were introduced. For odd power of 2 constellations, rectangular QAM (RQAM) is most commonly used. However, RQAM is not a good choice and modified cross QAM (XQAM) constellation is preferred which provides improved power efficiency over RQAM due to its energy efficient two dimensional (2D) structure. The increasing demand for high data-rates has further encouraged research towards more compact 2D constellations which leads to hexagonal lattice structure based hexagonal QAM (HQAM) constellations. In this work, various QAM constellations are discussed and detailed study of star QAM, XQAM, and HQAM is presented. Generation, peak and average energies, peak-to-average-power ratio, symbol-error-rate, decision boundaries, bit mapping, Gray code penalty, and bit-error-rate of star QAM, XQAM, and HQAM constellations for different constellation orders are presented. Finally, a comparative study of various QAM constellations is presented which justifies the supremacy of HQAM over other QAM constellations for various wireless communication systems and a potential modulation scheme for future standards.

optimization of all these factors may not be feasible at the same time. For example, if the power efficiency is targeted then lower order modulation is preferred which leads to lower bandwidth efficiency and lower data-rates. Hence, there is a trade-off between various expectations from modulation schemes. The optimization/trade-off of these parameters is application oriented in digital radio frequency (RF) system design. Considering a terrestrial microwave radio link design, bandwidth efficiency with low bit-error-rate (BER) is given high priority, since, the RF stations are connected to power source. Hence, power efficiency is not a prime concern and also receiver's cost/complexity is not considered because few receivers are required. On the other hand, in cellular communication, power efficiency is a major area of concern for system designers since the mobile phone sensors or Internet of things (IoT) type communications operate on a limited battery power. In mobile communication, both power and cost efficiency are stronger constraints than bandwidth efficiency, and thus have made their way into 5G standards.

A. VARIOUS DIGITAL MODULATION SCHEMES
Digital modulation schemes play a crucial role to achieve high data-rates with power and bandwidth efficiency. However, they are highly impacted by noise although, these schemes provide flexibility of multiplexing various information formats at high data-rates with good qualityof-service (QoS). Digital modulation is classified based on the variation in transmitted signal in terms of amplitude, phase or frequency with respect to the digital message signal. If the amplitude or phase of the transmitted signal is varied with respect to the message signal, the resultant signal is termed as amplitude shift keying (ASK) or phase shift keying (PSK), respectively. ASK and PSK are known as the linear modulation techniques since, they follow the principle of superposition and scaling. If the transmitted signal frequency varies with respect to the message signal, this results in frequency shift keying (FSK). Since, FSK is a non-linear modulation technique, it is not spectrally efficient as compared to the linear modulation schemes. Thus, linear modulation techniques are widely employed in wireless communications [2]. There is yet another advance modulation technique where both the amplitude and phase of the transmitted signal varies with respect to the digital message signal, known as quadrature amplitude modulation (QAM). Among the available digital modulation schemes, QAM and quadrature phase shift keying (QPSK) are widely used in communication standards because of the bandwidth and power efficiency. Further, M-ary QAM is more power efficient than M-ary PSK modulation [2] and thus, is widely preferred in the modern wireless communication standards. A flowchart of these modulation schemes with their family is shown in Fig. 1.

B. MOTIVATION
In modern and future wireless communication systems, power efficient high data-rate transmission with efficient utilization of the limited bandwidth is a major challenge. Power efficiency can be achieved by reducing the average transmit power of the constellation for a fixed BER. However, high data-rates for a limited bandwidth require high transmit power to maintain the same performance. Adaptive modulation is one such solution for spectrally efficient high data-rate communication with optimum power efficiency for the present and future wireless communication systems. In wireless communication, received signal-to-noise ratio (SNR) varies due to the multipath fading. In such scenario, adaptive modulation plays an important role by associating high data-rates to the channels with good channel conditions (high SNR) and low data-rates to the channels with bad channel conditions (low SNR) to improve overall system capacity for a limited bandwidth. Hence, an appropriate modulation scheme with variable constellation order is selected based on the channel condition for fairness. In adaptive modulation, constant SNR is maintained by varying various parameters such as transmitted power, data-rates, and modulation orders [1], [3]- [10]. Adaptive modulation is embodied in the present day wireless standards since the 3 rd generation (3G) [11] mobile networks. Adaptive modulation finds its applications in the high-speed modems [12], [13], satellite links [14]- [16], and in the applications where high QoS is to be maintained [17].
Recently, the family of QAMs has achieved tremendous attention in the wireless communication networks due to its capability to provide high data-rates with improved power and bandwidth efficiency. Family of QAMs mainly includes square QAM (SQAM), rectangular QAM (RQAM), cross QAM (XQAM), star QAM, and hexagonal QAM (HQAM). SQAM is usually preferred for even power of 2 QAM constellations. For better rate and channel adaptation, odd power of 2 constellation such as RQAM is preferred. However, RQAM is not a good choice as it has higher peak and average powers. To resolve this, a cross shaped constellation named as cross QAM (XQAM) is proposed which is obtained by removing the outer corner points of RQAM and arranging them in a cross shape such that the average energy of the constellations is reduced [18]. The requirement of high data-rates at low energy further directed the research towards more power efficient two dimensional (2D) hexagonal shaped constellation referred to as HQAM. HQAM has the densest 2D packing which reduces the peak and average power of the constellation and makes the constellation more power efficient than the other existing constellations [19]- [21]. Therefore, in this work, we have discussed these modulation schemes in details.
We view this as an area that over the years has lacked research attention, however with focus on bandwidth and energy efficiency in 5G and beyond communication systems with vastly varying data-rates needed in different industry verticals, there is a need to know how various modulation schemes compare, to be able to identify the relevant modulation scheme for an application in an industry vertical. Moreover, as we start mulling about 6G, we look at the evolution, design, and performance of futuristic higher-order modulation techniques, which play a crucial role in attaining high data-rates with bandwidth and power efficiency.

C. CONTRIBUTIONS
In this work, a detailed study of various higher order QAM constellations is presented. The basic QAM constellations such as SQAM and RQAM have been discussed in details in various works available in the literature. Hence, in this work, SQAM and RQAM are considered for the completeness of the study. Some works also discuss about the star QAM constellations. However, the details of star QAM such as their bit mapping, Gray code penalty, and decision boundaries are not completely available in the literature, and hence, in this work, bit mapping, Gray code penalty, and decision boundaries of star QAM are also discussed in details along with its generation, symbol-error-rate (SER), and BER analysis for various constellation orders. This study further explores XQAM constellations in details. Their generation, peak and average energies, peak-to-average-power ratio (PAPR), SER, decision boundaries, bit mapping, Gray code penalty, and BER for different constellation orders are presented. Since, now a days, the HQAM constellations have received significant research attention due to their densest 2D packing, in this work, various HQAM constellations with different constellation orders are discussed in details. Such details are not completely available in the literature. Hence, in this work, various regular and irregular HQAM constellations with their generation, peak and average energies, PAPR, SER, decision boundaries, bit mapping, Gray code penalty, and BER for different constellation orders up to 1024 are presented in details. An algorithm to obtain the irregular HQAM from the regular HQAM constellation is also proposed. Further, generalized approximate expressions of various selection parameters for irregular HQAM are also obtained. Finally, a comparative study of various QAM constellations is presented, and the applications of different QAM constellations in various areas are explored. In the following subsections, a detailed list of abbreviations used in this work and paper organization are presented.

D. LIST OF ABBREVIATIONS
Various abbreviations used in this work are shown in Table 1.

E. PAPER ORGANIZATION
In Section II, the family of QAMs is discussed in details. Section III, Section IV, and Section V discuss detailed study of star QAM, XQAM, and HQAM constellations, respectively. In Section VI, probabilistic shaping is discussed. In Section VII, applications of various QAM constellations in wireless communication systems are discussed. Finally, conclusions and future directions are presented in Section VIII.

A. EVOLUTION OF QAM
In the late 1950s, digital phase modulation schemes gained considerable research attention along with the digital amplitude modulation for transmission. It was an extension to the amplitude modulation by considering both the amplitude and phase modulations for transmission. The QAM was first proposed by Cahn [22] in 1960. Cahn's work was extended by Hancock and Lucy in [23] where it is realized that the performance of a circular constellation can be improved by placing the constellation points on the concentric circles with less points in the inner circle and more points in the outer circles. They named Type I to the Cahn's constellation and Type II to their newly proposed constellation. In 1962, Campopiano and Glazer proposed properly organized square QAM constellations and denoted it as Type III constellation [24]. From the results it was concluded that the performance of Type II constellation is little better as compared to Type III constellation, but implementation and detection complexity of Type III constellation is significantly low as compared to Type I and II constellations. All three constellations are shown in Fig. 2.
Later in 1971, Salz et. al. of Bell Labs proposed some actual QAM constellations [25]. They used both the coherent and non-coherent demodulations and implemented circular constellations with the help of 2 and 4 amplitude positions along with 4 and 8 phase positions. In [26], Simon and Smith proposed some tightly packed 2D spherical shaped honeycomb type structure (hexagonal QAM constellations), and simple generation and detection techniques for such constellations were discussed. In [27], Foschini et al. proposed optimum signal constellations with circular constellation envelop for large constellation size (M) which had optimum BER performance however their detection complexity was high. Foschini et al. extended their work in [28], and error optimization technique for optimum 2D signal constellations was proposed. In the same year, Thomas et al. [29] proposed empirically generated 29 amplitude-phase shift keying (APSK) signal sets, ranging from M = 4 to M = 128 to determine an optimum signal constellation through symbol-error-probability (SEP) bound for both the average and peak SNRs. In [18], Smith proposed various odd bits QAM constellations and a comparative study of rectangular and symmetric constellations was performed. In [30], Weber proposed a differential encoding technique for reducing the performance penalty by minimizing the phase ambiguity for multiple APSK systems. In [33], Forney et al. presented a detailed survey on various power efficient rectangular and hexagonal QAM constellations with coding fundamentals for band limited channels like telephone channels. These seminal works are also shown in Table 2. Since then, applications of all these proposed constellations have widely been seen in various communication systems and standards which are discussed in the upcoming Section.

B. VARIOUS QAM CONSTELLATIONS
In wireless communication systems, a power efficient signal constellation which targets the best QoS with high spectral efficiency has always been an exciting research area since the genesis of early modern mobile communication systems. A breakthrough in the research on modulation schemes happened with the invention of QAM in the early 1960's. Interestingly, various QAM constellations proposed years ago are still actively being used in commercial communication systems. With the recent developments in advanced signal processing algorithms, family of QAM has gained increased attention in present mobile communication systems and is widely adopted in various wireless communication standards and commercial applications. In QAM, information is encoded in both the amplitude and phase of the transmitted signal. Hence, for a given average energy, more bits per symbol can be encoded in QAM which makes it more spectrally efficient. An overview of some of these prominent QAM constellations is presented next.

1) SQUARE QAM (SQAM)
In 1962, Campopiano and Glazer [24] extended the works done by Cahn, Hancock and Lucy, by proposing properly organized square or rectangle shaped QAM constellations. Campopiano and Glazer denoted it as Type III constellation which is later known as QAM constellation. Type III QAM provides better error performance than the predecessors Type-I and Type-II QAMs. For QAM constellations, simple maximum likelihood (ML) detection with rectangular or squared boundaries is preferred. SQAM usually takes a perfect square shape for the even power of 2 signaling points 4, 16, 64, 256, 1024, 4096, and so on. SQAM has the maximum possible minimum Euclidean distance between the constellation points for a given average symbol power and requires simple ML detection technique. M-ary QAM requires less carrier-to-noise power ratio than the Mary PSK [37]. Thus, QAM is widely considered in various wireless communication systems and IEEE standards. Lower order QAM constellations have lower spectral efficiency, provide better cell overlap control and tolerance to distortion or SER/BER performance. However, high data-rates can be achieved with higher order QAM constellations at the cost of strict SER/BER requirements, severe cell-to-cell interference, smaller coverage radii, and hardware complexity. SQAM is widely deployed in various wireless standards such as in 3G/4G/5G digital video broadcast-cable/satellite/terrestrial communications, satellite communications, wireless fidelity (Wi-Fi), worldwide interoperability for microwave access (WiMAX), asymmetric digital subscribers line (ADSL), very high speed digital subscribers line (VDSL), power line ethernets, microwave backhaul systems and others. Various QAM applications with different constellation orders are presented in Table 3. As an example, 16-SQAM constellation is shown in Fig. 3.

2) RECTANGULAR QAM (RQAM)
RQAM is a Type-III QAM with non-square constellation which is commonly preferred for transmitting odd number of bits per symbol. As spectral efficiency depends on the modulation order M, higher spectral efficiency is achieved with higher value of M. Spectral efficiency also depends on the channel conditions. To improve spectral efficiency, adaptive modulation is used in practice to maximize spectral efficiency for a given channel conditions according to M [1], [6]. Further, both the even as well as odd power of 2 constellations are required for granular adaptation based on the channel conditions. For this, rectangular QAM (RQAM) is commonly preferred due to its generalized behavior as various modulation schemes such as multilevel ASK, binary PSK (BPSK), QPSK, orthogonal binary FSK, and SQAM as its special cases [38]. RQAM is a suboptimal QAM for M ≥ 16, since the average transmitted power required to achieve minimum distance is slightly greater than the average power required for the best M-ary QAM signal constellation. Thus, generalorder RQAM is preferred in practical telecommunication systems [39], [40]. RQAM constellations can be obtained by considering the constellation points in a rectangular shape. The obtained rectangular constellation can be parallel to in-phase axis or quadrature-phase axis with identical average energy. The 32-RQAM constellations are shown in the Fig. 4.

3) STAR QAM
For mobile radio applications, SQAM is widely preferred over the last few decades. However, SQAM is optimum if the communication channel is Gaussian. However, in practice, the Gaussian channel assumption does not apply and hence, severe shortcomings were observed for mobile radio channels over fading environment. Carrier recovery and automatic gain control (AGC) are required when SQAM is opted even with differential coding which are complex to achieve and sustain in a real time communication system. To avoid the use of AGC and to mitigate the false lock problem, star QAM constellation was proposed. Star QAM is a special case of circular APSK, which outperforms the SQAM in peak power limited systems. It consists of multiple concentric PSK circles with equal constellation points in each circle and identical phase angle between them. Amplitude and phase of the constellation points are mutually independent [41], [42]. Hence, differential detection can be applied successfully rather than the coherent detection, which omits the need for accurate phase tracking and channel estimation at the receiver [43], [44]. Due to the above favorable features, star QAM is adopted in various satellite communication standards such as in digital video broadcasting satellite standard-second generation (DVB-S2), digital video broadcasting-satellite services to handheld (DVB-SH), advanced broadcasting system via satellite, and Internet protocol over satellite [41], [45]. A detailed study of star QAM constellations is presented later.

4) CROSS QAM (XQAM)
For odd bits per symbol, RQAM is not a good choice as it has higher peak and average powers. Smith proposed [18] an improved cross shaped constellation to resolve this issue by removing the outer corner points and arranging them in a cross shape such that the average energy of the constellations is reduced. This type of constellation is named as XQAM. XQAM has lower peak and average powers than RQAM, and provides at least 1 dB gain over the RQAM constellations [20], [21], [46]. XQAM has been adopted in various communication system such as in ADSL and VDSL with 5-15 bits [47], [48], 32-XQAM and 128-XQAM in digital video broadcasting-cable (DVB-C) [49]. XQAM has commonly been applied in applications requiring blind equalization [50]- [52]. A detailed study of XQAM constellations is presented later.

5) HEXAGONAL QAM (HQAM)
Requirement for high data-rates at low energy further directed the research towards more power efficient 2D hexagonal shaped constellation referred to as HQAM. HQAM has the densest 2D packing which reduces the peak and average power of the constellation, and makes the constellation more power efficient than the other existing constellations. In most of the existing communication systems, SQAM is widely preferred due to its simpler ML detection than the HQAM constellation, however, with the advancement in technology, implementation complexity of HQAM detection is reduced considerably. Due to superiority of HQAM over other constellations, it can be considered in various applications such as in multicarrier systems [53], multiple-antenna systems [54]- [56], physical-layer network coding [57], small cell [58], optical communications [59], and advanced channel coding [60]- [62]. This work focuses on the HQAM constellation and a detailed study of HQAM constellations is presented later.

C. APPLICATION OF DIFFERENT QAM CONSTELLATIONS IN EXISTING COMMUNICATION SYSTEMS AND STANDARDS
QAM and QPSK are the most widely preferred constellations in various communication standards. Further, M-ary QAM has better bandwidth and power efficiency than M-ary PSK constellation [2] and thus, is widely employed in various modern communication standards. A list of various communication standards where different QAM constellations with various constellation orders are employed is shown in Table 3.

D. CHALLENGES IN QAM RESEARCH
The details of research challenges which were faced during the evolution and progress in the QAM research is presented in the evolution of QAM in the extended form. Further, some of the QAM research challenges are as follows: • In the early development phase of QAM, design of the modems to realize higher-order QAM constellations was a challenging task due to the lack of technological advancement, bulky circuitry, power, and cost inefficiency. With the evolution of technological advancements in signal processing algorithms and circuitry design, QAM has found applications in various wireless standards. • Over the years, with the increase in number of users and multimedia applications, bandwidth and power efficient high data-rate transmission is a major challenge. The demand of high-data rates directs the research towards the higher order compact 2D constellations. Hence, in this work, the design and modeling of various higher order QAM constellations, and various challenges like optimum power efficiency, peak-to-average power ratio, bit mapping, and Gray code penalty are discussed. • The detection complexity for various QAM constellations is also a major issue that gives rise to the BER. Also, increase in constellation order M will further increase the detection complexity. Hence, for various QAM constellations, this has also been explained in detail in this survey article. • Impact of thermal noise, aging, noise, and distortions are severe with the increase in bits/symbol specially in case of QAM which consists of more number of bits/symbol to support high data-rates. Thus, there are high chances of symbol lock synchronization failure at the receiver which results into error with loss of information. To overcome this, more power is required during the transmission. This indicates about the trade-off between the required high data-rates and power efficiency for a particular wireless communication application. Hence, in this work, we have discussed various compact 2D QAM constellations to improve the power efficiency. Further, design of filters and modulators is very complex for the next generation mobile communications at very high frequencies due to the severe impact of non-linearity of the devices. • An appropriate power efficient modulation scheme with variable constellation order is always been a point of interest for any wireless application to achieve desired date-rates for a given channel condition. This has been explained in details with the applications of different QAM constellations in various present and future wireless communication systems in the upcoming Application Section. Further, it will be interesting to see the application of various higher order QAM schemes in next generation communication systems such as in optical wireless communication (OWC), non-orthogonal multiple access (NOMA), intelligent reflecting surface (IRS), and integrated access and backhaul (IAB). The design of circuitry for accurate synchronization of carrier phases is also an open research problem.

III. STAR QAM CONSTELLATIONS
In the initial studies, SQAM was widely preferred in mobile radio applications. However, severe limitations of SQAM were observed over mobile radio channels (with multipath fading) [36]. Since, SQAM require both the carrier recovery and AGC at the receiver, mainly two AGC methods; averaging method and forced update method were adopted for SQAM. Both performed similarly poor, however, averaging method was preferred over the other, as it has no bandwidth overhead. There were some serious issues with carrier recovery as false locks were observed not only at multiple of 90 • , but also at 26 + (l × 90) and 52 + (l × 90) degrees with l = 0, 1, 2, 3 as highlighted in [36]. To improve the performance and to avoid the need of AGC and false lock problem, star QAM was proposed [36]. Star QAM consists of multiple concentric PSK circles with equal constellation points in each circle and identical phase angle between them. Star QAM consists of 8 possible lock positions for lower order constellations (16 and 32-star QAM constellations) which increases for higher order constellations (64-star QAM constellation). All the constellation points are rotated with a fixed angle for these positions. After differential encoding, all the false lock problems can be eliminated. Modeling of various star QAM constellations is shown in Fig. 5.

A. DECISION REGIONS
Decision regions for star-QAM are not straight-forward as compared with SQAM/RQAM regions. It does not consist of horizontal and vertical boundaries only. For decision making, inclined regions with some phase angles are also required. Let us consider 16 and 32-star QAM constellations. For decision making, initially, the entire constellation is divided into 8 sub-regions with phase angle (2l + 1) × π/8, for the constellation points situated at an phase angle of l × π/4, where l = 0, 1, . . . , 7. Then in each sub-region, the constellation In this case, symbol decoding is converted into simple comparison between the current and previously decoded symbol. Let, the amplitude of inner and outer rings be r i and r o , respectively. Let, the received symbol amplitude are r t and r t+1 at time instance t and t + 1, respectively. Then the first bit is set to 1 only if [100] r t+1 otherwise set to 0. If θ t and θ t+1 are the phase angles of the received symbol at time instance t and t + 1, respectively, then the demodulated angle will be [36] Finally, the decoded angle is quantized to the closest multiple of π/4, and then a lookup table is prepared to decode the remaining bits. For 16 and 32 points star QAM constellations, decision regions are shown in Fig. 6. Similar procedure can be applied for the higher-order star QAM constellations.

B. SER ANALYSIS
For an M-ary equiprobable digital modulation signal with arbitrary placement of constellation points in 2D plane, SEP expression over the additive white Gaussian noise (AWGN) channel can be given by the weighted sum of probabilities of each sub-regions for all the constellation points as [101]  where N S denotes the number of sub-regions, μ u represents a priori probability that a symbol belongs to the u th subregion, and a u , ζ u , and φ u are some important parameters related to the u th sub-region for analysis which are shown in Fig. 7. Considering a 16-star QAM constellation for average SEP (ASEP) calculation as shown in Fig. 7. For analysis, perfect phase tracking is assumed at the receiver. Inner and outer ring's radius are denoted by r i and r o , respectively, which corresponds to a ring ratio R = r i /r o . For symmetry, the ASEP is written in terms of error probabilities when symbol S 0 and S 12 are transmitted as dθ, respectively, where L u denotes the lengths (u = 1, . . . , 4) as shown in Fig. 7. Thus, the ASEP expression of 16-star QAM can be given as [102] where Similar procedure can be followed for the higher order star QAM constellations.

C. BIT MAPPING, GRAY CODE PENALTY, AND BER
Star QAM is a special case of circular APSK, where both the amplitude and phase of constellation points vary. It consists of multiple concentric PSK circles with equal constellation points in each circle. Further, there are total 8 possible phase angles for lower order constellations which will increase with the constellation size. Constellation points in all the circles have identical phase angle between them. Let us consider 16 and 32-star QAM constellations whose bit mapping is shown in Fig. 8. As there are total 8 possible phase angles, last 3 bits of the bit stream are used to represent the phase angles. Remaining starting bits are used to represent the change in amplitude of the constellation points in the concentric circles. For 16-star QAM (Fig. 8, left), there are 2 concentric circles, each having 8 constellation points. There are total 8 phase angles l × π/4 with l = 0, 1, . . . , 7. 1 Hence, the last 3 bits of the bit stream represent the change in phase angle by π/4 with the adjacent constellation point in the circle. If 000 is considered it means the symbol is transmitted with no phase change, if 001 is considered that means the signal is transmitted with π/4 phase shift than the previous symbol and so on. Remaining 1 bit (starting bit) represents the change in amplitude of the concentric circles. If first bit is 0, inner circle is selected, else outer circle is considered. For 32-star QAM (Fig. 8, right), there are 4 concentric circles, each having 8 constellation points. Hence, the starting 2 bits are used to represent the change in amplitude and the last 3 bits are used to show the change in phase. Similar procedure can be used for all the star QAM constellations. For 64-star QAM constellation, there are 16 possible lock positions. Hence, the last 4 bits of the 6 bits stream will be used to represent the change in phase, and the starting 2 bits denotes the change in amplitude ring.
In star QAM, each symbol shares its boundaries with maximum of 4 adjacent symbols (as can be seen from 32star QAM). The inner-most and outer-most ring symbols share their boundaries with 3 adjacent symbols. Hence, bit mapping is performed in such a manner that each symbol differ with maximum of 1 bit from its nearby symbol. Hence, star QAM constellations have perfect Gray coding, and their 1. The star QAM constellation can be rotated with any arbitrary angle, however, a fixed phase difference should be maintained between the adjacent constellation points in a circle.
Gray code penalty (G P ) is 1. The G p is introduced by Smith which is defined as the average bit difference between the two neighboring symbols in a constellation as where M represents the constellation order, s k denotes the k th data symbol, G s k P is the Gray code penalty of the k th data symbol. For a given SEP, approximated BEP expression can be given as [18] where P s represents the SEP which is given in (3) for the star QAM. Finally, from (7), the BEP of star QAM can be achieved. However, if perfect coherent detection is assumed, the BER performance of SQAM is better than the star QAM as in [103], where BER performance of different 16 point constellations is compared. For the locally generated reference carrier, ambiguity arises in exact phase orientation due to the symmetric constellation which can be resolved through differential encoding. However, error propagation occurs in differentially encoded bits which degrades the BER performance. Still, the differentially encoded SQAM (DE-SQAM) outperforms the star QAM which is explained mathematically in [103] for 16 point constellations.
In this work, comparative study of various QAM constellations with a fixed separation of 2d between the constellation points is presented. However, fixed separation of 2d between constellation points is not maintained in star QAM, and hence, its comparison with the other QAM constellations is not presented due to the lack of common platform.

IV. XQAM CONSTELLATIONS
For even bits per symbol, the transmitted signal is modulated with SQAM, whereas for odd bits per symbol, RQAM is commonly preferred. However, as discussed above, RQAM is not a good choice since RQAM constellations have high peak and average energies and hence, are power inefficient. To overcome this, Smith proposed a XQAM constellation which provides at least 1 dB gain over the RQAM constellations due to its reduced peak and average powers than RQAM [20], [21], [46]. If channel conditions are good, then instead of going for a constellation size of M from 2 2n to 2 2n+2 as in SQAM, a more gradual increase from 2 2n to 2 2n+1 is possible with XQAM which can adapt to the channel conditions more efficiently than RQAM [65], [104]. Here n denotes the number of bits per symbol which can take any value greater than or equal to 2. For XQAM constellation, M = 2 2n+1 , hence, M = 32, 128, 512, 2048, . . . , are possible constellations with XQAM. Construction of 32-XQAM and 128-XQAM from 32-RQAM and 128-RQAM constellations are shown in Fig. 9. Here, corner points are shifted to top-and-bottom near the y-axis such that the peak and average energies of the constellation are reduced. In the following subsections, detailed study of the decision regions, SER, bit mapping, Gray code penalty, and BER of various XQAM constellations is presented.

A. DECISION REGIONS
Decision regions for both SQAM and RQAM are quite straight-forward, since, in-phase and quadrature-phase bits have vertical and horizontal decision regions, respectively. However, only the horizontal and vertical decision regions are not sufficient for XQAM constellations. In XQAM, three types of symbols exist; edge symbols, corner symbols, and interior symbols. As the XQAM constellations are symmetric around the origin, we consider only one quadrant for analysis which is shown in Fig. 10.
Number of interior, edge, and corner symbols/points in each quadrant are given as respectively. For XQAM constellations, the end columns symbols are moved to new cross type positions in the constellation. For interior symbols, decision regions are closed squares, however, for edge symbols, decision regions are semi-infinite rectangles. Decision regions for corner symbols are not straight-forward and consist of 45 • lines along with the horizontal and vertical regions.
As an example, we consider a 5 bits (i 1 i 2 i 3 q 1 q 2 ) 32-XQAM constellation. Decision regions for i 1 and q 1 bits are vertical and horizontal regions. However, for i 2 , i 3 , and q 2 bits, decision regions are formed with horizontal, vertical, and 45 • inclined regions, which are shown in Fig. 11. Similar procedure can be applied to obtain the decision regions for the other XQAM constellations.

B. SER ANALYSIS
The 32-XQAM and 128-XQAM constellations are shown in Fig. 9. Here, corner points are shifted to top-and-bottom near the y-axis such that the peak and average energies of the constellation are reduced. For equiprobable constellation points, average symbol energy of XQAM is given as where λ = 48 31M−32 . Thus, the SNR can be given as where η = d √ N 0 /2 represents the minimum distance between a constellation point and decision boundary, N 0 /2 denotes the two sided power spectral density of noise, and η = √ 2λγ from (10). For the interior and edge symbols, respective SEP expressions in terms of η can be given as [40] (11) where Q(·) represents the Gaussian-Q function. SEP expressions for the i th corner point from the bottom-to-top, or from left-to-right can be given as [65] where represents the probability density function (PDF) of the standard normal distribution. Finally, considering all the points, exact average SEP for the generalized XQAM is given as Finally, solving (13), the conditional SEP of general order XQAM constellation in AWGN channel can be given as [65] P X e (e|γ ) where ) is related to the 1D and 2D Gaussian Q-functions as

C. BIT MAPPING, GRAY CODE PENALTY, AND BER
A detailed study about XQAM labeling is discussed in [18] and given for completeness. Binary and Gray codes for both the dimensions of an RQAM constellation are defined as where g l k and b l k represent the l th bit of the k th symbol of Gray and binary codes, respectively. However, these relationships are not valid in XQAM where the far left and far right columns symbols are shifted to a new position to form a cross structure. Hence, perfect Gray coding is not possible in XQAM. The problem of modulation and labeling was solved by Smith [18] by adding one extra bit to create a 2n + 2 bits Gray coding. Here, this extra bit is not transmitted, it is only used for modulation and demodulation, and all the BER analysis is performed with 2n + 1 bits pseudo Gray codes. However, it is also not a feasible solution as it requires 1 extra bit for mapping. From Fig. 12, it is observed that with the use of the 2n + 1 bits pseudo Gray codes (5 bits Gray coding for 32-XQAM), topmost or bottom most blue colored symbols differ by 2 bits from its neighboring newly generated black symbols, however the other inner blue symbols only have one bit difference with its neighboring symbol. From Fig. 12, it is also observed that the 2n + 1 bits pseudo Gray codes are imperfect, and hence, G p arise which is defined in (6). For example, G p of 32-XQAM is calculated as The generalized G p for the higher order M-ary XQAM constellations can be given as [104]. Please note that, for a constellation with perfect Gray coding, G p is always 1. In any constellation, bit mapping should be performed very precisely and carefully to minimize the BEP for a given SEP for the constellation. Finally, substituting the SEP given in (14) for XQAM, approximated BEP expression can be obtained. From (7), it is clear that the BEP is directly dependent on G P since log 2 M is constant. Hence, optimum bit mapping is required to minimize the G P for minimum BEP and any arbitrary bit mapping will increase the G p , and consequently the BEP for a given SEP.

V. HQAM CONSTELLATIONS
Energy efficient 2D signal constellation which can save considerable transmission energy is the most basic requirement of the present and future wireless communication systems. As we see 5G being the first standard to incorporate battery life, the discussed HQAM constellation can be considered for 6G standards as an energy and bandwidth efficient futuristic modulation scheme. For a 2D signal constellation, SER is mainly affected by the minimum separation between the two neighboring constellation points, and average symbol energy which depends upon the mean squared distance of constellation points from the origin. Considering these points in mind, an optimum 2D hexagonal lattice based HQAM constellation is proposed where constellation points are situated at the center of the concentric spheres. For a minimum distance separation of 2d between the two adjacent constellation points in hexagonal constellation, area of the hexagonal region is 2 √ 3d 2 which is 0.866 times the area of the rectangle regions with the same dimensions. This allows hexagonal regions to provide around 0.6 dB gain over the rectangle regions [33]. HQAM is also termed as triangular QAM (TQAM) in the literature. Placement and separation of the constellation points on the 2D plane is same for both the HQAM and TQAM constellations. If the constellation is classified as HQAM, the constellation points are considered at the center of an equilateral hexagon and separated with the neighboring point hexagon with 2d distance. If the constellation is classified as TQAM, then the constellation points are considered at the vertices of well connected equilateral triangles of side-length 2d. However, the decision boundaries are hexagonal shaped as shown in Fig. 13.
Based on the placement of the constellation points, HQAM constellations are further categorized into regular and irregular HQAM structures. Regular HQAM has comparatively simpler detection, its power efficiency or BER performance can further be improved for larger values of M. The irregular HQAM provides improved power efficiency and optimum performance, however, at the cost of increased detection complexity [105]. Since, the distance between the constellation points is equal, in the inner regions, the ML detection boundaries are in equilateral hexagonal shapes. However, at the boundaries, detection regions vary as per the constellation geometry.
Constellation for regular HQAM, are symmetric around the origin [106], [107]. For, even power of 2, regular HQAM constellations are in square shapes [106] such as for M = 16, 64, 256, 1024, . . . , however, for odd power of 2, regular HQAM constellations are in cross shapes [107] such as for M = 32, 128, 512, . . . For, irregular HQAM, constellations are in circular shapes as M increases which makes the irregular HQAM constellations more compact than others [108]. For example, the regular and irregular (optimum) 16-HQAM constellations are shown in Fig. 14. Further, various regular HQAM and irregular HQAM (optimum) constellations are shown in the Appendix in Fig. 19 and Fig. 20, respectively. Average energy of HQAM constellations is less than the SQAM constellations for any value of M except for M = 4. This is due to the fact, that 4-HQAM has same average energy of the constellation as 4-SQAM, however is having higher value of NNs. Further, irregular HQAM constellations have comparatively less average energy than the regular HQAM constellations, however, are more complex to generate and detect.
In the following subsection, generation of irregular HQAM from the regular HQAM constellation is explained.  M=number of constellation points. 2: Initialize: Generate regular HQAM, set counter l = 1.
Shift one row of the generated regular HQAM to X-axis. 4

: else
Shift one row of the generated regular HQAM to √ 3/4 distance above X-axis. 5 Process to generate an irregular HQAM (optimum) constellation (Fig. 15) is shown in Algorithm 1, where, d vs represents the vertical spacing, E c represents the energy of a constellation point, and E c represents the array of constellation points energies.

A. DECISION REGIONS
ML detection is the most popularly used detection scheme where distance between each signal point and the received symbol is calculated, and decision (as an estimate of the transmitted symbol) is made to the nearest symbol. ML detection has less detection complexity however, is not applicable for HQAM detection in a straight-forward manner. Thus, an optimum and simple detection scheme for HQAM constellations is proposed where entire constellation is initially divided into vertical regions. A signal point is first located in a region and then the nearest constellation point in that region is found. This process ultimately reduces the number of distance calculations as compared to the traditional ML detection scheme applied for HQAM detection [106].
In Fig. 16, a simplified detection technique for 64-ary irregular HQAM constellation is shown which is explained as follows. Initially, the entire constellation is divided into R 1 − R 16 regions. When a symbol, lets say z = x + iy is received (where x and y are the constant values and j = √ −1), based on the the arbitrary value x, one of the sixteen regions is selected. Further, imaginary values for the decision points are calculated and decision for a symbol is made with a comparison between the imaginary value y of the demodulated symbol and calculated imaginary boundaries. For different regions, calculated imaginary boundaries are shown in Table 5. For illustration, depending on the arbitrary value of x, initially a region (R 9 ) is selected for the demodulated symbol z as shown in Fig. 16. Next, to detect 9 signaling points which lie in R 9 region, 8 boundary points (P 1 − P 8 ) are marked on the dotted line and their imaginary values are calculated to estimate a signaling point out of the 9 signaling points. The imaginary values of 8 boundary points for R 9 region are shown in Table 4. Following similar procedure for the other regions, imaginary values for the boundaries points are calculated and summarized in Table 5.
Hence, number of vertical decision regions (N R ) and maximum number of constellation points (N P ) per region are required to decide the detection complexity of HQAM constellations. Let, O(N R ) and O(N P ) define the computational complexity in obtaining the decision regions and maximum  possible constellation points per region, respectively, then the overall detection complexity (O(M d )) of HQAM constellations can be calculated as [105] (18), the overall detection complexity can be calculated as 97. It is to be noted that with the increase in constellation order (M), the number of decision regions and the maximum number of points per region also increases and hence, the overall detection complexity will also increase. A detailed analysis of detection complexity can be seen in [105]. Similar procedure can be followed for the other HQAM constellations for symbol detection and for the overall detection complexity. For various irregular HQAM constellations (M varies from 4 to 1024), decision regions are shown in the Appendix in Fig. 21.

B. SER ANALYSIS
Let us consider a set S M ∈ {s 0 , s 1 , . . . , s M−1 } for M-ary HQAM constellation. SEP for HQAM constellation is based on the shape of the decision regions. For this, a suitable correction to NN approximation is applied [109] which is expressed as Here τ represents the average number of NNs and is defined as τ = 1 M M−1 k=0 τ (k) wherein τ (k) is the number of NNs for k th symbol s k , Q(·) represents the Gaussian Qfunction, d is the half the distance between constellation points, and σ is the AWGN standard deviation.
For illustration, simplest HQAM structure is the 3-PSK structure as shown in Fig. 17. The 3-PSK consists of three symbols s 0 , s 1 , and s 2 . Starting with s 0 which has two NNs s 1 and s 2 . s 1 is also an NN of s 2 . Thus, s 1 and s 2 are the couple of NNs of s 0 . Two half planes with partial overlapping cover the error region in NN approximation which produces an overestimation of SEP. This can be clarified from Fig. 17. Thus, exact SEP expression for 3-PSK is given as where C 1 is a correction term, used to compensate for the double counting of overlapping. For 3-PSK, C 1 is given in [40]. However, for simplification, integral can be avoided and C 1 is approximated with C 2 as [110] Hence, approximate SEP expression for 3-PSK can be given as This correction approach is easily applicable for generalized HQAM constellation. In HQAM, multiple overlaps with a 2π/3 angle occur which generate couple of NNs. Therefore, for AWGN channel, approximate SEP expression for HQAM can be given as where is the average of couple of NNs, wherein τ c (k) represents couple of NNs for s k symbol. Further, considering K = 1 γ ( d 2σ ) 2 with average transmit SNR γ = P s /N 0 , and substituting C 2 from (21) in (23), generalized conditional SEP of HQAM is given as Here, K is related to the average energy of the constellation which varies with M. Further, τ and τ c are the average number of NNs and couple of NNs, respectively, which also vary according to M. Various regular HQAM constellations and irregular HQAM constellations (optimum) are shown in Fig. 19 and Fig. 20, respectively. Based on the constellations (regular or irregular HQAM) and modulation order M; different values of K, τ , and τ c are given in Table 6 which are used to select various HQAM constellations with different constellation orders. . However, for an arbitrary power of 2 constellation, no such pattern can be found in the literature for different values of K, τ , and τ c for both regular and irregular HQAM constellations. A detailed study on various HQAM or TQAM constellations can be seen in [19], [33], [105], [107]- [109]. Solving the equations given for K, τ , and τ c , generalized approximate values of K, τ , and τ c , for a well connected M-ary irregular HQAM can be given as

C. BIT MAPPING, GRAY CODE PENALTY, AND BER
BER performance is significantly affected by the bit mapping of the constellation points. For an M-ary constellation, bit mapping should be done such that the bit difference between the neighboring points is minimum. The optimum bit mapping is achieved with the least average number of bit difference between neighboring symbols. In SQAM, each constellation point shares its boundaries with maximum 4 constellation points and differ with its neighbors by only a single bit. Thus, SQAM has the perfect Gray coding and its G P is always 1 [111]. However, for HQAM, perfect Gray mapping is not possible [112], since each symbol shares its boundaries with maximum of six neighboring symbols. This generates G P which is expressed as where s k denotes the k th data symbol, G s k P is the Gray code penalty of k th data symbol, b d (s k , s j ) represents the bit difference of k th data symbol with its neighboring symbol s j , and N(s k ) is the NN count of k th data symbol s k .
In any constellation, bit mapping should be performed very precisely and carefully to minimize the BEP for a given SEP for the constellation. For a given SEP, approximated BEP expression is given in (7) which is directly proportional to G P . Hence, optimum bit mapping is required to minimize G P to get minimum BER. Optimum bit mapping can only be obtained with exhaustive search over all the possible bit mappings for the constellation. Further, it should be noticed that there is not an unique optimum bit mapping, others may exist which have the same G P . Practically, for an M-ary signal constellation, there are M! bit mappings and finding an optimum bit mapping for optimum G P for a large signal constellation is extremely tedious and time consuming. Thus, a sub-optimum bit mapping is preferred for a signal constellation with M > 16. For an M-ary signal constellation, each symbol is represented with n bits, and 2 n bit streams appear to represent all the symbols in the constellation. In practice, a sub-optimum bit mapping is preferred which is performed in three steps. Initially, regular HQAM constellations are generated with the same bit mapping as preferred for SQAM (or RQAM) constellations. Further, irregular HQAM constellations are generated by shifting the regular HQAM constellations such that the maximum signal points get matched on I/Q coordinates. Next, for the matched signaling points of irregular HQAM, the same bit mapping is preferred as used for regular HQAM constellation. Finally, for the remaining symbols, a partially optimized bit mapping is considered where each symbol picks its favorable NNs symbols from the available ones such that it has minimum bit difference with its NNs. As an example, for M = 64, bit mapping for irregular HQAM is shown in Fig. 18. For M = 64 = 2 6 , 6 bits are required for mapping. Initially, 8 rows and 8 columns are selected, and 3 bits are assigned to represent the rows and rest of the 3 bits represent columns. Lower three bits represent the rows and upper three bits represent the columns. Bit mapping for irregular HQAM is completed in three steps. In the first step, regular HQAM is formed similar to SQAM formation. The same bit mapping is adopted for regular HQAM constellation as preferred by SQAM constellation. Further, irregular HQAM constellation is formed with a little shift in constellation points of regular HQAM on I/Q plane. For the matched points in irregular HQAM constellation, the same bit mapping is preferred as in regular HQAM constellation. Finally, for the remaining points, a sub-optimum bit mapping is performed by considering that s k consists of the NNs with minimum G P from the available signaling points. Thus, the G P for irregular 64-HQAM is 1.35. Finally, substituting G P and SER in (7), approximate BER expression for HQAM can be obtained. As the G P of regular and irregular HQAM constellations are always greater than 1, the bit error performance of HQAM constellations is always poor as compared to the symbol error performance because G P of SQAM constellations is always 1.

D. COMPARISON OF IRREGULAR HQAM CONSTELLATION WITH OTHER QAM CONSTELLATIONS
A comparative analysis between the irregular HQAM and other QAM constellations for different constellation orders is illustrated through Table 7. Here, a fixed Euclidean distance of 2d is maintained between the neighborhood constellation points for all the constellations and their average constellation energy or power and PAPR are calculated. From the Table 7, it can be concluded that for all the constellation orders (even as well as odd bits constellations), irregular HQAM has the reduced peak and average energies or powers. Hence, irregular HQAM is the most energy or power efficient constellation and can be concluded as the optimum constellation which provides better ASER performance than the other constellations. For various SISO RF relay networks, superiority of HQAM is verified through the results shown in [20], [21], [46], [113]. For various multiple-input and multiple-output (MIMO) RF relay networks, this is verified through the results shown in [114]- [116]. For the first time, for a mixed RF/free space optics (FSO) based relay network, the superiority of HQAM is verified through the results obtained in [117]. Further, [118]- [120], authors have justified HQAM superiority in NLoS ultraviolet communication (UVC) based relay networks.

VI. PROBABILISTIC SHAPING
The development of optical transport networks is pushed forward due to the growing demands of Internet traffic in the past decades. Advanced modulation schemes (specially the family of QAMs) are adopted for better spectral efficiency; and various multiplexing techniques are implemented for improved capacity. It is observed, for transmission and detection in the modern day optical networks, usually uniform QAM constellations are used which results in a loss of around 1.53 dB gain towards the Shannon limit [121]. However, the Shannon limit can be achieved by implementing shaping and coding techniques. Shaping gain can be achieved through probabilistic and geometric amplitude shaping. In uniform QAM constellation, all the constellation points are equiprobable and equidistant on the 2D plain. However, in geometrically shaped QAM, constellation points are non-equidistant on 2D plain, whereas in probabilistically shaped QAM, constellation points are nonequiprobable on 2D plain. In this work, all the constellations discussed in the previous sections are geometrically shaped. However, we can further improve their performance by probabilistic shaping. Considerable works on the probabilistically shaped QAM constellations are observed in the literature. Works on the comparative study of both the probabilistically and geometrically shaped QAM constellations are presented below.
In [122], [123], probabilistically shaped QAM constellations are compared with the uniform QAM constellations for the optical communication system. In [124], for equidistant constellation points, probabilistic amplitude shaping is applied to bipolar ASK which is applied to SQAM. In [125], for non-linear fiber channel, probabilistic shaping of QAM is performed. In [126], for AWGN channel, both the geometric and probabilistic shaping are studied. In [127], generalized pairwise optimization scheme is reviewed for geometrically and probabilistically shaped constellations. In [121], [128], [129], comparison with geometrically shaped QAM; the probabilistically shaped QAM is presented for optical communication system. In [130], for IM/DD based FSO channel, probabilistic shaping for M-ary pulse amplitude modulation is proposed. Most of the works focus on the probabilistic and geometric shaping in optical communication system, however, this can also be extended in wireless systems for improved power efficiency along with the spectral efficiency.

VII. APPLICATION OF QAM CONSTELLATIONS IN WIRELESS COMMUNICATION SYSTEMS
The QAM was first proposed by Cahn [22] in 1960 and later extended by Hancock and Lucy in [23], who termed Type I to the Cahn's constellation and Type II to their proposed constellation. Later in 1962, Campopiano and Glazer proposed properly organized square QAM constellation and denoted it as Type III constellation [24]. There were some initial works [18], [25]- [29], where various QAM constellations were proposed. Since then, such constellations have received tremendous attention for high data-rates in various fields of communication with different applications. These areas of communication with applications are described below in details.
Further, except for some initial works on HQAM or TQAM constellations [26], [28], [29], [33], [101], significant research attention in BER/SER analysis of HQAM or TQAM constellations is reported recently in [19]- [21], [46], [60], [105]- [109], [112]- [119], [131]- [134]. In [106]- [109], [112], Park has presented an extensive research for various regular and irregular TQAM (or HQAM) constellations with different constellation orders. In these works, various TQAM constellations with their peak and average powers, bit mapping, detection regions, etc. are presented separately. In [60], for non-power of 2 HQAM constellations, multilevel coded modulation scheme is presented. In [135], performance of θ -QAM family (including SQAM and TQAM) is analyzed in both the AWGN and Nakagami-m fading channels. In [19], a non-relay system is considered and SEP of HQAM over Rayleigh fading channel is derived. For a multi-relay network over independent and nonidentically distributed (i.n.i.d.) Nakagami-m fading links, ASER performance of various signal constellations such as HQAM, RQAM, XQAM, De-QPSK, and π/4-QPSK are analyzed in [20]. In [21], impact of imperfect CSI is observed on the performance of amplify-and-forward (AF) relay system over i.n.i.d. Nakagami-m fading links and closedform expressions of outage probability and ASER of HQAM and RQAM constellations are derived. In [105], Abdelaziz and Gulliver have examined both power of 2 and non-power of 2 constellations. Further, BER performance with detection complexities of regular, irregular, and sub-optimum TQAM constellations are analyzed. Work done in [20], [21] is extended in [46], where a multi-relay AF system over i.n.i.d. Nakagami-m fading channels with both integer and non-integer fading parameters is considered in presence of imperfect CSI and NLPA at the relay, and analytical expressions of outage probability, asymptotic outage probability, and ASER of HQAM, RQAM, and XQAM constellations are derived.
The major areas of applications for various QAM constellations are: • Mobile communications • Broadcasting-cable, satellite, microwave • Spatial diversity-multicarrier -(OFDM, NOMA, MIMO), cooperative relaying • Emerging areas for high data-rate communications

A. MOBILE COMMUNICATIONS
Due to increase in the number of wireless devices, the requirements of voice, data, and video applications are increasing day-by-day. However, to access the limited spectrum, efficient spectrum utilization techniques are required. At the same time, Internet and other data related wireless applications requires very high data-rates. Spectrally efficient high data-rate can be achieved through adaptive digital modulation schemes at an affordable cost. Hence, research is directed towards various spectrally and power efficient compact 2D constellations. In this subsection, we have explored a detailed survey on various signal constellations, adopted in mobile communications from the early days till now.
In [25], authors presented performance of a digital data modem transmission, employing combined amplitude and phase modulation. In [31], authors experimentally implemented the BER of 4-PSK, 8-PSK, and 16-QAM, for a bit-rate of 140 Mbps by considering high capacity digital radio-relay system. In [136], authors investigated the impact of multipath fading on a 200 Mbps 16-QAM digital radio system experimentally. In [137], Oetting compared various constellations up-to 16 constellation orders over AWGN channel which were applicable to digital radio systems. In [138], Prabhu determined the co-channel interference immunity of high capacity QAM constellations for digital radio applications. In [139], the authors analyzed the impact of multipath fading over a high-speed digital radio systems operating in 4 − 11 GHz band by considering offset 4-PSK, conventional 8-PSK and 16-QAM constellations. In [140], authors concluded that the decision feedback equalizer with few taps is efficient in the presence of multipath fading and constructed a 140 Mb/s 16-QAM modem. In [141], Hill et al. presented the performance of nonlinearly amplified (NLA) 64-QAM and the sensitivity over group delays and amplitude distortions was discussed for digital radio systems. In [142]- [145], authors presented their works on QAM constellations in digital radio systems. In [146], authors proposed a blind equalization technique and verified its performance of one and two carrier systems with QAM and V29 constellations. In [33], Forney et al. presented a detailed survey on various power efficient RQAM and HQAM constellations for band limited channels like telephone channels. In [147], authors proposed a futuristic mobile communication system based on the integration of a high-capacity digital cellular mobile radio system with a packet-switched routing network and discussed various modulation formats including QAM. In [148], authors presents a comparative study of 16, 32, 64, and 128-QAM in a digital radio system environment by considering the effects of filtering, interference, amplifier non-linearities, and selective fading. In [149], authors analyzed the performance of 256-QAM over distorted channels by considering amplitude distortions such as linear, parabolic, and sinusoidal characteristics along with group delay. In [150], authors analyzed the performance of 512-QAM by considering the effects of linear channel distortions. In [151], performance of transversal and decision feedback equalizers for 16-QAM and 64-QAM digital radio systems with tap spacing was analyzed. In 1987, Sundberg et al. [34], [35] considered SQAM constellation for voice communication over a Rayleigh fading channel. This was the first major consideration of QAM in mobile radio applications. In [152], authors described the performance of a 400 Mbps transmission capacity 256-QAM modem and introduced some novel approaches such as forward-error-correction (FEC) to reduce residual errors, good phase jitter performance and no false lock phenomenon to attain good system performance. In [153], authors investigated the performance of M-ary QAM by using times-four method of carrier recovery in a digital radio link. In [154], authors evaluated the effects of transmit amplifier nonlinearities on digital radio link performance by considering 256-level constellations. In [155], authors presented the performance of multilevel QAM radio system over selective fading by considering adaptive baseband equalization to reduce the signatures. In [156], authors proposed a progressive filter concept for co-channel transmission with negligible interference employing 16-QAM with a 0.19 roll-off at 140 Mbps rate. In [157], technical and design requirements for 256-QAM data in voice modems such as signal processing for implementation, error correction codec etc. were described. In [158], authors reviewed the importance of coded modulation for a 1.544 Mbps data-rate in voice modems employing 256-QAM. In [159], authors presented the performance of a dual polarized M-ary QAM up-to 64 levels by considering coherence between cross talk and the desired signal. In [160], authors proposed an equalization technique for multilevel QAM systems which eliminates linear distortions with easy implementation at high data-rates. In [155], authors presented the performance of multilevel QAM radio system in selective fading by considering adaptive baseband equalization to reduce the signatures. In [161], performance of dual 16-QAM system over dispersive radio channels was analyzed by considering the cross-polarization interference cancelers. In [162], analysis of 140 Mbps 64-QAM is performed by considering the digital adaptive equalization technique. In [163], comparative study of 16, 32, 64, 128-QAM constellations at 140 Mbps by considering both linear and non-linear equalizers was presented. In [164], highly precise crosspolarization interference canceler was developed by using digital transversal filter and performance of 256-QAM at 12.5 Mbps was examined. In [165], authors performed a simulation case study of M-ary QAM constellations for a high capacity digital radio link, subjected to frequency selective fading with different types of channel encoding. In [166], authors presented the performance of 16-QAM, 64-QAM, 49-QPR, and 81-QPR over a 45 Mbps line-of-sight digital radio systems. In [167], authors investigated the impact of delay spread over non-linearly filtered QAM.
In the initial studies, the decision regions for most of the signal constellations were straight boundaries (including some special cases of QAM constellations), and exact and simple BER expressions were obtained. However, more compact 2D signal constellations such as family of QAM do not have boundaries with perpendicular angles. For such constellations exact BER expressions are not simple and consists of special functions during the integral calculation. Considering this point in mind, Craig [168] presented simple and exact BER expressions with polygonal decision boundaries for such more practical signal constellations. This approach was also considered in [169]. In [170], authors examined the performance of 16-QAM by using sub-band coded speech and BCH error correcting codes. In [100], authors presented the differential encoding technique and circular star QAM constellation mapping to improve the BER performance. In [36], author discussed about the application of variable rate square and star QAM constellations for microcellular network based mobile radio applications and the hardware development of a QAM modem. In [103], authors examined the BER performance of various 16-ary QAM constellations and observed that 16-SQAM outperforms 16star QAM. In [171], author presented the details of best suitable QAM constellations for micro-cellular and macrocellular personal communication systems. In [172], authors proposed a post-detection maximum ratio combining (MRC) for space diversity in digital land mobile communications by considering M-ary QAM constellations over Rayleigh fading channels. In [173], authors examined the performance of M-ary QAM constellations by considering pilot symbol aided scheme to compensate Rayleigh fading for land mobile communications. In [174], authors proposed a trellis code based 32-QAM constellation to improve the BER performance of the system for fading channels. In [175], authors analyzed the performance of DS/CDMA system over multipath fading channel with RAKE receiver by employing non-coherent M-ary orthogonal modulation. In [176], [177], considering the amplitude and phase errors, performance of various coherently modulated non-constant envelope like Mary RQAM constellations were studied over the AWGN and various fading channels. In [5], authors proposed an SQAM constellation for land mobile communications to achieve high data-rate with high QoS for flat and selective fading channels. In [6], authors investigated the performance of adaptive QAM constellation in the presence of co-channel interference. In [9], authors proposed a system to control both modulation level and SER according to channel conditions and monitor the channel condition based on the delay profile measurement. In [178], authors optimized the switching levels for BPSK, QPSK, 16-QAM, and 64-QAM based on received signal strength. In [179], a method of superimposing the closet codes with adaptive M-ary QAM over fading channels was proposed. In [180], authors investigated the performance of L-branch diversity reception system and the SER expressions of M-ary QAM over Rayleigh fading and AWGN channels were derived. In [181], analytical exact ASER expressions for different constellations over various fading channels were derived. In [101], Dong et al. extended the work done by Craig, and performance of various coherent 8 and 16-ary signal constellations was analyzed for both the AWGN and fading channels. In [182], authors employed non-coherent orthogonal M-ary modulation with RAKE receiver for CDMA over fast fading and combined fast fading, shadowing and power control channels. Further, authors determine the BER of the adaptive signaling constellations from 2 2 to 2 9 symbols. In the [183], authors mitigated the effects of wideband multipath Rayleigh fading channel with a decision feedback equalizer for adaptive modulation and obtained the BER and bits per symbol performance of BPSK, QPSK, 16-QAM, and 64-QAM constellations. In [184], authors proposed adaptive modulation schemes for simultaneous transmission of voice and data transmission over Nakagami-m fading channels. In [185], authors illustrated the impact of power control over adaptive modulation schemes in presence of interference in a multi-user environment. In [186], authors investigated the performance of adaptive modulation over slow fading channels by considering co-channel interference. In [187], authors analyzed the performance of adaptive M-ary QAM techniques by considering channel estimation errors over Rayleigh fading channel. In [188], authors proposed the recursive algorithm for M-ary QAM constellations to obtain the accurate BER over AWGN channels. In [189], authors introduced the adaptive modulation over Nakagami-m multipath fading channels and determined the average spectral efficiency for 2L-dimensional trellis codes. In [190], authors introduced adaptive bit-interleaved coded modulation to combat the estimation errors. In [191], authors analyzed cellular network scenario by considering adaptive antennas based on fixed and channel allocation schemes. In [192], authors determined the adaptive modulation schemes under constraints of datarate, transmit power, and instantaneous BER to maximize the spectral efficiency. In [193], authors presented the generalized expression for 1D and 2D amplitude modulations schemes encoded with gray mapping. In [194], authors derived the generalized log likelihood ratio for soft decision channel decoding for QAM constellations. In [195], authors analyzed the performance of multi-rate CDMA system and determined the optimal adaptive rate and power allocation strategies to maximize the throughput of the system. In [196], authors proposed the optimal modulation-mode switching levels for the various PSK and QAM constellations and their BER performance were analyzed. In [197], authors analyzed the performance of a turbo coded system and optimize the adaptive turbo codes and transmitter power. Further, BER performance under average power constraint was obtained. In [198], authors analyzed the performance of a cellular system with different link adaptation strategies by considering practical constraints and proposed a hybrid link to overcome losses due to the practical constraints. In [199], authors optimized the adaptive modulation schemes assisted with channel prediction of Rayleigh fading channels for uncoded M-ary QAM. In [200], authors proposed a soft decision equalization techniques with low complexity and better error performance for MIMO frequency selective channels and BER of QAM constellations was evaluated. In [201], authors analyzed the impact of CSI prediction employing the linear fading predictor over trellis coded Rayleigh fading channel with adaptive modulation and closed-form expressions of BER and spectral efficiency were obtained. In [202], authors derived the BER expressions of M-ary QAM with pilot symbol assisted channel estimation of static and Rayleigh fading channels by considering the impact of noise and the estimator de-correlation. In [203], authors analyzed the effect of linear MMSE channel estimation and channel prediction errors on BER and developed the adaptive pilot symbol assisted modulation schemes to maximize the spectral efficiency. In [204], authors developed a framework for single user throughput maximization by optimizing the symbol rate, packet length, and M-QAM constellation size over AWGN and Rayleigh fading channels. In [205], authors proposed an optimum power allocation policy for the discrete signaling constellations limited in PAPR to maximize the mutual information with arbitrary input distributions. In [206], authors analyzed the bit-interleaved coded OFDM packet transmission with adaptive modulation and determined the pairwise error probability over slow fading channels. In [207], authors investigated the performance of a slow adaptive modulation scheme employing M-ary QAM constellations with antenna diversity. In [208], authors analyzed the impact of constellation size of an M-ary QAM constellations, and the impact of trellis coded modulation on energy efficiency of wireless network through game theoretic approach was analyzed. Further, a number of research works on the BER/SER performance of various QAM constellations in wireless communication systems, operating under different fading environments have been addressed in [38], [209]- [213].

B. CABLE, SATELLITE, AND MICROWAVE COMMUNICATIONS
In a limited frequency band, the high data transmission through satellite leads towards the use of highly spectral efficient modulation schemes in satellite communication. Further, digital microwave radio system is an efficient way of transmission when digitizing a communication system. In designing a microwave radio system, RF spectrum must be used in an efficient way, which can be done through adaptive modulation scheme. Hence, in this subsection, a survey on various modulation schemes used in satellite and microwave radio communications is presented.
For the very first time, Thomas et al. [29] mentioned QAM application in satellite communication and considered the non-linear distortion effect due to the use of traveling wave tube amplifier in transponder. In [214], authors implemented multi-level QAM constellations using PSK modems for realizing high baud-rate multi-level transmission for high speed digital-microwave, millimeter-wave, and satellite communication system. In [215], authors proposed a long-haul 16-QAM digital radio system with 200 Mbps in 40 MHz bandwidth as an application for satellite and microwave links. In [216], Prabhu evaluated the performance of 16-QAM in presence of co-channel interference over 8-PSK and 16-PSK by developing a theory in evaluating the error probabilities as an application for terrestrial and satellite communications. In [217], from spectral efficiency point of view, 16-SQAM constellation was adopted for digital microwave radio system operating in 4-5 GHz band. For satellite communication, a turning point came in 1982, when K. Feher proposed a new method of SQAM generation with the use of nonlinear amplifiers which was termed as NLA-QAM [32], where 16 and 64-ary NLA-QAM constellations were proposed. In [218], authors presented the work on digital communications in satellite communications. In [219], author proposed the equalization techniques for digital terrestrial radio systems employing QAM techniques by tap adaptation. In [220] author addressed FEC coded 1024-QAM modems, staggered 1024-QAM, and 256-QAM modems for microwave and cable system application for the first time, where practical constraints of phase noise, amplitude distortions, and group delays were considered. In [221], authors considered 64-256 QAM constellations to calculate the system parameters for high speed data transmission to deploy 1.54 Mbps modems in microwave radio. In [222], authors analyzed the satellite channels by considering the weather induced impairments such as scintillation and rain attenuation, and developed predictors using autoregressive models to improve the efficiency along with the adaptive modulation. In [223], authors investigated the performance of non-linear precoding to mitigate the interference in the broadband MIMO satellite communications and performed BER of uncoded BPSK and QAM constellations. Further, in [45], [49], [63]- [67], QAM constellations were standardized for digital signal broadcasting over cable and satellite links.

C. SPATIAL DIVERSITY
Multipath fading is one of the prominent feature of the wireless communication channel which severely degrades the reliability, robustness, and coverage of the system. To combat this, spatial diversity is one of the solutions which improves the performance of a wireless communication system. Spatial diversity can be achieved through multi-carrier system by deploying multiple antennas at the communication nodes. A communication system with multiple antennas at the transmitter and receiver is referred to as MIMO system. In MIMO systems, multiple paths between the source and destination are established with the help of multiple antennas to improve diversity of the communication system. However, multiple antennas at each node are not always possible due to hardware limitations and device size (such as in mobile handsets and sensor networks). In such cases, cooperative communication is preferred as an alternative to improve the spatial diversity of the communication system. Next, we will study the performance of such constellations. Multicarrier systems are explored by using OFDM and MIMO systems. A detailed study of OFDM systems is given in [224]- [226]. Various applications employing OFDM techniques are listed below: a) OFDM: In [227], author proposed DFT based orthogonally multiplexed QAM system for multichannel system. In [144], authors investigate the multi-carrier system and the application of higher-order QAM constellations. In [228], authors realized the 16-QAM from scaled 4-PSK signals to reduce the peak-to-average envelope power ratio employing Golay sequences for OFDM systems. In [206], authors analyzed the bit-interleaved coded OFDM packet transmission with adaptive modulation and determined the pairwise error probability of the system over slow fading channels. In [229], authors developed loading algorithms to minimize the transmit power of OFDM system with rate and error probability constraints under three different CSI scenarios for QAM constellations. In [230], authors presented the Golay code based 16-star QAM constellation constructed from sum of two scaled QPSK to reduce the peak-to-mean envelope power for OFDM systems. In [231], authors investigated OFDM system with adaptive transmission for maximizing energy efficiency with transmit power and its allocation constraint. b) NOMA: The fundamental concept of NOMA is to support multiple users through superposition coding at the transmitter and multi-user detection algorithms such as successive interference cancellation (SIC) techniques are employed at the receiver for decoding the desired signal [232], [233]. In general, NOMA is a multiple access technique which enables multiple users to access the same bandwidth for information transfer. In power domain NOMA (PD-NOMA), multiple users information is superimposed over the same frequency/time resources at different power levels. The power distribution among the superimposed signals is assigned in such a way that the weak user with poor channel conditions is allocated with more power than the strong user with the better channel conditions. Thus, in NOMA, user fairness is guaranteed through trade-off among the users throughput. Even though user fairness is maintained in NOMA, the quality of service insights for the weak and strong users can be drawn from the error rate analysis [234]. In NOMA, in order to superimpose two modulated symbols, their power allocations constraints should be met in order to decode them and these power allocation coefficients are dependent on the constellation size. In [234]- [237], authors have investigated the average symbol error rate of QAM schemes in the NOMA system. In [236], authors investigated the down-link PD-NOMA and presented the power allocation criterion for symbol level detection of practicable QAM constellation. The power allocation criterion shown in [236] can be employed for HQAM analysis in NOMA system. The detection complexity at the receiver increases with increase in number of users and the constellation size [234], [236]. c) MIMO: In [238], authors investigated the performance of M-ary QAM techniques by considering the L-fold antenna diversity over Nakagami-m fading channels with MRC receiver for independent and correlated channels and equal gain combining receiver for independent channels. In [239], authors examined the performance of MIMO system with adaptive modulation and adaptive array processing at the receiver in interference limited cellular systems. In [240], authors developed a framework for mean spectral efficiency by considering mean SNR at the cell boundary, propagation exponent, coding technique and the number of antennas. Further, authors highlighted the potential benefits of adaptive modulation with multiple transmissions in terms of spectral efficiency. In [241], authors illustrated the impact of partial CSI over adaptive MIMO OFDM QAM mapped symbols with 2D space-time coder beamformer. In [242], authors investigated the impact of imperfect CSI and outdated CSI over a multi-antenna system with adaptive modulation. In [199], authors optimized the adaptive modulation schemes assisted with channel prediction of Rayleigh fading channels for uncoded M-ary QAM. In [243], authors presented a pilot symbol assisted modulation channel predictor for MIMO Rayleigh fading channels, the impact of prediction errors on BER performance was analyzed. In [242], authors investigated the impact of imperfect and outdated CSI on a multi-antenna system based on adaptive modulation. In [244], authors examined the multi-user multi-antenna scenario and emphasized the advantages of non-linear pre-equalization, and SER analysis was performed for QAM constellations. In [245], authors investigated the impact of both the perfect and imperfect CSI at both the transmitter and receiver under average transmit power and instantaneous constraint for MIMO systems. In [246], authors proposed adaptive resource allocation based on subcarrier allocation, power, and bit distribution for multi-user MIMO/OFDM systems based on channel conditions. In [247], authors optimized the performance of a multi-antenna system with bandwidth limited feedback link based on adaptive modulation and transmit beamforming. In [248], authors analyzed a point-to-point MIMO system with perfect CSI at transmitter and receiver, and optimized the constellation selection and linear transceiver based on BER. In [249], authors investigated the multicode CDMA systems with interference cancellers to support high data-rate communications and BER of M-ary QAM was obtained. In [250], authors analyzed the performance of rate-adaptive M-ary QAM constellations with orthogonal space time block codes (STBC) and derive the closed-form BER expressions. In [251], authors investigated cross-layer approach for transmit antenna selection to maximize the throughput of the system and also adaptive modulation to improve systems performance. In [252], authors investigated half-duplex non-orthogonal AF MIMO multi-relay network with STBC and performed diversity multiplexing trade-off and frame-error-rate analysis employing QAM constellations. In [253], authors developed a framework for opportunistic encryption to maximize the throughput with desired security constraints based on channel characteristics. Further, authors optimized the encryption, modulation, and employed FEC codes to combat bit errors. In [254], authors investigated the performance of a half-duplex cooperative MIMO system and asymptotic closed-form pairwise error probability expression was derived. In [255], authors investigated the performance of space time codes based MIMO system over Rayleigh fading channels with adaptive modulation by considering imperfections in channel estimation. In [256], authors investigated QAM mapping based multiuser MIMO OFDM system and optimized transmit power with user rate constraint. In [257], authors investigated multiuser MIMO relay cellular system with adaptive modulation and derived the upper and lower bounds for achievable sum rate. In [258], authors introduced space shift keying (SSK) employing QAM mapping for MIMO systems, and analyzed the system performance under spatial correlation and estimation errors. In [259], authors investigated the MIMO half-duplex AF one and two-way relay systems and proposed algorithms to optimize linear transceivers at both source and relay nodes with sum-rate and mean-square error constraints, and determined the BER of QAM constellations. In [260], authors presented a VLSI architecture for MIMO decoder which supports 4 × 4 antenna configuration with 64-QAM based on depth-first and breadth-first approach and proposed an algorithm which reduces complexity for both the soft and hard decoding. In [261], authors proposed a hybrid algorithm combining technique using reactive tabu search algorithm and belief propagation algorithm to improve the performance of large MIMO detection with higher order QAM constellations. In [262], authors proposed a near optimum detection algorithm for coherently detected space time shift keying techniques with L-point PSK and QAM constellations. In [242], [263]- [288], authors presented the works considering multiple antennas in various system models including multi-node communication and performed SER analysis of QAM constellations. In [289], authors presented a unified asymptotic framework for transmit antenna selection (TAS)-MIMO multi-relay network with various fading channels and derived the closed-form expressions of the outage probability and SER of M-ary PSK and M-ary QAM constellations. In [290], authors proposed the TAS algorithms and obtained the BER performance of the QAM constellation. In [291], authors presented an analysis of OFDM with offset QAM. In [241], authors presented the impact of partial CSI over adaptive MIMO OFDM QAM mapped symbols with 2D space-time coder-beamformer. In [246], authors proposed the adaptive resource allocation based on subcarrier allocation, power and bit distribution for multiuser MIMO/OFDM systems based on channel conditions. In [256], authors investigated QAM mapping based multiuser MIMO OFDM system and optimized transmit power with user rate constraint. In [292], authors developed a framework for OFDM based spatial modulation over Rayleigh fading channels and derived the SER expressions for QAM constellations. In [293], authors analyzed an OFDMA based cellular downlink by considering non-Gaussian intercell interference to improve the transmission rates of cell edge users, and proposed new modulation scheme frequency and QAM by combining FSK and QAM constellations. In [294], authors analyzed the performance of a MIMO OFDM system with adaptive modulation based on machine learning as an application to 5G new radio systems. In [295], authors introduced STBC using multiple transmit and receiver antennas. Further, the authors determined the maximum diversity order attained and also maximum achievable code rate for complex constellations such as QAM and PAM. In [296], authors investigated the STBC diversity systems over Rayleigh fading channels and SEP of RQAM constellations was obtained. In [297], authors analyzed the performance of half-duplex two-way AF relaying system by considering the Alamouti coding and derived the lower and upper bounds for average sum-rate and pairwise error probability by considering QAM constellations. In [298], authors investigated a MIMO system and proposed a virtual antipodal detection scheme for gray coded higher order RQAM based on semidefinite programming relaxation. In [299]- [302], authors analyzed STBC systems by considering QAM constellations. In [303]- [305], authors proposed semi-definite relaxation (SDR) approach for MIMO detection with higher order QAM constellations. In [306], for M-ary QAM constellations based systems, a semi-definite programming relaxation (SDPR) approach is proposed to combat the multiuser detection problems. In [307], authors proposed SDR based quasi-maximum likelihood algorithm for MIMO systems and introduced several relaxation models with increasing complexity and performed the SER analysis for QAM and PSK constellations. In [308], authors investigated the performance of three different SDR such as polynomial-inspired SDR, bound-constrained SDR, and virtually antipodal SDR over MIMO channels for BPSK, QPSK, and higher order QAM constellations.

2) COOPERATIVE RELAYING
Cooperative relaying is one of the prominent techniques to achieve spatial diversity in wireless communication.
Cooperative relaying is extensively explored because of its capability to provide improved capacity, coverage, and power and spectral efficiency due to the diversity gain without using multiple antennas at different communication nodes. Hence, it is also considered as a virtual MIMO system. A detailed study of cooperative relaying can be seen in [309], [310]. In this subsection, we have explored various works on the SER or BER performance of different cooperative relaying networks in various fading environments.
In [311], authors investigated single and multi-channel communication, and presented the BER analysis for M-ary constellations over Nakagami-m and Rayleigh fading channels. In [312], authors investigated the SER of M-ary PSK and QAM constellations over Nakagami-m fading channels for hybrid selection/MRC diversity. In [202], authors investigated the effect of estimator de-correlation on the BER performance of M-ary QAM constellations over Rayleigh fading channels. In [313], authors investigated the performance of cooperative relay system for three different time division multiple access transmission protocols and determined the SER of 4-QAM over fading channels. In [314], authors derived the average error rates of various M-ary constellations and outage probability using characteristic function based approach over various fading channels. In [315], authors analyzed cooperative diversity by considering arbitrary cooperative branches with arbitrary hops per branch over various fading channels and derived the SEP for various constellations including QAM. In [316], authors derived the closed-form expressions of QPSK and QAM for a cooperative systems and optimized the power allocation. In [317], author analyzed the AF cooperative multi-relay system employing selection combining and obtained the SER analysis of QAM constellations. In [263], authors investigated multi-node cooperative relay system and derived the SER expressions for M-PSK and M-QAM constellations and optimized the power allocation. In [318], authors investigated the performance of multi-hop cellular network and proposed the routing and power allocation algorithms to maximize the throughput and BER is evaluated for QAM constellations. In [319], authors analyzed the impact of channel estimation error on the BER performance of RQAM and SQAM constellations. In [320], authors analyzed the performance of cooperative multi-relay system over i.n.i.d. Nakagami-m fading channels and derived the outage probability and error rate employing QAM constellations. In [321], authors investigated the performance of cooperative relay system with DF and AF relaying protocols and analyzed the SER for M-ary QAM and PSK constellations with optimum power allocation. In [322], authors investigated the selection detection and forward and AF half-duplex OFDM relay systems and proposed margin adaptive bit and power loading techniques to minimize the transmit-power consumption, and determined the SER of QAM constellations. In [323], authors investigated the performance of clustered relay network based on STBC and evaluated the packet error rate employing BPSK and 16-QAM constellations. In [324], authors investigated the single and multi-channel diversity reception over i.n.i.d. Nakagami-m fading channels and average SEP expression of RQAM was derived. In [267], authors investigated the SER performance of multi-relay DF system over Nakagami-m fading channels for QAM constellations and optimized the power allocation. In [325], author analyzed DF based single and multi-relay systems and proposed maximum likelihood and piece-wise linear decoders for complex-valued unitary and non-unitary transmissions, and performed the SER analysis of M-ary QAM and PSK constellations. In [326], authors analyzed the performance of OFDM based cooperative relay system and optimized the BER of QAM techniques with respect data rate based on optimal power, bit, and joint power and bit loading. In [327], authors investigated the DF cooperative system and derived the maximum likelihood decoder for M-ary PSK, PAM, and QAM constellations. Apart from the works presented above and reference therein, a lot of work is presented in the literature on the BER/SER performance of various constellations for various relay networks over different fading channels till now.

D. EMERGING AREAS FOR HIGH DATA-RATE COMMUNICATIONS 1) OPTICAL WIRELESS COMMUNICATIONS
Optical wireless communications (OWC) operates in 350 − 1550 nm band, through visible light (VL), infra-red, and UVC bands; has gained significant research attention for the future wireless broadband access. OWC is affordable with high data-rates (30 Gbps) and virtually infinite bandwidth [328], [329]. Various OWC technologies are VLC, FSO communication, optical camera communication, light fidelity, and light detection and ranging [330]. OWC operating in VL band (390 − 750 nm) is known as VLC. FSO is also known as terrestrial point-to-point OWC and operates near infra-red frequencies (750 − −1600 nm) [331], [332]. UVC is an non-line-of-sight (NLoS) communications operates in ultraviolet band (200 − −280 nm). OWC is employed for both indoor and outdoor communications. OWC also finds its potential application in underwater communications [331], [333].
In VLC, achievable high data-rates are limited by the low modulation bandwidth of the GaN-based white light emitting diodes (LEDs) due to the slow response time. Bandwidth can be increased by using blue LEDs at the cost of intensity loss of 10 dB [334]. Spectral efficiency can be increased by employing MIMO, massive MIMO, and higher order modulation schemes like optical-OFDM (O-OFDM) [335]. However, MIMO channel are limited by the receiver position, making channel matrix either ill conditioned or rank-deficit which makes data recovery almost impossible. O-OFDM limited by Hermitian symmetry constraints, cyclic prefix, PAPR, and dynamic range of LEDs. As an alternative to O-OFDM, carrier-less amplitude and phase modulation has gained significant interest in optical communications for high data-rate transmissions [336]. Bandpass carrier-less amplitude and phase (CAP) modulation is a QAM modulation where two orthogonal finite impulse response digital filters are employed to achieve it. Various seminal works using CAP modulation schemes to improve the spectral efficiency of the band-limited LEDs are presented in [337]- [354].
In underwater communications, OWC provides very high data-rates and bandwidth when compared with acoustic, RF, and optical wave communications. In underwater OWC, spectral efficiency is increased by employing coherent modulation, where information is transmitted through amplitude, phase, or polarization of the optical field [355].
Coherent modulation includes M-ary PSK, M-ary QAM, and multilevel polarization shift keying. In underwater OWC, coherent modulation schemes are very attractive due to high receiver sensitivity, spectral efficiency, and robust to background noise and interfering signals with increased cost and implementation complexity [333], [355]. Some of the notable works which presented the advantages of QAM schemes over intensity-modulation direct-detection systems are [333], [356]- [366].

2) MACHINE LEARNING-5G, SECURITY
In [387], authors addressed the application of adaptive modulation schemes for 3G wireless communication systems and developed method to select the modulation and coding based on the channel conditions with minimum frame error rate constraint. In [253], authors developed a framework for the opportunistic encryption to maximize the throughput with the desired security constraints based on channel characteristics. Further, authors optimized the encryption, modulation and employ FEC codes to combat bit errors. In [213], [388]- [392], authors investigated the SER performance of different QAM constellations over η − μ, κ-μ, and two wave with diffuse power fading channels. In [393], author presented a survey on approximation of Q function and its significance in error probability evaluation. Further, author presented the closed-form SEP of various constellations such as QPSK, QAM, HQAM, and XQAM with approximated Q-functions. In [394]- [396], authors presented a survey on 5G new radio and also discussed the application of QAM constellations. In [397]- [403], authors presented the work based on filter bank multi-carrier systems using QAM constellations as an application for future wireless communications. In [292], authors developed a framework for MIMO systems with spatial modulation schemes over generalized fading channels and computed the average BEP for PSK and QAM constellations. In [404], authors analyzed the performance of MIMO system with SSK and generalized SSK modulation by considering multiple-access interference for PSK and QAM constellations. In [405], authors presented a state-of-theart survey on spatial modulation based MIMO system with QAM for the emerging and future wireless communications and discussed the advantages, research challenges, and experimental activities. In [45], authors investigated the performance of star constellation based spatial modulation. In [406], authors investigated the energy harvesting based distributed spatial modulation and evaluated the error probability of energy recycling DSM with QAM constellations. In [407], authors presented a survey on spatial modulation schemes and the application of spatially modulated QAM is presented. In [408]- [410], authors investigated spatial modulation techniques employing QAM.

3) EDGE CACHING, ENERGY HARVESTING, FULL-DUPLEX COMMUNICATIONS, GAME THEORY
In [411], authors developed a framework to optimize the modulation scheme to be employed by reducing the energy consumption over energy constraint nodes for both coded and uncoded systems. In [412], authors investigated sensor network and proposed variable length TDMA schemes to minimize energy consumption for transmission of QAM constellations by optimizing routing, scheduling, and link adaptation strategies. In [208], authors analyzed the impact of constellation size of M-ary QAM constellations and also the impact of trellis coded modulation on energy efficiency of wireless network through game theoretic approach. In [323], authors analyzed the STBC based clustered wireless networks using cooperative communications and performed packet error rate analysis for QAM constellations to minimize the energy consumption with transmit power allocation, number of sensors, and distance between clusters as constraints. In [413]- [416], authors investigated the wireless sensor networks and presented various techniques for minimizing the energy consumption by considering QAM transmissions. In [417], authors investigated the performance of full-duplex AF relay system under residual self interference and derived the pairwise error probability for uncoded systems and BER for coded systems. In [418], authors presented a survey on solutions provided for improving energy efficiency of multimedia based battery operated hand-held devices by considering various modulation schemes for transmission including QAM constellations. In [419], authors investigated the performance of spatial modulation based full-duplex two-way relay systems and derived the tight upper bound for average BEP and asymptotic BEP for QAM constellations. In [419]- [423], authors analyzed full-duplex cooperative relay system by considering spatial modulation based QAM. In [424], authors investigated the performance of full-duplex relay system and compared the performance of media based modulation schemes with QAM.

VIII. CONCLUSION AND FUTURE DIRECTIONS
As bandwidth and power efficiency are prominent constraints for 5G and beyond wireless communication systems, a detailed study of various QAM constellations has been presented in this work. This study started with an introduction to QAM constellations used since early 1960s to more complex higher order power and bandwidth efficient QAM constellations with their applications in various wireless communication systems, ITU, 3GPP, and IEEE standards. Constellation modeling, bit mapping, Gray code penalty, decision regions, peak and average powers, and PAPR of the advance XQAM and HQAM constellations have also been discussed in details. Study of HQAM includes regular and irregular HQAM constellations with various constellation orders. Finally, a comparison of various QAM constellations has been presented which concludes the irregular HQAM as the optimum constellation which is highly power efficient than the other constellations employed in the existing wireless communication systems and standards. Till now, the HQAM constellations have not been considered in the existing wireless communication systems and standards, however, from this study it can be claimed that the HQAM can be adopted in various wireless applications targeted for beyond 5G or 6G wireless communication systems where high data-rates with good QoS is required within the limited power and bandwidth. This also opens up the possibilities for other 2D or higher dimensional constellations to be researched. The constellations discussed in this work focus on the geometric constellation shaping. The performance can be improved further by probabilistic amplitude shaping of the HQAM constellations in the near future. Also, it will be interesting to see the applications of the various higher order QAM constellations in the emerging and next generation high data-rate communication systems like RF/FSO based terrestrial-satellite communication systems, high altitude platform (HAP) communication, NOMA, IRS, and IAB.

APPENDIX
Various regular HQAM and irregular HQAM (optimum) constellations are shown in the Fig. 19 and Fig. 20, respectively.
For various irregular HQAM constellations (M varies from 4 to 1024), decision regions are shown in the Fig. 21.