Practical Optimization and Game Theory for 6G Ultra-Dense Networks: Overview and Research Challenges

Ultra-dense networks (UDNs) have been employed to solve the pressing problems in relation to the increasing demand for higher coverage and capacity of the fifth generation (5G) wireless networks. The deployment of UDNs in a very large scale has been envisioned to break the fundamental deadlocks of beyond 5G or the sixth generation (6G) networks and deliver many more orders of magnitude gains that today’s technologies achieve. However, the mathematical tool to optimize the system performance under the stringent radio resource constraints is widely recognized to be a formidable challenge. System-level performance optimization of current UDNs are usually conducted by relying on numerical simulations, which are often time-consuming and have become extremely difficult in the context of 6G with extremely high density. As such, there is an urgent need for developing a realistic mathematical model for optimizing the 6G UDNs. In this paper, we introduce challenges as well as issues that have to be thoroughly considered while deploying UDNs in realistic environment. We revisit efficient mathematical techniques including game theory and real-time optimization in the context of optimizing UDNs performance. In addition, emerging technologies which are suitable to apply in UDNs are also discussed. Some of them have already been used in UDNs with high efficiency while the others which are still under investigation are expected to boost the performance of UDNs to achieve the requirements of 6G. Importantly, for the first time, we introduce the joint optimal approach between realtime optimization and game theory (ROG) which is an effective tool to solve the optimization problems of large-scale UDNs with low complexity. Then, we describe two approaches for using ROG in UDNs. Finally, some case study of ROG are given to illustrate how to apply ROG for solving the problems of different applications in UDNs.


I. INTRODUCTION
Since 2020, the fifth generation (5G) networks have begun rolling out in many countries [1]. However, it is predicted that 5G may have not enough capability to be applied in services with the requirements of data rate to achieve terabits per second, latency to be less than hundreds of microseconds, and connectivity to be more than tens of million connections per km 2 in the near future [2], [3]. Therefore, the sixth generation (6G) networks have recently attracted both the industry and academia in some countries such as Finland, the United Kingdom, Germany, the United States, etc [4]. Moreover, the vision of 6G networks is towards ubiquitous 3D coverage (space, aerial and underwater environments) [4], intelligent networks (apply or support artificial intelligence technologies) [4], [5], flexible and reliable networks (movable property or quick deployment for emergencies) [6], green networks (efficient energy for the sustainable development of wireless communications) [7].
Internet-of-Thing (IoT) is one of the most prevalent technologies in 5G and has so much potential in 6G. In IoT systems, sensors, vehicles, and devices, which are connected via Internet links, form many services to adapt human activities, such as smart city, smart home, automation, environmental monitoring, healthcare, or even remote laboratories, automated digital contact tracing in the period of Coronavirus Disease 2019 (COVID- 19) [8]- [11]. According to the Ericsson Mobility Report, the number of IoT connections has explosively grown and is forecast to reach 26.4 billion in 2026 in which Massive IoT technologies will make up 46 percent [12]. Massive IoT deployments are made up of hundreds to millions of connected IoT devices with low throughput, very low power, and low latency. Obviously, the traditional cellular networks have not enough capacity to connect to all devices in massive IoT scenarios. With the development of cellular network technology, using ultradense networks (UDNs) is a promising solution for this issue. In fact, UDNs, which can be seen as a key technology for 5G, can serve simultaneously the vast number of user equipments (UEs) with high density, increase network capacity, improve the coverage and quality of service (QoS) with smooth handoff and low latency [13], [14]. The fundamental idea of UDNs is that the base stations (BSs) are deployed to be as close as possible to UEs to enhance the quality of transmitted signal and increase the efficient utilization of the limited spectrum. In addition, decreasing the distance between transmitters and receivers helps the networks to be easily integrated with extremely high frequency (EHF) technology which improves considerably the capacity. In terms of UE connections, UDNs are divided into two tiers (a macrocell tier and a small cell tier) so that UDNs are also named multitiered heterogeneous networks (HetNets). In each macrocell, a macro base station (MBS) characterized by its high power consumption transmits the signal to UEs in an area with a coverage radius to be up to several kilometers. Macrocells guarantee the minimum throughput to serve UEs. In small cells, low-cost small base stations (SBSs) using low power are deployed with high densities in the coverage of macrocell depending on the density of randomly distributed UEs. The phrase "ultra-dense" in UDNs means that the number of small cells is extremely larger than the number of active users and the number of cells in the traditional networks. The deployment of the massive small cells is a breakthrough of UDNs compared to the traditional cellular networks.
To evaluate the performance of wireless networks, optimization algorithms can be deployed. Optimization can be seen as a bridge between mathematics and engineering because it uses mathematical researches for solving realistic problems in engineering. From an optimization point of view, the technical issues are handled easily and efficiently by constructing the optimization problems (OPs) and finding the methods to solve them. Especially when UDN scenarios are considered, optimization is widely used in resource allocation, interference management, network deployment, backhauling, congestion management [15]. In addition, various objectives of problems in UDNs also need to optimize such as energy efficiency (EE) maximization, spectral efficiency (SE) maximization, system capacity maximization, interference minimization, power consumption minimization, weighted sum rate maximization [16]. However, using optimization in realistic applications in UDNs witnesses many challenges such as creating a suitable model, solving method, the complexity of the solution, realtime computation. Realtime optimization in practical scenarios requires that the processing time for finding the optimal solution has to be lower than a given time-bound. In UDNs with massive data, the large amount of BSs, UEs, multiple tiers, designing a realtime optimization method for an OP is much more challenging.
The traditional iterative algorithms for solving OPs are inefficient due to loaded complexity when they are applied in large-scale systems. Game theory (GT) as a promising solution is a distributed optimal framework to apply efficiently in UDNs which has complex interactions between network elements. Nowadays, the optimization methods imitated the natural behaviors of animals such as Particle Swarm Optimization (PSO), Genetic Algorithms (GA), Whale Optimization Algorithm (WOA), are widely applicable [17]. GT is also inspired by human games with competition or cooperation in the relationship of players. All players in a game want to maximize their interest illustrated as a utility function by choosing the good strategies. In contrast to traditional optimization methods which concentrate to achieve only one objective function, the solution of GT satisfies all rational and selfish players or no one has any incentive to change its own strategy. Depending on different criteria, GT has many types: Non-cooperative game (NCG) and cooperative game (CG) (conflict), static game and dynamic game (time), complete information game and incomplete information game (information), perfect information game and imperfect information game (history) [18]. In UDNs as a multi-tier, multi-cell, and multi-device environment, GT is suitable to apply and control the elements.
The combination of practical optimization and GT for solving the problems in UDNs is a promising method. GT divides the original complex OP into multiple easy sub-OPs which can be solved by parallelism techniques. In addition, these sub-OPs might be convex OPs if the utility functions are thoroughly designed. Accordingly, many powerful programming tools in different programming languages can solve these convex OPs such as CVX in MATLAB, CVXPY in PYTHON, CVXR in R, JuliaOPT in JULIA [19]. The main contributions of this paper are summarized as follows • We discuss some potential technologies and solutions, which are open and efficient research directions for applying UDNs with high-quality network performance. • We develop an amalgamated GT and practical optimization method, relying on real models of UDNs in realtime contexts. This approach is proposed for a significant reduction of the computational complexity and processing time of large-scale UDN scenarios. The joint GT and realtime optimization algorithms can deal with large-scale problems in wireless networks by exploiting a learningbased method for utilizing the fundamental knowledge and advantages of GT and practical optimization . • We provide practical case studies of high-performance UDN deployments, that are applicable to be implemented using the proposed GT optimization techniques in some scenarios of UDNs. The rest of this paper is organized as follows. Firstly, we introduce challenges of the deployment of ultra-dense small cells in Section II. Then, promising technologies or solutions for UDNs are described in Section III. Next, we show the combination of optimization and GT in UDNs in Section IV. Some case studies and conclusions are described in Section V and Section VI, respectively.

II. CHALLENGES OF ULTRA DENSIFICATION FOR THE FUTURE NETWORKS
UDNs as a promising technology in 5G and 6G are used for achieving a higher density of served UEs, higher capacity, lower latency, improving coverage with low power BSs, and having efficient utilization of resources. Despite many advantages, at the same time, some challenges need to be carefully studied related to ultra densification of elements in networks.

A. INTERFERENCE MITIGATION
Interference and noise always exist in all realistic communication systems. In UDNs, interference becomes a big obstacle that has to be investigated [20]. Figure 1 illustrates the interference scenario in one macro cell of an UDN. Interference can be separated into two types: inter-tier interference (e.g. macro-to-small interference marked by I1, small-tomacro interference marked by I2) and intra-tier interference (e.g. intra-cell interference marked by I3, inter-cell interference marked by I4). There are three major reasons for the interference issue in UDNs. Firstly, when SBSs shared the same spectrum were located densely, at the same time, the overlapping areas between their coverage are expanded. This leads to an increase in the probability that UEs encounter extremely serious interference in these areas. Secondly, with the deployment of the low size of small cells, the interference level is intensified with line-of-sight (LoS) propagation from adjacent cells. The third reason is that the SBSs in UDNs are deployed randomly according to the stochastic distribution of UEs. Hence, interference also has a random property that is hard to manage.

B. MOBILITY/HANDOVER
In cellular networks, when a mobile UE moves between two cells, it releases the connection with the old cell and connects to the new one. Additionally, the network has to guarantee the continuity of data transmission and reception in this back-and-forth handover process. Because of the delay and the miss of connection, it is expected that the frequency of this process is as low as possible [21]. Unfortunately, the deployment of a bunch of small cells with dense densities in a small area causes unnecessary and frequent handover between SBSs and the mobile UEs. Consequently, UEs need I1 I2

SBS SUE
The desired signal Inter-tier interference Intra-tier interference FIGURE 1: Interference cases in an UDN. more energy consumption for the hand-off process, and the network performance is simultaneously declined by high latency, fail handover, and high computing complexity.

C. ENERGY CONSUMPTION
In [22], the amount of carbon dioxide (CO 2 ) footprint produced by mobile communications was predicted to witness growth from 86 to 235 million tonnes between 2007 and 2020. This causes many problems related to health and environment [23]. In addition, increasing the number of base stations and UEs in UDNs leads to extremely high energy consumption. Therefore, applying green networking into UDNs is essential. The metric which is usually used to optimize energy consumption in UDNs is EE. EE represents the efficiency level of power utilization to transmit the signal to receivers in networks. Several studies are investigated to optimize EE in different UDN scenarios [24]- [26]. However, EE is not sufficient since EE can be optimized when data rate increases faster than power consumption [27].

D. MULTI-HOP RELAY OPTIMIZATION
With wired backhauling, BSs use the total of assigned resources to serve UEs. Nevertheless, in the ultra densification scenario of small cells in UDNs, deploying wired backhauling to every single SBSs is not feasible. Therefore, the resource needs to be split up into two actions: the access from UEs to the network and the relay for transmitting the backhaul traffic from SBSs to the core network. There are two kinds of arrangement: fixed access and backhaul and integrated access backhaul (IAB) [28]. It is importantly noted that SBSs have to find the multi-hop relay links to transmit the wireless backhaul traffic to the given gateways due to the small coverage of small cells [29]. Thus, the routing algorithm for the optimal multi-hop relay links is also a challenge in UDNs. There are some investigated researches [30]- [33].

E. COOPERATIVE TRANSMISSION
In UDNs, there are many idle SBSs that do not serve any UE since the number of SBSs is much larger than the number of active UEs. In many cases, turning on and using these sleeping SBSs to serve simultaneously any nearby UE along VOLUME xxxx, xxxx with its authored SBS is essential to consider [34]. This cooperation supports to not only increase the desired signal power but also decline the number of handover processes with highspeed mobile UEs. On the other hand, when adjacent small cells vary greatly on the number of UEs, the combination for creating a new virtual small cell allocates fairly the load data in some small cells and mitigates the interference between them. However, more SBSs or virtual small cells generate more energy consumption and interference with other small cells. Therefore, cooperative transmission still is a challenge of the deployment of massive SBSs.

III. POTENTIAL TECHNOLOGIES AND SOLUTIONS FOR THE ENABLING OF UDNS
With the ultra-dense deployment of network elements, four main aspects that need to be considered are interference management, huge data and information exchange, complexity, energy consumption. In this section, we introduce the promising technologies as well as potential solutions to deal with the challenges and the requirements of large-scale UDNs.

A. CLUSTERING
In realistic networks, UDNs with very large numbers of BSs and UEs are deployed in many geographical areas with different objectives. This leads to difficulty for the core network to manage, control, and solve complicated problems. Cell clustering methods help to not only disperse the big scenario of UDNs to decrease the computational complexity but localize the different objectives. There are three categories of cell clustering methods: network-based (static), user-based (dynamic), and hybrid (semi-dynamic) ones [35]. Networkbased clustering methods divide cells into clusters according to given targets, and the clusters do not change over time. Instead of independence to use the same resource, the cells in a cluster are collaborative to serve UEs. The methods in [36], [37] create clusters to eliminate strong interference and save energy, respectively. In other words, with userbased clustering methods, the BSs in clusters are determined to dynamically adapt to the changes in the channel state information (CSI) of the network so that the elements of the clusters are updated over time. Examples for the userbased clustering methods in UDNs were found in [38]- [41] with the improvements of EE, interference mitigation, spectral efficiency. The network-based clustering scheme is less flexible with interference, and the user-based clustering one is more complex with system scheduling and exhaustive information exchange. The hybrid clustering methods consider the tradeoff of these two joint categories [35]. In [42], [43], measurement BS clusters (MBCs) are formed according to measurement information and CSI. Then, coordination BS clusters (CBCs) which are the subsets of the fixed MBCs are created by cooperative BSs. Furthermore, user clustering methods considered in [24], [35], [44] are used as the next stages after the cell clustering methods to minimize the intracluster interference.

B. RESOURCE ALLOCATION (RA)
In UDNs, many access points (APs) (MBSs, SBSs, relay nodes, radio remote heads (RRHs)) and many UEs share the limited available signaling resources. Also, one or more of the requirements about Energy Efficiency (EE), power consumption, interference, throughput, quality of service (QoS), fairness, priorities, and computational complexity may be considered as the criteria to evaluate the performance of an UDN [16]. Therefore, to optimize the network performance, efficient resource allocation methods are essential in UDNs with high computational complexity, significant signaling overhead, and flexible handovers. There are two kinds of manageable resources: fundamental resources (power, spectrum, time, and space) and comprehensive resources (available channels, backhaul/fronthaul capacity, computing capability, etc). In [45], Lin et al. proposed a three-stage sequential solution based on the joint clustering and resource block allocation in user-centric UDNs to maximize the sum rate. In [46], a RA method is developed to maximize the system EE in UDNs integrating NOMA and beamforming. To mitigate the interference in UDNs while guaranteeing the SINR requirement, a power control algorithm based on Mean Field Game was proposed by Zhang et al. in [47]. It improves both EE and spectrum efficiency (SE) compared to UDNs without traffic offloading.

C. MILLIMETER WAVE (MMWAVE)
In UDNs, the distances between transmitters and receivers become smaller than ever to improve the channel links and the densities of serving users. However, this leads to overwhelming interference in the networks. Therefore, using very high-frequency bands is necessary for the interfering signal to drop quickly according to the distance increase. Apart from inter-cell interference mitigation, using high frequency causes an increasing spectrum that is directly proportional to the capacity. In addition, the sub-3GHz spectrum became extremely crowded by a lot of available wireless communication systems. MmWave technology uses frequency bands between 30 and 300 GHz for the signal interchange so that the available spectrum resources are 200 times wider than ones of the conventional cellular networks [48]. In addition, the very small wavelengths of mmWave help to easily integrate multiple antenna arrays in both transmitters and receivers to achieve further channel gain by narrowing the beams [49]. MmWave band is essential to be used in UDNs to achieve high data rates, mitigate path loss and interference thanks to narrow pencil beams [49], [50]. In [51], authors appreciated that both EE and SE increase when applying mmWave technology in UDNs. Authors in [52] proposed a method to solve the joint problem considering time RA, user association, and mmWave beamforming. In [53], using mmWave communication improve the average service delay in a cellular network.

D. TERAHERTZ (THZ) COMMUNICATIONS
With the explosive growth of global wireless connections and global mobile data traffic, and the revolutionary applications such as virtual/augmented reality, Internet of Everything (IoE), autonomous driving, 5G networks are not enough capabilities of data rates and latency to adapt [54]. THz communication is one of the most promising technologies for 6G and beyond [55], [56]. THz systems, which transfer the information using THz band (0.1-10 THz), provide the applications with ultra-wide bandwidth, ultra-high data rates, and the computing power is expected to approach the processing power of the human brain [54]. Similar to mmWave, THz waves witness a significant decrease in the power in propagation. However, THz waves are robust to atmospheric effects [56], and the multi-band ultra-massive multiple-input multiple-output (UM-MIMO) using massive nano-antenna arrays helps it to overcome this distance problem [57]. In [58], Jornet et al. proposed a propagation model for THz-based electromagnetic wireless nanonetworks in nanotechnology applications which are communications between nano-machines. In [59], a framework using the joint of THz communications and massive antenna technology was proposed to design and analyze the performance of the hybrid beamforming architecture in a DenseTeraNet which mimics an UDN. The summary of THz-enabled ultra-fast small cell and 3D network modeling is provided in [60].

E. MASSIVE MULTIPLE-INPUT MULTIPLE-OUTPUT (MMIMO)
UDNs require that the densities of small cells have to be larger than these of UEs [61]. In reality, it is impossible to build commercial networks with the number of BSs to be greater than the number of UEs. Thus, using multiple access techniques is essential in UDNs. In addition, the requirements of boosting multiplexing and diversity gain are strict standards in 5G and beyond 5G (B5G). Therefore, mMIMO technology (also known as large-scale antennas or very large MIMO) which integrates massive low-power and cheap antenna arrays in BSs solves these problems. In the mMIMObased BS, the number of antennas is much larger than the number of active devices so that one BS can serve many UEs in the same resource without cross-talk interference between UEs [14]. The very high frequency in mmWave or THz technologies is essential to pack hundreds or thousands of antennas into a compact area. Therefore, mMIMO BSs using very high frequency have small sizes and are easily deployed in different locations. In addition, mMIMO systems also have other benefits: interference mitigation, simple signal processing, capacity increase, reduction of latency, and ultra-high reliabilities [14]. In [62], authors discussed the benefits and challenges of using mmWave mMIMO-based wireless backhaul in 5G UDNs. Furthermore, optimization methods were proposed to improve EE in strong interference environments in mMIMO-based UDNs.

F. INTELLIGENT REFLECTING SURFACE (IRS)
In wireless communications, physical phenomenons such as reflection, diffraction, and scattering are the opportunities for transmitting the signal to receivers even though they make the signal processing and computing more complex. The very high-frequency signal in mmWave or THz technologies which are potential to UDNs are easily blocked by even thin obstacles. To tackle this problem, IRS technology, which is used for reflecting the signal to receivers, becomes more attractive. An IRS known as passive beamforming is a planar array that combines a bunch of low-cost passive reflecting elements. Each element controlled by a smart controller has the capability of independently inducing a reflecting signal with a certain amplitude and a phase on an incident signal [63]. IRSs can be attached to surfaces that are in LoS with APs like walls, glasses, ceiling, etc., to create smart reflecting radio environments [56]. Therefore, IRS is a solution for significantly channel gain improvement, network coverage improvement, boosting spectral efficiency, low cost of implementation, lower power consumption, co-channel interference mitigation [56], [64]. In [63], [65], IRS is considered in one cell wireless system where a multi-antenna AP serves single-antenna users. In [66], Hashida et al. proposed an IRS-aided cellular network with cooperative IRSs and BSs to communicate with aerial users, and this network outperforms the conventional system without IRSs for intercell interference mitigation. In addition, IRS-enabled cellular networks are considered with the different objectives for maximizing the number of users, minimizing the transmit power, or maximizing the weighted minimum rate of all users in [67], [68].

G. CELL-FREE MMIMO
Cell-free mMIMO is a promising transmission technology where a large number of distributed APs connected to a centralized processor to serve UEs coherently. In cell-free mMIMO, each UE is served by surrounding APs which combine to form a virtual mMIMO BS, and one AP can join multiple virtual BSs to serve UEs [69]. Thus, there is no clear cell boundary between small cells like conventional cellular networks. Because of user-centric full cooperation among APs, the implementation of cell-free mMIMO makes UDNs more robust to inter-cell interference [70]. In [71], [72], the authors combined cell-free mMIMO and mmWave in UDN environments to evaluate the effect of shadowing correlation and radio frequency beamforming schemes. In [73], the authors proposed a CSI compression mechanism to avoid the upload overhead and improve SE in cell-free mMIMO UDNs.

H. UNMANNED AERIAL VEHICLES (UAVS)
UAVs known as drones are aircrafts that are remotely controlled by human operators or autonomous programs to perform some given tasks. In wireless communications, using UAVs as flying BSs in terrestrial cellular networks is a promising solution in 5G and B5G [74], [75]. The flying BSs VOLUME xxxx, xxxx can provide LoS connections, can be dynamically adjusted to suit the communication environment, boost spectral efficiency and user quality of experience, enhance the capacity and coverage of cellular networks with the characters of low cost and controllable mobility [76], [77]. In the UAV-assisted cellular networks, UAV-BSs can work alone or assist traditional fixed BSs by improving the coverage (UAV-mounted flying relays) or offloading an excessive load of data exchange [74]. To overcome the unusual traffic challenge of unexpected events, a resilient UDN using UAVs as flying BSs is designed to serve an acceptable QoS [78]. In [79], the authors introduced two sub-channel access schemes in the UAV-aided UDNs to decrease the interference from adjacent UAVs. In [80], a link-adaptive constellation division multiple-access technique was proposed to be used in UAV-aided mmWavebased UDNs to cancel intra-beam interference and mitigate adjacent-beam interference. In [81], Chen et al. proposed a Deep Q-Network based RA to maximize the EE in UAVaided UDNs.

I. DEEP LEARNING (DL)
DL which is a powerful branch of machine learning is inspired by the human brain with many node layers [82]. Each neural network combined multiple layers of nodes with an activation function, and weights are used for connecting layers. In the training process, backpropagation phases calibrate these weights replied on training data. This trained neural network consists of dominant features of training data. Then, in the testing process, new input data is passed through this neural network to generate the output. The interesting thing is that the testing time is extremely lower than the training time, or DL reduces the online testing time by increasing the offline training time. Therefore, DL is a powerful tool in realtime systems such as object detection or recognition, selfdriving cars, virtual assistance, human behavior analysis [82]. Clearly, in wireless communication, a trained neural network cannot offer a better solution compared with a known optimal algorithm with a model. However, we cannot always form models for optimization problems in UDNs or it is too complex or impossible to solve. In these situations, DL is a suitable alternative option. In [83]- [85], authors proposed DL-based approaches to estimate channel state information (CSI) which is hard to model because of stochastic and timevarying characteristics. This can support avoiding excessive feedback overhead in UDNs since the processors at core networks can predict CSI without feedback from UEs to control BSs to transmit effectively.

IV. PRACTICAL OPTIMIZATION AND GAME THEORY FOR LARGE-SCALE MODELS OF UDNS
In large-scale systems of 6G, almost all services require real-time computing and ultra-reliable low-latency communications (URLLC). Therefore, within time-varying wireless channels, multi-tier and dense environments, distributed, and self-organizing optimal algorithms have an important role [86]. In this section, we present an introduction of GT, which is a powerful distributed framework, and realtime optimization. In addition, the combination of them for solving effectively the problems of UDNs is also described.

A. GAME THEORY
GT is used as a mathematical tool to understand and model cooperative or competitive situations which have multiple rational and selfish decision-makers. Several models of games are designed for different situations to be suitable with the relationships as well as the state of players. There are four fundamental criterias to classify games. Firstly, depending on the cooperation level, games can be divided into CGs and NCGs. Secondly, if the players sequentially take their actions, these games are called as sequential games (or extensive games) and called simultaneous games where the actions of players are chosen simultaneously. Thirdly, when each player fully knows the actions of the others in their turn, these games are perfect information games and are imperfect information games, vice versa. Fourthly, games are called as complete information games if all players know all information of games (such as strategies, payoffs, etc), and called as incomplete information games if one or more players know the part of the information [87]. Generally, every game combines three main components that are players, possible actions, and payoffs of actions. The strategy of a player maps the instant information of the game to its action. Each player only cares about itself, tries to maximize its payoffs by choosing a strategy that instructs to act depending on its own available knowledge. A Nash equilibrium (NE) is usually used as a solution of a game. The NE profile consists of strategies of all players with the condition to be that each strategy brings the maximum payoff to each player. Intuitively, nobody has an incentive to deviate from their strategy if an equilibrium profile is played. A game can have more than one NE solution so Pareto efficiency (PE) is used for evaluating the performance of NE [88]. In the following contents, we denote matrices and vectors as uppercase bold and lowercase bold letters respectively, sets as calligraphy font letters and scalars, functions as no-bold letters.
To form mathematical definition, we assume that a game have N players with s i , u i denoted the strategy and the utility function of player i (0 < i ≤ N ), respectively. Let S i denote the set of possible strategies of player i so we have s i ∈ S i . Let s −i and S −i be respectively the strategies and the set of all possible strategies of all players except player i. Therefore, we have s −i = s 1 , ..., s i−1 , s i+1 , ..., s N , s −i ∈ S −i , the strategy profile s = (s i , s −i ) which consists all strategies of all players.
• NE definition: Given the strategies s −i of all players except player i, the best response s * i strategy of player i is defined as a strategy that satisfies (1). There may be more than one best response of s −i so we denote the set of these best responses as BR(s −i ) (s * i ∈ BR(s −i )) [88].
Intuitively, with the best response s * i , player i does the best in his capability to maximize his interest (the payoff). In the NE profile, strategies of all players are their best responses, and as a result there is no player who wants to change his or her strategy to get lower payoffs. Therefore, we can say that the equilibrium is established. A NE satisfies the following condition as [18] s • PE known as Pareto optimality: With player i, the strategy s i strictly dominates s i if The strategy profile s Pareto-dominates s if any strategy s i ∈ s strictly dominates s i ∈ s . In other words, all players strictly prefer s to s . Thus, a strategy profile is Pareto-optimal if there is no strategy profile that Pareto-dominates it. A Pareto optimal NE s p satisfies the condition in (4) [18], [88].
where s * and S * are the NE and the set of NEs, respectively. When Pareto efficient strategy profile is played, there is no player who can get more payoff by changing its strategy without harming the payoff of other players.

B. REALTIME OPTIMIZATION
In general, an OP can be expressed as: where f (x) is the objective function, g i (x) and h j (x) are the constraint functions for creating the feasible set F , x is the vector of variables. There are two types of OPs: convex and non-convex OPs. Because of no formal assurance of finding the optimal solution in non-convex problems, the solving methods and software lack efficiency [89], [90]. Meanwhile, there are many powerful methods as well as programming to solve convex problems such as linear programming (LP), quadratic programming (QP), quadratically constrained quadratic programming (QCQP), semi-definite programming (SDP), or Simplex Method, Interior-point Method, Lagrange duality method etc. [17], [91], [92]. However, almost all optimal problems in wireless communications are non-convex. Thus, non-convex problems need to be transformed into convex problems which provide an acceptable approximate solution to the original problems [89], [93], [94].
To solve the problems in UDNs, the mathematical formulas treated as the objectives and constraints are used for forming OPs defined in problem (5) in a given scenario. Most of these problems are non-convex and complex. In UDNs as large-scale models with a large number of BSs and UEs, the non-convex OPs become extremely complex and quite challenging to solve [16], [89], [95]. It is a long journey between the optimization theory and the optimization applied in realistic systems [19], [96]. On the other hand, realtime optimization is very crucial in UDN-enabled models since UDNs are applied in 5G and B5G which strictly require low processing time. The systems using realtime optimization algorithms have trouble or even fail if the processing time exceeds the timing constraint (deadline time limit). Therefore, realtime OPs have to consider both two subproblems: finding the optimal feasible point and satisfying timing constraint [19], [97]. Both the available computing capability of processors and the complexity of the OP need to be simultaneously considered to design solving method.

C. REALTIME OPTIMIZATION AND GAME THEORY FOR UDNS (ROG-UDN)
GT and optimization are used in many different applications in UDNs such as power allocation, spectrum allocation, interference management, user association, offloading, effective capacity, physical layer security, edge caching, etc. A summary of recent approaches of optimization and GT in UDNs is provided in Table 1 with LD denoted leader, FL denoted follower, CHs denoted cluster heads, and ANs denoted access nodes. In these given UDNs, GT with a variety of game models can be used for describing the relationship between the network elements, the information being available to them, and their objectives in different applications. There is no clear boundary between GT and optimization since the solution of a game (NE) is defined from OPs. Clearly, we can rewrite (1) as follow Therefore, GT and optimization can join together to effectively solve a problem. Figure 2 shows two different approaches for solving a distributed OP in UDNs. The first approach is that the given problem is modeled by a game model after determining the relationship, knowledge, and objective of players. The game is divided into multiple sub OPs which are solved to obtain the solution. Secondly, an OP is designed to describe the given problem. This OP is usually very complex and even is a non-convex. Then, the OP is transformed into a game with multiple convex sub-OPs which is easy to find the solution.
Using GT, the complexity of an OP which is one of the key factors for real-time optimization becomes lower. The complex OP can be divided into several sub-OPs for every player in the game. This method can use parallel computing for solving so that it is suitable to apply in real-time applications with large-scale models like UDNs [89]. In addition, original non-convex OP can be automatically transformed into several convex OPs for every player in the game if the utility functions are well defined. Therefore, the joint of optimization and GT is a promising method for large-scale UDN-enabled models with massive data, a large amount of BSs, UEs, multiple tiers. VOLUME xxxx, xxxx

V. SOME CASE STUDIES OF ROG-UDN
In this section, we give typical case studies using clustering, game theory, and optimization to solve some problems in UDNs. The descriptions emphasize the process of designing system models as well as forming the game models. Case studies are arranged according to the process of serving UEs. Firstly, we begin with Case Study 1 and Case Study 2 generating BS-UE associations. After that, a game-based cell clustering method for cooperative transmission is described in Case Study 3. Then, Case Study 4 presents resource allocation (both sub-channel allocation and power allocation) for one-tier networks. Finally, power allocation based on Stackelberg game for two-tier networks in a sub-channel is presented in Case Study 5. The computing platform which uses the CVX and CVXPY tools [124], [125] in MATLAB and PYTHON for solving convex OPs is used for extracting the results. The processing unit is with a PC having CPU @3.70GHz and 32GB memory.

A. CASE STUDY 1: HUNGARIAN ALGORITHM
In this case study, we investigate BS-UE associations using the well-known Hungarian algorithm. Pairing a BS and a UE is an efficient approach for saving network resources and reducing the mobile data traffic of UDNs. The channel model from the BSs to the UEs is represented as β i,k g i,k , where β i,k is the path loss and large-scale fading of the association of the ith BS to kth UE, while g i,k ∈ CN (0, 1) presents the small-scale fading. The path loss from BS to UE is estimated as 128.1 + 37.6 log 10 R [dB], where R be the distance in km.
The signal received at UE k by BS i is where x i,k is the information from BS i intended to UE k and n k is the additive white Gaussian noise (AWGN) at UE k. The signal-to-interference (SIR) of BS i and UE k link is given by By following matrix represents, an associating problem is with M BSs to be matched with K UEs. The BSs are with different rows, the UEs are with different columns. The entries of the matrix represent the SIR of the communication link of the BS associated with the row is matched with the UE associated with the column.
Thus, a UE is to be assigned to a BS so that the total SIR of K UEs in the network will be maximum. To sum up, an association BS-UE problem can be expressed as where The restrictions of (9b) and (9c) are represented that the kth UE will be assigned only by one BS and the ith BS will be done by one UE.
To solve the associated problem (9), an efficient Hungarian algorithm can be proposed in Algorithm 1.
To evaluate the performance of the associated problem in the scenarios of UDNs, we consider a circular cell network with radius 500 m, where M BSs and K UEs are randomly distributed location in the cell. There are M = {50, 100, 150} BSs and K = {10, 30, 50} UEs in the considered network. The average execution time for performing Algorithm 1 in three different models are at {0.75, 1, 3.5} s. The results of BS-UE association are illustrated in Figure 3. Subtract the smallest number in each row of SIR minprob from all numbers in that row. 6: Subtract the smallest number in each column of SIR minprob from all numbers in that column. 7: Repeat 8: Draw lines through rows and columns to cover all zeros in modified matrix with minimum number of lines n min . 9: if n min < max(M, K) then 10: From the elements that are not covered by any line, subtract the lowest number from these elements. 11: Add this lowest number to elements crossed by any two lines 12: end if 13: Until n min = max(M, K). 14: Obtain the optimal association from zeros. 15: Output: The association matrix X = {x i,k | i = 1, ..., M and k = 1, ..., K}.

B. CASE STUDY 2: K-MEANS AND GAME THEORY
Different from paired-association algorithm in Case Study 1, K-means algorithm is a popular approach for clustering problem with multi-association. From that, we exploit Kmeans algorithm for two clustering cases of UDN scenarios. Firstly, we investigate the network model where the number of BSs (M ) is much larger than the number of UEs (K). Secondly, we consider the case of K M . In the first case, we provide a K-means algorithm for clustering a UE supported by multiple BSs in a cooperative communication network. In the last case, we focus on a K-means clustering for grouping multiple UEs based on a coverage network of a BS.
For an instant, we investigate a constrained K-means clustering with the SIR performance as defined in (8) for K M . Therefore, a UE can be served (clustered) by a BS within its coverage if and only if the SIR of the link between the UE and the BS must be greater than a given threshold. An efficient constrained K-means clustering can be found in [92]. From that, two types of association constraints are provided as • Must-link constraints (i, k) ∈ L must indicate that the kth UE has to be served in BS i with satisfied a SIR constraint. • Cannot-link constraints (i, l) ∈ L not imply that the lth UE should not be placed in cluster i. Then, Algorithm 2 where we propose a constrained K- means clustering algorithm to identify the optimal number of UE-BS clusters is used to obtain results. The BSs' locations BS loc,i will set as the centroid {θ Assign appropriate UEs into the cluster with the largest SIR performance. 10: Update the centroids: 11: The updated centroid will be the BS location with nearest distance 12: argmin{ k∈Km U E loc,k , m = 1, .., J clus .

13:
Until The cluster members do not change or the procedure reaches to N max . 14: Set M * = J clus . Set J clus = J clus − 1. For a simulation, we consider a circular cell network with radius 500 m, where M BSs and K UEs are randomly distributed in the cell. By using CVXPY tool on the considered processing unit for implementing Algorithm 2, the clustering results and execution times under scenarios of M = {20, 50, 80} BSs and K = {100, 500, 800} UEs are illustrated in Figure 4 and Table 2. Similarly, for M K, centroids are the UE locations, and the roles of BSs and UEs are swapped. However, the number of centroids cannot be decreased unlike the K M scenario. Therefore, Algorithm 2 can be used to obtain a clustering result with the constraint of BS-UE distances and without considering to choose the optimal number of centroids.

C. CASE STUDY 3: COOPERATIVE TRANSMISSION -COALITION GAME
With the very high density of small cells in UDNs, the intercell interference is a big challenge. In this case study, we consider cooperative transmission to deal with this kind of interference. Cooperative transmission forms SBSs, which extremely interfere with each other, into clusters described in Figure 5. In a cluster, there is no intra-cluster interference since the SBSs shake hands together to transmit the signal. Therefore, each SBS has an incentive to consider whether or not it cooperates with other SBSs to form a coalition to serve their UEs. This problem is modeled by a cooperative game. .., U } to be the set of all SBSs and UEs, respectively. In this case study, inactive SBSs which are serving no UE are also considered together with active SBSs since they may join clusters to cooperatively boost the performance of serving UEs. Both SBSs and UEs are equipped with one omni-directional antennas per UE. Additionally, we denote C = {C 1 , C 2 , ..., C C } to be the set of all clusters. Each cluster combines SBSs which are cooperating with each other to serve their UEs. Thus, SBSs in cluster C i = {1, 2, ..., C i } transmit cooperatively signals to UEs in their small cells, and we denote the set of these UEs as U Ci = {1, 2, ..., U Ci }. For simplicity, we adopt two assumptions as follows • All the SBSs in C use the power of P max to transmit the signal to each UE. • In a cluster, there is no intra-cluster interference since all SBSs serve only one UE at any time, and UEs are served in the same duration of time slot. Therefore, the signal-to-interference-plus-noise ratio (SINR) of UE u served by cluster C i is expressed by where h u,b combining path loss and small-scale fading (Rayleigh fading) represents the channel response from SBS b to UE u, and σ 2 u is the power of AWGN at UE u. The collection of SBSs which join cooperatively clusters to serve their UEs is formed as a coalition game GAM E = P L, v with non-transferable utility (NTU) where P L = B is the set of players (SBSs), and v(C) ⊆ R C is a set of VOLUME xxxx, xxxx payoff vectors [126]. Each cluster C i ∈ C is the coalition whose value is each element v(C i ) of the set v(C). To avoid an overhead increment of complexity and CSI data exchange, we limit the number of SBSs in each cluster by N max . The value of a coalition C i is defined as where U Ci is the number of UEs served by SBSs in C i , and BW is the channel bandwidth. The preference condition is defined that SBS b prefers to leave coalition C j to join coalition C i denoted by Intuitively, SBS b decides to cooperate with a coalition for increasing the sum of utility values. There are two kinds of operation for changing players between the coalitions are used [118].
• Split and merge: The operation of SBS b ∈ C j leaving C j to join C i can be expressed as • Swap: If one or two of coalitions C i and C j has the number of SBSs equaling N max , and SBS b ∈ C i prefers C j to C i while SBS b ∈ C j prefers C i to C j , then they swap their positions as To extract the set of clusters C, we use Algorithm 3. For a simulation, a network covers an area with hexagon radius of 1000 m which includes 50 SBSs and 30 UEs with uniform distribution. Each SBS has one antennas with P max = 30 dBm and the bandwidth of 20 MHz. Figure  6 illustrates the implementation of clustering SBSs using coalition game with the maximum number of 3 SBSs being clustered in each cluster. For large-scale models, three scenarios are also performed for {200, 400, 600} SBSs and {70, 100, 200} UEs by spending {125, 350, 840} ms of our processing unit, respectively.

D. CASE STUDY 4: RESOURCE ALLOCATION (RA) -NCG
GT is a distributed computational tool to solve effectively RA problems in UDNs with lower complexity. In this case study, a method that combines a sub-channel allocation (SCA) process and a power allocation (PA) process is used for RA to optimize network throughput. A NCG is built to provide optimal power to each SBSs. We consider a multi-cell downlink transmission system using orthogonal frequency division modulation (OFDM) in an UDN where SBSs, SUEs are randomly high-density distributed. It is assumed that macrocells and small cells use two separated spectrum resources. Therefore, there is no inter-tier interference. Small cells use the same spectrum resource of N orthogonal sub-channels with a bandwidth of BW per sub-channel to transmit downlink Algorithm 3 Coalition game algorithm for clustering SBSs Input: The channel matrix H, the power of AWGN σ 2 , the channel bandwidth BW , the maximum power to each UE of SBSs P max , the set of players P L. Output: The optimal set of all clusters C 1: Initialization: Initialize a random partition C (0) 2: Assign l ← 0 3: while C (l) change do 4: for b ∈ B do 5: Given b ∈ C j 6: if |C i | = N max then l ← l + 1 22: end while data to UEs. Thus, inter-cell interference is only taken into account in this case study.
We denote B = {1, 2, ..., B} to be the set of active SBSs which are serving at least one UE u ∈ U where U = {1, 2, ..., U } is the set of UEs. One UE associates with only one SBS, and one SBS can serve multiple UEs with the number limited by N in different sub-channels. U b represents the cluster of UEs which are served by SBS b, and as a result we have U = ∪ b∈B U b and U b ∩ U b = ∅, ∀b = b . The SINR of SUE u served by SBS b in sub-channel n is expressed by where p (n) u,b is the desired transmit power using at SBS b to serve UE u, p (n) u ,b represents the interference power using at SBS b to serve its UE u , h (n) u,b is the channel response between SBS b and UE u in sub-channel n, σ 2 u is the power of AWGN at UE u, a The RA method is decomposed into two processes. The SCA process: Firstly, UEs connect to the nearest SBSs (also known as SUEs), and the number of UEs served by a SBS is limited by N . In small cell b, sub-channel n is allocated to UE u which has the maximum channel gain [107]. Given b ∈ B and n ∈ {1, 2, ..., N }, this is described mathematically as Then, we assign a The PA process: The objective is to maximize the network throughput. The PA is described by an OP as follows where P is a matrix of the transmit power of SBSs in all subchannels, P (n) max is the maximum transmit power of each SBS in sub-channel n, U (n) represents the set of UEs served in sub-channel n. (18b) represents that the power in each subchannel does not exceed P (n) max . (18c) indicates that the minimum throughput of each UE equals to T p min to guarantee the QoS. Clearly, OP (18) is a non-convex and very complex OP with massive SBSs and UEs in UDNs. After the SCA, OP (18) can be divided into |N | PA sub-OPs corresponding to different orthogonal sub-channels since there is no intra-cell interference in each small cell. Each sub-OP corresponds to a PA problem where each small cell serving only one UE. The sub-OP of sub-channel n can be expressed as follows where p (n) denotes the vector of the transmit power of SBSs serving UEs in U (n) . In consideration of small cells corresponding to self-interested players, we model OP (19) as a NCG denoted by the normal form GAM E = P L, {P i } i∈P L , {T p i } i∈P L . P L is the set of players (i.e., small cells), max } represents the set of possible transmit power strategy of the player i, T p i denotes the utility function (i.e., throughput) of the player i defined as (16) with UE u to be served by player i. A strategy profile is a vector denoted as p = (p 1 , p 2 , ..., p B ) in the joint strategy space of all players. The optimal profile p * = (p * 1 , p * 2 , ..., p B ) * is the NE profile to be expressed as Each OP in (20) is convex with one variable p i . To extract the solution, we use Algorithm 4.

Algorithm 4
Iterative PA for extracting the NE profile in a sub-channel Input: The channel matrix H, the power of AWGN σ 2 , the bandwidth of each sub-channel BW , the maximum transmit power in each sub-channel of SBSs P (n) max , the minimum throughput T p min , the set of players P L Output: The vector of optimal transmit power of all players p * in sub-channel n 1: Initialization: Initialize p 2: while the convergence is not reached do update the best respond power of player i according to the convex OP where p −i is the set of the power of players except player i 5: end for 6: end while For a simulation, we assume the coverage of a hexagon area with the radius of 1000m. The SBSs and UEs are uniformly distributed with the numbers of 50 and 30, respectively. The spectral resource is divided into 5 orthogonal sub-channels with 4 MHz per sub-channel. The maximum power of SBSs is 30 dBm. The PYTHON code generated by VOLUME xxxx, xxxx CVXPY is used for solving the convex OPs in Algorithm 4. Table 3 shows the network performance and the execution time for implementing Algorithm 4. For an instance of 50 SBSs and 30 UEs, the result shows that the average throughput of UEs equals to 15 Mbps, the minimum throughput of UEs equals to 2 Mbps and the processing unit spends 2.5 s for solving the problem.

E. CASE STUDY 5: PA IN A TWO-TIER NETWORK -STACKELBERG GAME
The fundamental structure of an UDN consists of two tiers that are a macro cell and small cells overlaid in this macro cell. If UEs go out the coverage of small cells, they need to be served at least the minimum constraint of throughput. Therefore, it is essential to propose an efficient method for controlling power between these two tiers. Stackelberg game is one of the extensive form games where players do not choose simultaneously their actions. In a duopolistic setting Stackelberg competition, players as leaders begin the game by choosing their actions. Then, players as followers observe the actions of leaders and choose their own suitable actions. In this case study, a Stackelberg game is built for power allocation with a MBS (leader) and SBSs (followers) in an UDN. We assume that after sub-channel allocation, each small cell serves only one UE in a given sub-channel. In addition, each BS is equipped with one antenna while the MBS has many antennas to serve multiple UEs using the same resources.
The received signal y m of MUE m and the received signal y u of SUE u served by SBS b is respectively expressed by is the channel response with β u,b be the path loss and large-scale fading, and the small-scale fading g u,b ∼ Rayleigh(1). The SINR of SUE u and MUE m are respectively written as Since a MBS transmits the power to be much more than that of each SBS and the power change of a SBS has negligible effect on MUEs, we approximate the interference of each MUE as [107] where P max SBS is the maximum transmit power of each SBS. We have that |g| 2 ∼ Exp(1/2) and the probability density function of d is f (d) = e −ρλπd 2 2πρλd where d denotes the distance of an association, λ is the density of SBSs, and ρ represents the active proportion of SBSs [107]. Thus, γ m can be approximated as a function of p m .
PA for the leader (MBS): To extract the optimal PA for the beams of MBS, we solve the convex OP as follows max pM m∈M p m ≥ 0, ∀m ∈ M, (27c) BW log 2 (1 + γ m ) ≥ T p min , ∀m ∈ M, where p M is a vector of all p m , ∀m ∈ M, P max M BS is the maximum transmit power of the MBS in the given subchannel, and T p min is the minimum throughput to serve each MUE.
PA for the followers (SBSs): After obtaining the optimal action of the leader, the followers substitute it into (24). The utility function of each follower b authorized SUE u is defined as We build a NCG with the normal form GAM E = B, {P b } b∈B , {T p b } b∈B to find optimal PA for SBSs to maximize the network throughput where B is the set of players (i.e., SBSs), P b = {p b |0 ≤ p b ≤ P max SBS } represents the set of possible transmit power strategy of player b. Algorithm 4 is used for obtaining the solution because this game is the same with the NCG in Example 4.
We build up a two-tier UDN with a 16-antenna MBS located at the center of the hexagon area of radius 1000 m and 3 MUEs. In small-cell tier, independent homogeneous poisson point processes (PPP) are used for modeling the locations of SBS over the macro cell with the density of the SBSs, λ = 40 BS/km 2 and active proportion ρ = 50 %. The distance between two any SBSs is at least 100 m (i.e. the radius of each small cell is 50 m). In addition, each active small cell have only one small UE in its coverage. The maximum power of each MBS and each SBS is 30 dBm and 50 dBm, respectively. Furthermore, the minimum capacity of each UE served with the bandwidth of 20 MHz is 0.1 Mbps. CVX package in MATLAB is used as a programming tool to solve the convex OPs in Example 5. The capacity at 3 MUEs equals to 0.1 Mbps, 0.1 Mbps, and 0.27 Mbps, respectively. The minimum capacity of SUEs is 2.147 Mbps.

VI. CONCLUSIONS
This work has investigated and proposed some potential approaches for 6G UDNs to deal with the challenges of the deployment of large-scale networks. Due to very high densities of network infrastructures, the optimization problems of UDNs have become very large and complicated under a limited resource with a stringent constraints of small execution time. The strict requirements of time processing in the next generation of wireless networks have led to smart system processing and realtime optimization in UDNs and have become more important than ever. To address this research challenge, we have developed some GT schemes for dealing with many players in multi-tier networks under competition and cooperation policies. By designing reasonable utility functions for games and low-complexity optimization algorithms, complex OPs have been transformed to convex OPs that can be easily solved in many different ways and by powerful programming tools. A novel amalgamation of GT and realtime optimization has been proposed to produce many effective approaches for fast and exact achieving solutions of the large-scale problems of UDNs. The proposed ROG-UDN will be a potential topic in the future of wireless communication systems.