Modeling and Thermal Stress Coupling Optimization Design of TSV Inductors in On-Chip DC/DC Converters

As advances in fabrication technology continue and the power density of microelectronic devices increases, efficient and highly integrated through-silicon via (TSV) inductors have emerged as a viable solution for addressing challenges in power and thermal stress management in on-chip direct current-to-direct current (DC/DC) converters. This study proposes a method for modeling and optimizing TSV inductors with magnetic cores based on thermal stress and lumped circuit. CoZrTa is selected as the magnetic core material, and RLCG circuit modeling is performed for multiple factors, including resistance, inductance value, and parasitic capacitance. To validate the accuracy of the model, simulation tests were conducted using ANSYS HFSS and SPICE tools. The results show that under low-frequency conditions (< 2GHz), the error between the model and the 3D full-wave simulation for $L$ and ${Q}$ parameters is less than 3.3%, proving the model’s accuracy. Furthermore, a deep reinforcement learning optimization design model based on the deep deterministic policy gradients (DDPG) algorithm is proposed, which comprehensively considers factors such as current ripple, power loss, parasitic parameters, and thermal stress. With almost no change in electrical performance, the thermal stress in the optimized TSV inductor array decreased by 11.91% and its distribution was improved, reducing the likelihood of fatigue damage.


I. INTRODUCTION
In integrated circuit and system-on-chip design, power management has become increasingly critical due to the shrinking size of transistors and rising power density.The efficient, compact, and high-performance design of on-chip DC/DC converters is receiving much attention.Inductors, as key components in converters, directly affect the efficiency of the converter.Traditional planar inductors are large in size and low in efficiency.In contrast, on-chip inductors manufactured using Through-Silicon Via technology offer the advantages of small size, high performance, and high integration.However, such designs also face the challenge of thermal stress, which not only impacts the performance of the inductors but could also affect their reliability.Thermal stress-coupled issues also need to be considered in optimization design.
The associate editor coordinating the review of this manuscript and approving it for publication was Norbert Herencsar .
The analytical solution for the stress field of TSV under varying temperature conditions has garnered widespread attention from researchers.Lu and colleagues used the finite element method to analyze the temperature stress field near a single TSV with respect to the spacing and derived a simple analytical solution for the stress around the TSV [1].Ding and team approached the TSV's dielectric layer as a pressure tunnel problem by applying equivalent temperature stress on both sides of the dielectric layer.They obtained the analytical solution for the temperature stress in a single TSV with a thick dielectric layer and analyzed the impact of different thermal expansion coefficients and Young's modulus on the energy release rate [2].Marella and others studied the stress distribution near a single TSV with a thick substrate and analyzed the impact of temperature stress on transistor electrical parameters in the wafer [3].
In recent years, with the development of 3D stacking technology, research on TSV inductance has also been widely noticed by scholars.Tida and colleagues were the first to conduct research on inductance based on TSV in 3D stacked structures [4].Kim and team studied the applications of TSV inductors in radio frequency and proposed a new TSV inductor structure that uses fewer ground layers [5].Mondal and associates further researched the grounding shielding issue of TSV inductance and introduced a new lumped-parameter model, reducing the simulation time for TSV inductance [6].Xiong and collaborators developed a wide-band lumped model for TSV inductance based on a physical model, taking into account skin effects, proximity effects, and parasitic effects in the Redistribution Layer (RDL) [7].Regarding research on TSV inductance in on-chip DC/DC converters, Tida and team were the first to propose this concept [8].Krishnamurthy from Intel Corporation reported the feasibility of TSV inductance and on-chip DC/DC converters, constructed under 14nm CMOS technology [9].Chen and colleagues utilized machine learning methods to extract the circuit equivalent model of TSV inductance and developed an optimization algorithm for TSV inductors based on this [10].
In this study, an RLCG model of TSV inductors was established to calculate electrical parameters and analyze losses.A multi-parameter optimization framework that includes thermal stress for TSV inductors with CoZrTa laminated magnetic cores was developed.Furthermore, a deep reinforcement learning optimization method based on DDPG algorithm was proposed.The highlights of this study can be summarized as follows: • An RLCG model for TSV inductance was established to calculate electrical parameters and power loss.
• The accuracy of the multi-dimensional parameter model of TSV inductance was cross-validated through finite element analysis and SPICE models.
• TSV inductance in on-chip DC/DC converters was optimized using a deep reinforcement learning algorithm to simultaneously improve electrical performance and reduce thermal stress.
The rest of this paper is organized as follows: Section II introduces the basic theories of electrical, thermal, and mechanical aspects related to TSV inductance.Section III discusses the circuit modeling and cross-validation of TSV inductance and proposes optimizations.Section IV presents the analysis of results before and after optimization, and Section V concludes the study.

II. THEORETICAL ANALYSIS A. ELECTRICAL PARAMETERS OF TSV INDUCTORS 1) CURRENT CHARACTERISTICS OF INDUCTORS IN SWITCHING CONVERTERS
For on-chip DC/DC buck converters, Figure 1 shows its typical topology, which includes an inductor, an output capacitor, a diode, and a MOSFET serving as the switching transistor.This topology is usually controlled by a Pulse Width Modulation (PWM) signal with a frequency of f and a duty cycle of D. For the buck converter, the inductor stores energy when the switching transistor is on and releases energy to the load when the switching transistor is off.Its state equation can be written as equation (1).
Based on the Volt-Second Balance Principle, the expression for the inductor under the condition of an output current I o can be written as equation (2).
In this context, r represents the current ripple ratio, defined as the ratio of the maximum change in current to the output current.

2) RLCG MODELING REQUIREMENTS FOR TSV INDUCTORS
With the introduction of TSV, new methods are needed to calculate the equivalent model parameters of TSV inductance.Because the inductor current in the converter contains both DC and AC components, both DC resistance and AC resistance must be considered when calculating its resistance [6].The inductance of the TSV is generally produced by the mutual inductance of the connecting lines [7].However, when a magnetic core is added, the inductance will increase due to the magnetic permeability of the core [11].Therefore, different calculation methods need to be adjusted based on whether a magnetic core is used.The dielectric layer on the outer side of the TSV structure creates a Metal-Oxide-Silicon (MOS) structure between the silicon substrate and the via.When modeling TSV inductance with a magnetic core, the parasitic effects contributing to capacitance and conductance parameters should also be considered [12].

3) POWER LOSS IN TSV INDUCTANCE
The power loss in TSV inductors with a magnetic core can have multiple sources and can be roughly summarized as follows: • Hysteresis loss originates from the hysteresis effect and can be calculated based on the area enclosed by the hysteresis loop.
• Eddy current loss in the magnetic core arises from induced currents within the core, and it becomes especially significant at high frequencies, leading to heat generation in the inductor.
• Winding losses (also known as copper losses) come from the resistive loop formed by the RDL circuitry and the TSV itself.These losses can be broken down into DC and AC components [13], [14].The total power loss in a TSV inductor with a magnetic core can be expressed as [15] equation (3).
In this formula, k h is a hysteresis coefficient related to the material, V is the volume of the magnetic core, f is the switching frequency, B max is the maximum magnetic flux density, u 0 is the magnetic permeability in a vacuum, u(H max ) is the relative magnetic permeability at maximum magnetic field strength, t is the thickness of the magnetic core laminate, ρ is the resistivity of the magnetic core, R dc is the DC resistance of the TSV inductor, and R ac is the AC resistance of the TSV inductor.

B. THERMAL STRESS IN TSV INDUCTANCE
The semi-analytical solution for the thermal stress in a single TSV within a semi-infinite wafer is illustrated in Figure 2 [16].When the TSV inductor is under steady-state operating conditions and under the assumption of frictionless contact, the stress field generated by a single via in the silicon substrate can be described by the equation ( 4).
In this context, σ r and σ θ represent the axial and radial stresses, respectively.D Via is the diameter of the via, r is the distance between the point in question and the center of the via, α Via and α Si are the coefficients of thermal expansion for the via and silicon substrate, respectively, T is the temperature change, E Via is the Young's modulus of the via, and υ Via and υ Si are the Poisson's ratios for the via and silicon substrate, respectively.In TSV inductors, the vias are arranged in an array.Under the assumption of linear elasticity, the stress distribution within each via can be considered as a superposition of the stress field of a single via in an infinitely large silicon wafer under temperature change, along with the stress fields generated at that via's location by other vias as equation (5).
The superscript i represents the index of the current via, i,single represents a via in an infinitely large wafer, jis the index of other vias, and n is the total number of vias.When considering the stress distribution of double-row vias, as shown in Figure 3, the stress at the center of via i can be expressed as equation (6).(6) In this expression, S represents the spacing between the vias, and P represents the distance between the two rows of vias (later referred to as the inductor's spacing).

III. CIRCUIT MODELING AND OPTIMIZATION DESIGN
There are various types of architectures for TSV inductors, such as vertical and toroidal inductors.To demonstrate the methodology in this paper, we have chosen a TSV inductor  with a dual-column, toroidal layout for modeling and optimization.Figure 4 illustrates the structure of the example TSV inductor.It's important to note that the methodology presented in this work is not limited to this example structure and can be extended to other TSV inductor designs following the same analytical approach.Intel Corporation has previously developed a TSV inductor with a magnetic core using a 14nm process, significantly enhancing the inductor's performance [17].Accordingly, our design also employs a CoZrTa laminated magnetic core for study.The magnetic core in the TSV inductor not only reduces high-frequency eddy current losses but also improves the quality factor of the inductor.
To set design constraints, several manufacturing requirements need to be clarified first: • Two magnetic holes are symmetrically distributed in the magnetic core, and the number of vias in each magnetic hole is the same.
133192 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.• The spacing between single-row vias within each magnetic hole is equal.
• The inner spacing of the magnetic hole is twice the distance from the outer edge of the magnetic hole to the edge of the magnetic core.
• The Meaning of Some Dimensional Parameters of TSV Inductors with Magnetic Cores.From this, we can derive the dimensional relationships for the example TSV inductor, which can be expressed as equation (7).

A. CIRCUIT MODELING OF THE TSV INDUCTOR
Figure 5 shows the typical structure of a TSV inductor with a CoZrTa magnetic core.The magnetic core is deposited on the upper layer of the silicon substrate using specialized techniques.The TSV array is embedded within the silicon substrate and is connected through RDLs on both the top and bottom sides.The equivalent circuit model of the TSV inductor is illustrated in Figure 5(b), where R dc , R ac , L, and C represent the DC resistance, AC resistance, inductance, and parasitic capacitance of the TSV inductor, respectively.
First, to calculate the inductance value, the relationship between the internal current and magnetic field strength for a TSV inductor with a magnetic core can be expressed as equation (8).
In the equation, H is the magnetic field strength, l m the magnetic path length, n is the number of turns in the coil, and I is the current flowing through the inductor.Under the conditions illustrated, the magnetic path length can be written as equation (9).
In the formula, λ is the magnetic path correction factor.Additionally, mutual inductance exists among the coils in the TSV array.Therefore, the approximate formula for calculating the inductance with the magnetic core can be written as equation (10).
In the formula, u represents the relative permeability of the CoZrTa magnetic core, u 0 is the permeability in a vacuum, γ is a correction factor, and A cross is the cross-sectional area of the magnetic core.The calculation formula for A cross can be written as equation (11).
A cross = W mag t mag n mag (11) Afterwards, the resistance of the TSV inductor is calculated.The DC resistance of a single through-hole and RDL can be directly determined using the conductor's resistance calculation formula [6] as equation (12).(12) In the formula, ρ represents the resistivity of the material used for the through-hole and RDL.l RDL is the length of the RDL connecting the two through-holes, W RDL is the width of the RDL, and t RDL is the thickness of the RDL.The AC resistance of a single through-hole needs to take into account the skin effect.Therefore, the resistance of the TSV inductor can be calculated using equation (13).
In the equation, l RDL ,1 and l RDL ,2 represent the lengths of the RDL connection lines on the upper and lower sides of the substrate, respectively.α eff and β eff are the correction factors for the skin effect.
To further calculate the parasitic capacitance of the TSV inductor, the parasitic capacitance mainly originates from three aspects: the MOS capacitance of the through-hole, the parasitic capacitance between the through-holes, and the parasitic capacitance between the RDLs.The calculation formula ( 14) for these can be found in [7].
Sl RDL,2 / W RDL ( S 2 +l 2 RDL,2 /4 + l RDL,2 /2) In the formula, t ox represents the thickness of the oxide layer, ε ox is the dielectric constant of the oxide layer, and ε Si is the dielectric constant of the silicon substrate.C dep is the capacitance of the depletion layer.

B. VALIDATION OF TSV INDUCTOR CIRCUIT MODEL
To validate the accuracy of the TSV equivalent circuit model proposed in Section III-A, we performed 3D full-wave simulations on TSV inductors with different numbers of turns and inter-hole spacings in the 0-10GHz frequency range using the HFSS tool in ANSYS software.The dual-port Y-parameters of the TSV inductors were extracted, and the inductance and quality factor were derived from these Y-parameters.The calculation formula can be expressed as equation (15).Simultaneously, based on formulas ( 8)-( 14) in section III-A, we generated the lumped model for the corresponding TSV inductance and conducted simulations using the SPICE tool.We also utilized SPICE to create and simulate the related TSV inductance's lumped model.Following the same procedure, we extracted the dual-port Y parameters for the lumped model within the 0-10GHz frequency range and, using formula (15), determined the inductance and quality factor of the lumped model.The simulation results are shown in Figure 6.When there are changes in inter-hole spacing and the number of inductor turns, the lumped model's inductance (L) has only a 3.3% error compared to the 3D full-wave simulation model at frequencies below 2GHz.At higher frequencies, the lumped model starts to deviate from the fullwave model, which can also be verified by comparing the 133194 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.quality factor (Q).It is especially worth noting that, as shown in Figure 6(c), the circuit resonates around 9GHz, causing a significant shift in the electrical parameters of the lumped model.In low-frequency conditions (< 2GHz), the lumped model and the 3D full-wave simulation results for inductance and quality factor are essentially aligned, confirming the accuracy of the TSV inductor circuit model.

C. OPTIMIZED DESIGN OF TSV INDUCTORS 1) OPTIMIZATION DESIGN MODEL
This section discusses the optimization design model for TSV inductors in on-chip DC/DC converters.For these inductors, it is essential to minimize current ripple, power loss, and parasitic parameters while ensuring normal operation of the converter within a given volume constraint.Additionally, the on-chip DC/DC converter may be subject to periodic thermal fluctuations within the wafer during operation.This can introduce periodic thermal stress to the TSV inductors, potentially accelerating their degradation.Therefore, it is also important to minimize thermal stress in the layout of the TSV array.During the design process, engineers usually need to specify the operating frequency, operating current, and current ripple factor of the on-chip DC/DC converter to determine the target inductance value.Subsequently, further design is carried out based on the maximum magnetic flux density and size constraints of the magnetic core to ensure that the overall inductor design meets the aforementioned conditions.
The optimization design model for TSV inductors aims to find the optimal parameters and layout for the through-hole array in the inductor, under specified current ripple rate and certain magnetic core size conditions, such that both the power loss (P L ) and the thermal stress in the TSV array are minimized.The optimized design parameters are S, S 1 , P, n, and n lam .The objective function for the optimization design of the TSV inductor can be written as equation (16).
In the above, the first inequality serves as the constraint for the current ripple, where r max is the maximum current ripple rate allowable for the inductor.The second inequality ensures that the magnetic core of the inductor does not reach saturation, with B max representing the saturation magnetic flux density (for CoZrTa, it is 1.5T).The third and fourth inequalities are constraints on the dimensions of the magnetic core.The fifth inequality constrains the number of turns in the coil, ensuring that the inductor does not have a non-integer number of turns.

2) OPTIMIZATION DESIGN ALGORITHM
The problem described above is a multi-objective optimization problem with constraints.Genetic algorithms or particle swarm algorithms are commonly used for optimization.However, due to the semi-analytical nature of stress distribution in the TSV array, it is challenging to explicitly define the optimization function using analytical formulas [18], [19].Therefore, we use Deep Reinforcement Learning (DRL) algorithms to solve the problem.In DRL, an intelligent agent constructed from deep neural networks interacts with the environment in a Markov Decision Process (MDP) defined by the tuple {S, A, T, R} and continually learns to optimize its strategy to obtain the optimal solution for the multi-objective optimization problem [20], [21], [22], [23], [24].In the tuple, S represents the state space, A denotes the action space, T is the state transition function, which is generally not needed to be defined in DRL, and R is the reward function.The   Deep Reinforcement Learning framework for the optimization design process of TSV inductors is shown in Figure 7.
Here, the state space S is the source of interaction information between the intelligent agent and the environment.Its state parameters include the area of the inductor's magnetic core, the maximum stress at the center of the through-hole, and circuit model parameters.The state space at time t is defined as equation (17).
In the formula, σ center max,t represents the stress at the center of the via.The action space A is the output of the intelligent agent's decisions.In this work, the intelligent agent adjusts the magnetic core loss, thermal stress, and electrical parameters of the TSV inductor by altering design optimization parameters.The action space at time t is defined as equation (18).
The reward function R is the return for the agent's strategy adjustment.The ultimate goal of deep reinforcement learning is to maximize this optimization return.In this work, we define the reward function at time t as equation (19).
In the formula, r c is the penalty for the agent when the optimization parameter exceeds the size constraint condition in formula ( 16); α and β represent the weights of magnetic loss and temperature stress in the optimization, respectively.Since the two optimization objectives are unrelated, they are both set to 1/2.The structure and algorithm of the intelligent agent, which is key in DRL, will be introduced in the following section.

3) DDPG-BASED AGENT ITERATION PROCESS
Although a semi-analytical solution for the stress distribution of TSV inductors has been obtained in previous sections, it can only guarantee accuracy when the TSV aspect ratio is very large, and it is difficult to give an analytical formula.Therefore, an agent based on the DDPG algorithm was constructed in reinforcement learning [25].The agent constructed by the DDPG algorithm contains four neural networks, and its iterative process is shown in Figure 8: 1) Initialize the parameters of the Actor and Critic networks, θ u and θ Q , and update the network parameters to the parameters of the Target-Actor and Target-Critic networks, θ u' and θ Q' .2) The Actor network interacts with the environment, and the obtained S t , A t , r t , S t +1 are stored in the experience pool.
3) The experience pool samples M groups of data, inputs the data into the Target-Critic network for update iteration, and simultaneously inputs S t , S t 0 +1 into the Target-Actor network for update iteration.4) The Target-Actor network outputs A t to interact with the environment, outputs A t +1 to the Target-Critic network, receives the Q value of the action function, and updates the network.5) The Target-Critic calculates the Q value, combines Q with the reward function R, and updates the network.
133196 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

IV. RESULT ANALYSIS
In this study, a thermal stress-coupled optimization design method was used to further optimize the TSV inductors in the previously reported on-chip DC/DC converter [15].In this study, the working temperature range of the TSV inductor was set at 293K-373K, and the requirements for the DC/DC converter parameters are shown in Table 2.The process parameters and the optimized design parameters are shown in Table 3, where L is the length of the magnetic core and T is the width of the magnetic core.We used the Python language and built the TSV inductor optimization design algorithm introduced in Section III under the PyTorch framework.We trained the agent with the Actor's learning rate set at 3 × 10 −4 and the Critic's at the same rate.The batch size was set to 256 with a discount rate of 0.99.As illustrated in Figure 9, after 1500 training iterations, the reward return graph shows convergence by approximately the 800th iteration.
The comparison of electrical parameters and dimensions before and after optimization is shown in Table 4.It can be seen that the electrical performance of the TSV inductor after optimization has not changed much compared to before optimization.The area occupied by the TSV inductor after optimization has increased compared to the inductor before optimization, mainly due to the misalignment of the two rows of through holes in the TSV inductor, forming a serrated arrangement and increasing the length of the magnetic hole in the magnetic core.
The temperature stress distribution of the through-hole array in the TSV inductor before and after optimization is shown in Figure 10.Before optimization, the through-hole array in the TSV inductor was arranged without misalignment, and its maximum stress value was 271.1 MPa, while the maximum stress value of the through-hole array in the TSV inductor after optimization decreased to 260.4 MPa.
The stress distribution of the through-hole array before and after optimization is shown in Figure 11.After optimization, the minimum stress of the through-hole appears on the outer surface of the through-hole, changing the stress distribution on the outer surface of the through-hole.With basically the same electrical performance, our optimization method slightly increased the volume of the TSV inductor, reduced the temperature stress of the TSV inductor array by 11.91%, improved its distribution, and reduced the probability of fatigue damage.

V. CONCLUSION
This study is based on the construction of an RLCG model for CoZrTa laminated magnetic core TSV inductors used in on-chip integrated DC/DC converters, and the optimization of its stress distribution through a deep learning neural network constructed by the DDPG algorithm.The conclusions are summarized as follows: • An RLCG electrical parameter model with a magnetic core TSV inductor is established to analyze the combined parameter effects of power loss and electrical characteristics.
• The accuracy of the model is cross-validated using SPICE tools and HFSS finite element analysis, and the optimization design objective function is pointed out.
• A deep learning framework based on the DDPG algorithm is proposed to optimize the complex parameter space of TSV inductors.After optimization, the temperature stress of TSV inductors drops to 260.4Mpa, a decrease of 11.91%, the distribution is improved, and the electrical parameters do not change much, and the probability of fatigue damage decreases.

FIGURE 1 .
FIGURE 1. Topology and key component waveforms of the DC/DC Buck Converter: (a) Topology Diagram; (b) Waveform Diagram.

FIGURE 2 .
FIGURE 2. The Semi-analytical solution for thermal stress in a single TSV: (a) Problem Decomposition; (b) Solution to the Lame Problem; (c) Solution to the Boussinesq Problem.

FIGURE 3 .
FIGURE 3. Stress in TSV Array: (a) Stress in the x-direction; (b) Stress in the y-direction.

FIGURE 4 .
FIGURE 4. The structure of a TSV inductor with a magnetic core.

FIGURE 5 .
FIGURE 5. Structure and equivalent circuit of the TSV inductor with magnetic core: (a) TSV Inductor in the Substrate; (b) Equivalent Circuit; (c) Coil of the TSV Inductor.

FIGURE 6 .
FIGURE 6.Comparison of inductance and quality factor calculated by magnetic-core-included TSV inductor circuit model and 3D full-wave simulation model under 0∼10GHz conditions: (a) Varying TSV Lengths; (b) Varying Number of Coil Turns; (c) Varying TSV Spacing; (d) Varying Inductor Spacing.

FIGURE 8 .
FIGURE 8.The iterative process of an agent based on the DDPG algorithm.

FIGURE 10 .
FIGURE 10.The stress distribution contours of the through-hole before and after optimization: (a) Represents before Optimization; (b) Represents after Optimization.

FIGURE 11 .
FIGURE 11.The stress distribution contours of the through-hole before and after optimization: (a) Represents before Optimization; (b) Represents after Optimization.

TABLE 1 .
The meaning of some dimensional parameters of TSV inductors with magnetic cores.

TABLE 2 .
DC/DC converter design specifications.

TABLE 3 .
Manufacturing and design parameters of TSV inductors with magnetic cores.

TABLE 4 .
The size and electrical parameters of TSV inductors before and after optimization.