Simulation and Optimization of IGZO-Based Neuromorphic System for Spiking Neural Networks

In this paper, we conducted a simulation of an indium-gallium-zinc oxide (IGZO)-based neuromorphic system and proposed layer-by-layer membrane capacitor (Cmem) optimization for integrate-and-fire (I&F) neuron circuits to minimize the accuracy drop in spiking neural network (SNN). The fabricated synaptic transistor exhibited linear 32 synaptic weights with a large dynamic range <inline-formula> <tex-math notation="LaTeX">$(\sim 846$ </tex-math></inline-formula>), and an n-type-only IGZO I&F neuron circuit was proposed and verified by HSPICE simulation. The network, consisting of three fully connected layers, was evaluated with an offline learning method employing synaptic transistor and I&F circuit models for three datasets: MNIST, Fashion-MNIST, and CIFAR-10. For offline learning, accuracy drop can occur due to information loss caused by overflow or underflow in neurons, which is largely affected by Cmem. To address this problem, we introduced a layer-by-layer <inline-formula> <tex-math notation="LaTeX">${\mathrm{ C}}_{\mathrm{ mem}}$ </tex-math></inline-formula> optimization method that adjusts appropriate <inline-formula> <tex-math notation="LaTeX">${\mathrm{ C}}_{\mathrm{ mem}}$ </tex-math></inline-formula> for each layer to minimize the information loss. As a result, high SNN accuracy was achieved for MNIST, Fashion-MNIST, and CIFAR-10 at 98.42%, 89.16%, and 48.06%, respectively. Furthermore, the optimized system showed minimal accuracy degradation under device-to-device variation.


I. INTRODUCTION
Neuromorphic computing systems for artificial neural networks (ANNs) and spiking neural networks (SNNs) have emerged as a promising solution to address memory bottleneck originating from von Neumann architecture with large-scale data [1], [2], [3].Notably, SNN is well-suited for neuromorphic systems due to its biologically plausible mechanism mirroring the human brain and its potential for highly efficient parallel computing [4].To realize a hardware SNN system, it is essential to implement: (1) artificial synapses that can mimic the synaptic plasticity and (2) artificial neurons that can emulate the spike behaviors [5], [6].Recently, various studies on artificial synapses and neurons using indium-gallium-zinc oxide (IGZO), which has the advantages of a moderate mobility, a low leakage current, and a low-temperature process, have been actively proposed to advance the SNN system [7], [8], [9].Configuring an SNN system using both IGZO-based artificial synapses and neurons enables its fabrication through a low-temperature process, offering several advantages such as application on flexible or stretchable substrates and compatibility with back-end-of-line (BEOL) process [10], [11].However, recent studies have primarily focused on developing either a synapse or a neuron individually, and there is still a lack of research on SNN systems that incorporate both synapses and neurons [12].
In order to achieve a high-performance SNN system, it is important to optimize the training method and system design that consider the properties of synapses and neurons [13].In the training methods for SNNs, there are mainly two approaches: online and offline learning [14], [15].For online learning, the spike-timing-dependent plasticity or c 2024 The Authors.This work is licensed under a Creative Commons Attribution 4.0 License.
For more information, see https://creativecommons.org/licenses/by/4.0/228 VOLUME 12, 2024 spike-rate-dependent plasticity are adopted as weight-update rules [16], [17].Online learning can offer a robust SNN system by compensating for variations in synaptic devices during real-time learning.However, achieving high accuracy for complex datasets is still challenging [18], [19], [20], [21], [22].Therefore, offline learning utilizing backpropagation algorithms has been studied as an alternative training method [23], [24], [25].Offline learning is advantageous for achieving high accuracy for complex datasets because ANN weights trained using backpropagation algorithms are transferred to the SNN system [26].However, the offline learning method suffers from accuracy drop during the ANNto-SNN conversion due to the different neuron behaviors in ANN and SNN [27].While ANN neurons pass inputs through an activation function and immediately produce an output, SNN neurons based on integration and fire (I&F) behavior must accumulate inputs until the membrane voltage (V mem ) exceeds a membrane threshold voltage (V MTH ) to generate a spike to the next layer [28].Therefore, depending on the extent of V mem change, information loss in SNNs can occur in two forms: overflow and underflow loss.Overflow loss occurs when voltage exceeding V MTH is not utilized for subsequent spike generation.Conversely, underflow loss occurs when an increase in V mem is insufficient to reach V MTH , thereby preventing the propagation of information to the next layer [24], [29].Especially, in deep SNNs with many layers, such information loss can accumulate as spikes pass through the layers, leading to error and performance degradation [27], [30].Given that the extent of V mem change is influenced by various factors including the conductance range of synaptic devices, neuron spike characteristics, and membrane capacitor (C mem ), it is imperative to design hardware SNN systems based on the properties of synapse and neuron.
In this paper, we simulated a hardware SNN system consisting of IGZO-based synaptic transistors and I&F neuron circuits.IGZO synaptic transistors were fabricated and measured to model synaptic weights and device-todevice variation.In addition, we proposed and verified an I&F neuron circuit by HSPICE simulation using the RPI polysilicon model, whose parameters were extracted by fitting the characteristics of fabricated IGZO thin-film transistors (TFTs).Subsequently, SNN simulations were conducted using MATLAB to evaluate the performance on three datasets: MNIST, Fashion-MNIST, and CIFAR-10.As the V mem change is influenced by C mem , we investigated the impact of C mem on SNN performance and proposed a layerby-layer C mem optimization method to minimize information loss that occurs at each layer.

II. IGZO NEUROMORPHIC SYSTEM A. IGZO SYNAPTIC TRANSISTOR
The neuromorphic system, which mimics the mechanism of the human brain, is composed of synaptic devices and neuron circuits.To implement the synaptic device, we fabricated IGZO synaptic transistors consisting of the Al 2 O 3 /IGZO/Al 2 O 3 gate stack and IGZO channel [31], as illustrated in Fig. 1(a).First, the buffer oxide (SiO 2 ) on Si substrate was cleaned using ultrasonication.A 70-nm titanium (Ti) layer was deposited using e-beam evaporator, and dry etched to form the gate electrode.A 70-nm Al 2 O 3 blocking layer (BL) was deposited by atomic layer deposition (ALD) at 150 • C using the H 2 O reactant and trimethylaluminum (TMA) precursor.A 40-nm IGZO charge trap layer (CTL) was deposited by RF sputtering using the IGZO target (In:Ga:Zn = 1:1:1 at%).The CTL was patterned by photolithography and wet etched.A 7-nm Al 2 O 3 tunneling layer (TL) was deposited by ALD at 150 • C. A 40-nm IGZO channel was deposited by RF sputtering and defined by photolithography and wet etching.Then, 70-nm Ti drain/source were deposited by e-beam evaporation and formed by a lift-off process.Finally, the devices were annealed at 250 • C for 0.75 h in an air atmosphere.Each layer of the fabricated device was clearly defined with channel width/length of 15/0.9 µm, as shown in Fig. 1(b).All electrical measurements were performed using a semiconductor parameter analyzer (4200 SCS & 4225 PMU, Keithley) in a dark box at RT.
The synaptic transistor can modulate the threshold voltage (V TH ) by controlling the amount of charge in the CTL via Fowler-Nordheim tunneling between the channel and CTL.Under a positive gate bias (program), electrons from the channel can be trapped in the CTL.Conversely, under a negative gate bias (erase), electrons in the CTL are detrapped to the channel.To investigate the electrical behavior and uniformity of the synaptic transistor, we measured the transfer curves of 15 devices with a gate double sweep ranging from −20 V to 20 V, which exhibited clockwise hysteresis and uniform hysteresis window (Fig. 2(a)).As the representability of synaptic weights in the synaptic device is important for the neuromorphic system, we measured the synaptic weights of 15 devices using potentiation pulses.The condition of the potentiation pulse (−20 V, 200 µs) was chosen to achieve optimal linearity and dynamic range.There is potential to lower the voltage of the potentiation pulse by further optimizing the BL and TL in the synaptic transistor.Before the measurement, one depression pulse (20 V, 1 s) was applied to initialize the synaptic weight to the first level, and then a series of potentiation pulses were applied.All conductance values were read at V GS of 0 V

B. IGZO I&F NEURON CIRCUIT
We proposed an IGZO-based neuron circuit that emulates the I&F behavior widely used in SNNs [32].The circuit consists of n-type-only TFTs because IGZO is inherently n-type material.There are four inverters, one reset TFT (T1), three delay capacitors (C D ), and one membrane capacitor (C mem ), as shown in Fig. 4(a).The circuit operation was verified by HSPICE simulation using the IGZO TFT library based on the fabricated device's electrical properties.The IGZO TFT was fabricated using the same process as the IGZO synaptic transistor, except for the deposition of IGZO CTL.The mobility, V TH , and subthreshold swing of the IGZO TFT were 9.6 cm 2 /V•s, 1.9 V, and 220 mV/dec, respectively.We adjusted HSPICE RPI polysilicon model parameters based on the measured data to simulate the circuit operation accurately.The measured and simulated electrical characteristics of the IGZO TFT are shown in Fig. 3.All transistors are 0.9 µm in length.The widths of pull-up transistors (T2, T4, T6, and T8) and pull-down transistors (T3, T5, T7, and T9) in the inverters are 1 µm and 35 µm, respectively.The large width of 60 µm for T1 is used for the fast discharge of C mem .The values of V DD , V SS , and C D are 6 V, −0.2 V, and 500 fF, respectively.Fig. 4(b) shows the simulated V mem and V spike when the constant current spikes are periodically accumulated in C mem .The neuron circuit successfully generates a spike and resets C mem when V mem exceeds the V MTH of 2.45 V.The voltage and width of the spike are 3.8 V and 3.5 µs, respectively.The width of the spike can be modulated by changing C D .When a spike is generated, V mem is reset to zero voltage through T1.

C. SNN SYSTEM CONFIGURATION
For the SNN simulation, we utilized a network that consists of one input layer and three fully connected layers (256-128-10), as shown in Fig. 5(a).The size of the input layer is determined by the input image size of the dataset.The input image is converted to a series of spikes based on left-justified rate coding, where the number of spikes is proportional to the grayscale of the image [33].The number of time steps is determined by the maximum grayscale of the input image because each input neuron propagates one spike per time step.The network layer can be configured in hardware utilizing IGZO-based synaptic transistors and I&F neurons, as shown in Fig. 5(b).The spikes generated by presynaptic neurons are converted to currents through the synapses, and then these currents accumulate in the C mem of each postsynaptic neuron.It is assumed that each postsynaptic neuron is connected to both inhibitory and excitatory synapse lines to implement positive and negative weights.

III. RESULTS AND DISCUSSION
Fig. 6(a) shows the ANN training accuracy during 200 epochs for three datasets in the pre-defined software (Python).The training parameters were as follows: ReLU activation function, Adam optimization, a learning rate of 0.0005, a batch size of 400, and a dropout probability of 10%.The trained ANN weights were normalized and quantized based on the synaptic weight levels and then rescaled to fit within the conductance range of the synaptic weight.To prevent loss of accuracy, the positive and negative weights from ANN should be implemented identically in SNN, requiring the two synapse lines: excitatory and inhibitory synapse lines.Before the weight transfer, all synapses in the SNN system were initialized to the first weight level.During the weight transfer process, synapses in either inhibitory or excitatory lines were updated based on the sign of the weight.The weight is described as the following equation: G E and G I are the conductance of the excitatory and inhibitory synapses, respectively.Fig. 6(b) shows the change in weight distribution before and after weight transfer to synapses.During the SNN simulation, the V mem of a postsynaptic neuron is determined by the following equation: V mem (t) = V mem (t − 1) V mem (t) is the membrane voltage at time step t.N is the number of presynaptic neurons.Since the extent of V mem change is affected by C mem , selecting the optimal value of C mem is essential to enable proper information transfer across the layers.For example, if C mem is excessively large, it can result in underflow, which suppresses spike generation in the neurons and hinders information transfer to the output neurons.On the other hand, if C mem is too small, it can lead to overflow, causing V mem to significantly exceed V MTH .Since V mem is reset to 0 V regardless of how much it exceeds V MTH , any voltage exceeding V MTH is not utilized for the next spike generation.Furthermore, due to overflow, spikes can be erroneously generated in neurons where spikes should not occur.For these reasons, optimization of C mem is crucial to minimize the accuracy drop that occurs during the ANNto-SNN conversion.Therefore, we first examined the impact of C mem on SNN accuracy and attempted to find optimal C mem in our system, assuming that neurons in all layers have identical C mem .For each dataset, SNN simulations were performed over a range of C mem from 0.5 pF to 15 pF, as shown in Fig. 7.For the MNIST dataset, the accuracy drop is negligible for most C mem values.SNN accuracy even exceeds ANN accuracy for C mem of 7 pF, where the SNN and ANN accuracies are 98.42% and 98.37%, respectively.The similar accuracy between ANN and SNN might be due to the reason that it is a simple dataset.However, for the Fashion-MNIST dataset, SNN accuracy starts to decrease when C mem exceeds 10 pF, indicating that the dataset is sensitive to information loss caused by underflow.The system shows the highest SNN accuracy of 89.12% at C mem of 6 pF, which is a 0.78% drop compared to ANN accuracy.In the case of the CIFAR-10 dataset, SNN accuracy is more sensitively affected by C mem , and the difference in accuracy between ANN and SNN is relatively larger compared to other datasets.Specifically, the SNN accuracy decreases to 38.03% when C mem is 0.5 pF and further decreases to 18.31% when C mem is 15 pF.Although the system can achieve its highest SNN accuracy of 43.76% at C mem of 5.5 pF, there is still a large accuracy drop of 9.34% compared to ANN accuracy.The more complex dataset shows greater sensitivity to C mem , accompanied by larger accuracy drop during the ANN-to-SNN conversion.
To investigate the reason for the large accuracy drop in the CIFAR-10 dataset, we analyzed the V mem change in the neuron that actively generates the spikes during the image classification.The V mem irregularly fluctuates according to the time step because the incoming synapse current varies depending on the spikes generated by presynaptic neurons.The neuron exhibits substantial overflow for most time steps, with V mem even reaching about twice V MTH at certain time steps, as shown in Fig. 8(a).In addition, it can be observed that neurons in layer 1 generate more spikes compared to other layers, as shown in Fig. 8(b).Thus, the first layer in SNN is more likely to experience overflow than other layers because it directly receives spikes from the input image, which exhibits the most active spike generation.However, the previous C mem optimization method that uses identical C mem for all layers has a limitation in that it does not consider the different degrees of overflow or underflow in each layer.
To overcome this limitation, we introduced a layerby-layer C mem optimization method that adjusts C mem individually for each layer.Fig. 9 shows the SNN accuracy for three datasets as a function of C mem for each layer.We narrowed the adjustment range of C mem from the first layer to the third layer (16 pF-8 pF-6 pF), considering the decrease in the number of generated spikes as the layer becomes deeper.Regarding the MNIST dataset, SNN accuracy remains nearly identical regardless of the change in each layer's C mem .When using C mem of 16/6/3 pF for layers 1/2/3, the system can achieve the highest SNN accuracy of 98.47%.For the Fashion-MNIST dataset, using large C mem for both layers 2 and 3 (>4 pF) leads to accuracy drop due to underflow, and the accuracy drop becomes more severe as C mem of layer 1 increases.The accuracy decreases up to 85.38% when using C mem of 16/8/6 pF for layers 1/2/3.Therefore, it is crucial to select appropriate C mem for layers 2 and 3 to prevent underflow because these layers receive fewer spikes compared to layer 1.When using C mem of 8/2/2 pF for layers 1/2/3, the system can achieve the highest SNN accuracy of 89.76%, which is close to ANN accuracy.For the CIFAR-10 dataset, increasing C mem of layer 1 leads to improved accuracy, possibly due to the suppression of overflow.It is noted that accuracy starts to decrease when the C mem of both layers 2 and 3 exceeds 4 pF.Therefore, configuring smaller C mem as the layers become deeper is beneficial in minimizing the accuracy drop.Notably, setting C mem as 16/8/1 pF for layers 1/2/3 results in the highest SNN accuracy of 48.06%, which represents a 4.3% improvement compared to the identical C mem configuration of 5.5 pF.
To analyze the effect of layer-by-layer optimization on the CIFAR-10 dataset, we quantitatively compared the extent of overflow occurring in all neurons within each layer.The metric for comparison was calculated using the following equations: M is the number of input images used for inference.N is the total number of neurons in a layer.T is the total number of time steps.Overflow total represents the cumulative overflow per neuron through all time steps when using M input images.The values of Overflow total were calculated according to C mem configuration using 100 input images in the CIFAR-10 dataset, as shown in Fig. 10.When using the identical C mem (5.5 pF), layer 1 exhibits the highest Overflow total of 38.42, while layers 2 and 3 show much lower Overflow total of 1.06 and 0.12, respectively, indicating that substantial amount of overflow still occurs in layer 1.
As expected from the neuron behavior in Fig. 8(a), using identical C mem for all layers is not enough to mitigate the overflow of layer 1, which receives the highest number of spikes among the three layers.In contrast, with optimized C mem (16/8/1 pF), the Overflow total of layers 1 and 2 decreased to 6.34 and 0.17, respectively, which is attributable to the increased C mem of layers 1 and 2. For layer 3, employing C mem of 1 pF is advantageous for minimizing information loss because increasing C mem of both layers 1 and 2 diminishes spike propagation to layer 3, which could result in underflow.These results indicate that layer-by-layer C mem optimization can reduce information loss by adjusting C mem in a way that minimizes either overflow or underflow in each layer.Fig. 11 represents the result of SNN accuracy according to the time step between identical and optimized C mem for three datasets.It is noteworthy that the SNN system can achieve ANN close accuracy for the MNIST and Fashion-MNIST datasets using optimized C mem .In addition, an improvement in accuracy can be observed for the CIFAR-10 dataset after time step 150.Table 1 summarizes the SNN accuracy  according to the C mem configuration for three datasets.Using C mem of 16/8/1 pF for layers 1/2/3 not only reduces the accuracy drop for the CIFAR-10 dataset but also maintains ANN close accuracy for other datasets.This result suggests that solely one C mem configuration can achieve high SNN accuracy for three datasets without the need for different C mem configurations for each dataset.We expect that C mem optimization, aimed at reducing information loss, will be more critical in deeper networks to minimize accuracy drop.
Finally, we investigated the impact of the weight variation on SNN accuracy, where the variation can be attributed to the device-to-device variation in synaptic transistors.The standard variation at each weight level of the synaptic transistor was extracted, and random Gaussian variation was applied to all synapses in SNN.The simulations were conducted 20 times for each dataset.Compared to the accuracy without variation, accuracy with variation for MNIST, Fashion-MNIST, and CIFAR-10 decreased by 0.11%, 1.02%, and 1.2%, respectively, as shown in Fig. 12.Despite the weight variation, the SNN system can operate well with little performance degradation.

IV. CONCLUSION
In summary, we performed a simulation of the IGZO-based neuromorphic system for SNN with MNIST, Fashion-MNIST, and CIFAR-10 datasets.The simulation utilized an offline learning method and models based on the fabricated IGZO synaptic transistors and simulated I&F neuron circuit.Since information loss can occur due to overflow or underflow depending on C mem , we optimized the C mem of neurons to minimize the accuracy drop using identical C mem for all layers.However, simulation results showed a significant accuracy drop (9.34%) for the CIFAR-10 dataset, primarily due to substantial overflow in layer 1.To address this issue, we introduced a layer-by-layer C mem optimization method, which adopts different C mem for each layer, considering the different degrees of overflow or underflow in each layer.This approach reduced the accuracy drop for the CIFAR-10 dataset from 9.54% to 5.04% and made ANN close accuracy for the MNIST and Fashion-MNIST datasets, suggesting that effective reduction of information loss is possible by layer-by-layer C mem optimization.Further, we investigated the impact of synaptic weight variation, and our SNN system maintained high accuracy with little accuracy degradation (less than 2%).Based on the results, we expect that the IGZO-based SNN system could be a promising candidate for high-performance neuromorphic systems.

FIGURE 3 .
FIGURE 3. Measured and simulated (a) transfer and (b) output curves of the IGZO TFT.

FIGURE 4 .
FIGURE 4. (a) Schematic of the proposed I&F neuron circuit based on IGZO TFTs.(b) Simulated output characteristic of the I&F neuron circuit.

FIGURE 5 .
FIGURE 5. (a) Schematic of spiking neural network with three fully connected layers.(b) Hardware implementation of the SNN using IGZO-based synaptic transistors and I&F neurons.

FIGURE 6 .
FIGURE 6.(a) ANN training accuracy of each dataset according to epoch.(b) Weight distribution change before and after synapse transfer.

FIGURE 7 .
FIGURE 7. SNN accuracy results according to C mem for three datasets (identical C mem ).

FIGURE 8 .
FIGURE 8. (a) V mem according to the time step for the 55th neuron in layer 1 (C mem = 5.5 pF).(b) The number of generated spikes during 255 time steps for all neurons when a CIFAR-10 input image is used.

FIGURE 10 .FIGURE 11 .
FIGURE 10.Calculated Overflow total for three layers with respect to C mem configuration.

FIGURE 12 .
FIGURE 12. SNN accuracy for three datasets under device-to-device variation of IGZO synaptic transistors (Solid line indicates the SNN accuracy without the variation).