Neuron Circuits for Low-Power Spiking Neural Networks Using Time-To-First-Spike Encoding

Hardware-based Spiking Neural Networks (SNNs) are regarded as promising candidates for the cognitive computing system due to its low power consumption and highly parallel operation. In this paper, we train the SNN in which the firing time carries information using temporal backpropagation. The temporally encoded SNN with 512 hidden neurons achieved an accuracy of 96.90% for the MNIST test set. Furthermore, the effect of the device variation on the accuracy in temporally encoded SNN is investigated and compared with that of the rate-encoded network. In a hardware configuration of our SNN, NOR-type analog memory having an asymmetric floating gate is used as a synaptic device. In addition, we propose a neuron circuit including a refractory period generator for temporally encoded SNN. The performance of the 2-layer neural network composed of synapses and proposed neurons is evaluated through circuit simulation using SPICE based on the BSIM3v3 model with $0.35~\mu \text{m}$ technology. The network with 128 hidden neurons achieved an accuracy of 94.9%, a 0.1% reduction compared to that of the system simulation of the MNIST dataset. Finally, each block’s latency and power consumption constituting the temporal network is analyzed and compared with those of the rate-encoded network depending on the total time step. Assuming that the network has 256 total time steps, the temporal network consumes 15.12 times less power than the rate-encoded network and makes decisions 5.68 times faster.


Introduction
Recently, artificial neural networks (ANNs) have shown remarkable results surpassing humans in some tasks such as pattern recognition, object detection, or natural language processing [1][2][3][4][5].The success of ANN has attributed to the multi-layered structure inspired by the nervous system and computing the complex nonlinear transformations [6,7].Conventional ANN, however, has fundamentally different structures from the human brain in that the time has no effect on the propagation of data and uses analog-valued neurons [8].Also, software-based ANNs are far from real-time and low power processing, making computing on the edge devices is challenging.In this perspective, SNNs using analog synaptic devices are regarded as a greatly competent network.In SNN, data propagates in the form of short spikes as in biological neural system [9,10].Such short pulses perform a read operation on each synaptic device, and the total current flowing in the array is integrated into the analog neuron by Kirchhoff's rule, allowing high-performance parallel computation such as Vector-by-Matrix multiplication (VMM).
There are several methods to encode the input data of multiple resolutions into the input pulse train of SNN.Commonly, the rate of pulses can be proportional to the intensity of the input data.In the rate-encoded network, the I&F behavior of the neuron is almost matched to the ReLU activation function [11].Therefore, the weights trained by ANNs can be used directly in SNNs, and these networks have shown great performance on the complex benchmark such as CIFAR [12] or ImageNet [11,13].However, encoding an analog input value in the form of a firing rate requires a large number of spikes to express the intensity of one input data.The rate-encoding method needs to be improved for efficient computing on the edge devices in terms of power consumption and latency.
Another candidate for encoding method is temporal encoding, where the input data is transformed to the firing time of the input spikes [14].Temporally encoded networks can generate a single spike at most regardless of the intensity of the data so that the firing of the neuron can be sparse.There have been several efforts to train the networks encoded by the firing time of spikes.However, many works used complex synaptic functions, which create an additional burden when implemented in hardware [15][16][17][18].Also, the system of some other works is not power-efficient due to its long duration of input pulses, not a spike [19].
In this paper, we configure SNN at the circuit level, where information is carried as the firing time of a single spike by adopting the temporal encoding method.First, by using temporal backpropagation algorithm [20], we evaluate the performance of SNN at a system-level on MNIST data sets and investigate the non-ideal issues that can occur in a hardware implementation.Afterward, we propose neuron circuit blocks to generate a refractory period for a single spike-SNN.By combining proposed neuron circuits with the synaptic device reported from our previous work, the full network is simulated at a circuit level using HSPICE.Finally, the power consumed by each block and the latency of the network are analyzed and compared with that of a rateencoded network with the same size.

Forward Propagation
In this network, the input information of the SNN is encoded using the time-to-first-spike (TTFS) method.As shown in eq. ( 1), the firing time of input neurons are inversely proportional to the input value (  ) of each individual neuron [20][21][22][23][24].The more salient features spike earlier, and the less input value corresponds to a late spike.Input neurons corresponding to data 0 fires at the last time step.We assume that each neuron can generate at most a single spike per each image because the fired neuron enters the refractory period and no longer responds to the image.
where    () denotes the cumulative input function of th neuron at time t, which is a parameter indicating whether the neuron is a fired state at time t.As shown in eq. ( 5), the membrane voltage of output neuron j can be calculated by multiplying cumulative input and weights:

Backward Propagation
First, note that the backward propagation process is a modified form of the algorithm of the previous work [20].In TTFS network, the output value of neuron k is expressed as the firing time (   ).Accordingly, we defined the error function as defined in eq. ( 6) and (7), so that the output neuron can fire as close as possible to the target firing time of each neuron ( , ).: In order to compute the change of the weight through backpropagation, the derivative of the firing time respect to its membrane potential is required.It is not clear to calculate this value corresponding to the activation function of a typical ANN.Instead, we conducted Monte Carlo simulation using various images of MNIST data sets to model the relationship as shown in Figure 1 (b).The x-axis of Fig. 1 (b) represents the value of the weighted sum, which is the product of input and weights.The y-axis denotes the firing time (   ) of the neuron, which is the time when the value of x-axis exceeds  ℎ  .The result of simulation in multiple MNIST data sets has shown that the derivative can be approximated to -1 if the firing time is not 0 or   .On the other hand, if the   does not exceed the neuron threshold (   =   ) or reach too quickly (   = 0), the firing time does not change with   , so the derivatives are assumed to be zero.This is the main difference in training algorithms between previous studies [20] and our work.
The weights are updated by eq. ( 5) and (8) as follows: Before updating weights, we normalized delta values in each layer to prevent from vanishing gradients in deep layers.
We also set the target firing time of output neurons based on the previous work [20].We defined  as the minimum value of the firing time among the output neurons.Correct output neuron is encouraged to fire first among the output neurons at time  , and output neurons fired wrongly around  have a higher risk of responding incorrectly, so that penalizing as   .Then, we set the target firing time of th output neurons as:

NOR-type Synaptic Device Having Asymmetric Floating Gate
On the other hand, various types of emerging memory are being reported as candidates for artificial synaptic devices, which is a key element for configuring SNN in hardware.In our previous work, NOR-type flash memory device was fabricated using a conventional CMOS process [25].As shown in Fig. 2 (a), this TFT type device has a poly channel and a half-covered poly-Si floating gate (FG) that functions as a charge storage layer.The thickness of blocking SiO2, FG, and tunneling SiO2 are 15 nm, 80 nm, and 7 nm, respectively, and the channel length (the length between source and drain) and the width are 0.5 μm each.Input pulses are presented to each gate (WL), and the currents of the synaptic devices are summed in the common drain line (BL).Hence, the output of vector-by-matrix multiplication (VMM) can be expressed as the current of each post-neuron.In addition, since the current is controlled by three terminals in this FET-type synaptic device, it is more resistant to sneak path issues [26,27] or off-current issues [28,29] than two-terminal devices such as RRAM.

Results
In this section, we designed a 2-layer fully-connected SNN consisting of NOR-type synaptic devices and neuron circuits.Then, the performance of the network is verified in both the system-level and the circuit-level simulations.

Performance of SNN on MNIST
The system-level simulation is performed on the MNIST datasets to evaluate the performance of SNN based on the NOR-type synaptic devices.To reduce the size of the network, the edge of the image is removed and resized to 20  20.The parameters used in the simulation are shown in Table 1.The input data is transformed into a TTFS spike train over 64 time steps, and the   mentioned in section 2.1.3is set to 1.The batch size of training is 1, and the initial learning rate is set to 0.02, but it gradually drops as training continues.The threshold of neurons in all layers is 1.6 V, and the winner neuron of SNN is determined as the neuron that fires first among neurons in the last readout-layer.However, if any output neuron does not spike until the last time step, it is evaluated by considering the membrane voltage of the output neuron at the last time step.The weights are initialized using the initialization method proposed by K.He [30].The initial weight distribution is given by: where    represents the number of input nodes in the th layer.However, by changing the mean of the normal distribution to a positive value (  ) rather than 0, many hidden neurons are fired at the start of training, leading to participation in data propagation.
Weights trained by the off-chip learning rules described in section 2.1 are transferred to the HNN.Prior to being transferred, the weights are normalized and quantized to the 101 states.The weight for one synapse in the SNN is represented by the conductance difference between two synaptic devices as follows: where    is one of the measured conductance value of the asymmetric FG synaptic device ((1), (2), . .(50)).For all the positive    ,   (−) is set to (1) (  ), and conversely, for all the negative    , If the    is 0, the conductance of both synaptic devices is set to (1) (  ) [31].The quantized target weights of each synapse can be transferred by applying the corresponding index number of pulses to devices in the synaptic array [32].
Fig. 3 (a) presents MNIST accuracy as a parameter of the width of the hidden layer in 2-layer SNN.The accuracy for 60,000 training sets and 10,000 test sets not used for training is indicated by dotted and solid lines, respectively.As the number of hidden neurons increased, it is observed that the accuracy increased.The accuracy of the network (400 -512 -10) is 99.21% for the training set, and 96.90% for the test set.The accuracy shows the degradation of about 1% compared to those of rate-encoded networks of similar size [33,34].Fig. 3 (b) shows the recognition accuracy with the total number of time steps per image.The time steps, representing the resolution of the image, can be reduced to 8 without significant degradation of the accuracy (0.15% degradation for 512 hidden neurons).

Effects on Accuracy by Variation in Hardware Implementation
Changes in device characteristics caused by process variations during manufacturing negatively affect the operation of synaptic devices and neuronal circuits, which reduces the recognition accuracy of the SNN implemented in hardware.Several types of variation have been analysed in previous studies [35][36][37][38][39][40].We classify the four major variations as follows: 1) device-to-device variation in the synaptic array [35,36], 2) firing threshold variation in the neuron circuits [37], 3) stuck-at-off variation in the synaptic array [38][39][40], and 4) stuck-at-off variation in the neuron circuits.
Note that our network does not take into account the pulse-to-pulse variation considered in many previous studies [24,41] since the weights obtained through off-chip learning are transferred once to the synaptic devices in the array.The recognition accuracy with the variation of device characteristics is compared with that of conventional rate-encoded networks with the same size.In the rate-encoded network, the number of input spikes follows a Poisson distribution, and the weights are also quantized with the same resolution in the same manner as in the TTFS network.
We first mathematically model the device-to-device variation in the synaptic array as follows: where   denotes the overall quantized weights, and (1,  ℎ 2 ) stands for the normal distribution with mean 1 and standard deviation  ℎ .Also, weights with a value of 0 are set to random numbers with normal distribution N(0,  ℎ ).In addition, variation of neuron thresholds is also modeled by normal distribution: However, negative neuron thresholds are very difficult to implement in hardware, so they follow a clipped normal distribution.Lastly, in large-sized synapse and neuron arrays, a stuck-at-off fault where one of the devices is not working should be considered.Dead synaptic devices can cause the current to not flow even the input pulse is applied, which results in a fatal error in the weighted sum.Further, if the neuron block dies, there may be cases where the current cannot be integrated into the capacitor, or the spike cannot be emitted even if the membrane voltage exceeds the neuron threshold.We defined the stuck-at-off ratio as the number of dormant synaptic devices (neurons) relative to the total number of devices (neurons) in the array and named it   (  ).The conductance of the dead synapse is assumed to 0, and the input by a dead neuron is assumed to 0 regardless of the membrane voltage.Compared to the rate-encoded network, TTFS network is vulnerable to the variation since a single neuron can contribute only one spike in the inference process.In TTFS network, even if only one spike disappears (or even one false spike occurs), it makes a big error in the overall weighted sum.In particular, it is observed that the network with a small number of hidden neurons shows severe degradation of accuracy due to the importance of each neuron.Therefore, synaptic arrays of TTFS networks should be finely controlled so that the variation to be minimized.For example, device-to-device variation can be reduced by precision tuning using the read-write-verify scheme in the weight transfer process [42].

Circuit Level Simulation
In the previous section, we verified that data encoded by TTFS method can be effectively trained through temporal backpropagation.However, non-ideal issues that have not been considered at system-level simulations, such as parasitic resistance or capacitance, can be a problem in large-sized arrays.In this section, we propose the neuron blocks suitable for TTFS encoded SNN, and simulate fully connected HNN at circuitlevel using SPICE.Through this, we will investigate the result of the system-level simulation is reasonable, and how much power the full network consumes in the inference process.

SNN Architecture
Fig. 5 (a) depicts a schematic diagram of fully connected HNN.The entire network is composed of the synaptic array and neuron circuits.The NOR-type synaptic device is modeled with a voltage-controlled current source (VCCS) behavioral model with 3 terminals (Gate, Drain, Source) as shown in Fig. 5 (b) [43,44].
The current between the drain and the source is determined by the voltage difference between the gate and the source.As shown in Fig. 2 (b), the behavior modeling of the synaptic device was based on the results measured by increasing the gate voltage from 0 V to 3 V in 60 mV steps.
In addition, neuron circuits consist of a current mirror, an I&F block, and a refractory period generator.Fig. 5 (b) depicts a modeled synaptic array and a current mirror designed for summing and subtracting currents.A single wordline (WL) corresponds to an input, and the weighted sum of 400 inputs is expressed as the sum of current flowing through the bitline (BL).Currents flowing in the positive and negative synaptic arrays are copied through the respective current mirrors so that the net charge is integrated in the membrane capacitor (  ).Before the input pulse is presented,   of all neurons are initialized to  2 by   .If the initial membrane voltage of the neuron is not  2 (e.g.0 V), the negative charge created by the inputs in the early stage of the time domain cannot be integrated into the capacitor, so the final result of the weighted sum can be distorted.Since SNN encoded by TTFS method assumes that one neuron spikes at most once, the neurons already fired should enter the refractory period so that no more spikes are generated.To implement this, the output neuron that has already fired keeps   in a high state until the corresponding input ends.This causes M7 and M8 to turn off so that no more current flows through the synaptic array.Fig. 5 (c) shows the circuit of the refractory period generator (RPG).The block for generating   is based on the structure of the latch.Before the input pulse is presented, M12 is turned on by   , so the output node of RPG is initialized to the ground state.Then, as soon as the I&F block fires, M10 and M11 turn on and   goes to high state, which is maintained until a new input data is presented.Fig. 5 (d) represents the I&F block constituting the neuron [45].If the membrane voltage exceeds the Vth of M14 by the integrated charge, node 1 in the high state changes to the low state.This brings node 2 to the high state.After the delay time by   , the voltage at node 2 turns M21 on and puts node 1 back in high state, and returns node 2 to original state.The W/L ratio of M16 acting as a resistor and the value of   determine the width of a spike generated in the output node.In addition, the W/L ratio of M14 and M21 determines the voltage of node 1, so it affects the threshold of the neuron.After I&F block fires,   turns M13 on to keep   as the ground state.

Performance in Circuit-Level Simulation
Among the networks simulated in the previous section 3.1, a relatively light network, the 400-128-10 sized network is simulated using a circuit simulator (HSPICE) with a predictive technology models (PTMs).Circuitlevel simulation was performed with 0.35 μm CMOS technology, and the parameters of the components in circuits are shown in Table 2.
Fig. 6 provides the waveforms of some nodes in the process of inferencing MNIST data sets.Before the input pulses are presented,   is first presented to initialize the membrane capacitors in I&F block and RPG block.After that, as shown in Fig. 6 (a), all inputs are transformed into time-to-first spike pulses with a duration of 0.5 μs over 8 time steps.The interval between each time step of the input pulse is also set to 0.5 μs.The rising and falling times of input pulses are each set to 0.1 μs.Fig. 6 (b) shows transient waveforms of some nodes in hidden neurons.The currents flowing through the synaptic array by the input pulses are integrated into the capacitor of the hidden neurons, and when   exceeds the threshold of the neuron, corresponding neuron fires and presents a spike with a width of 0.5 μs to the post-layer.At the very moment the neuron fires,   generated by each RPG prevents further integration of charge into the fired neuron.Finally, Fig. 6 (c) represents the membrane voltage of neurons in the output layer.As in the system simulation, the class of the earliest fired output neuron is the result predicted by SNN.However, in some rare cases when no output neuron fires, the neuron with the highest membrane voltage is considered as the winner neuron.Fig. 7 compares the results of system-level and circuit-level simulations.The size of the network is 400 -128 -10, and the number of time step for each image is set to 8. Fig. 7 (a)-(c) shows the firing times of input, hidden and output layers in the system-level simulation of one image.The x-axis of the three raster plots represents the time and the y-axis stands for the index number of neurons in each layer.Fig. 7 (e) and (f) depict a raster plot of spike timing in hidden neurons and   of output neurons in the circuit-level simulation for the same image.By comparing the firing times of hidden neuron and output neuron shown in (b), (e) and (c), (f), it is observed that the results of both simulations are similar.
In addition, we also simulated the circuits for 1000 randomly selected MNIST data sets.Fig. 7 (d) shows the result of comparing the firing time of the winner neuron obtained by simulations at system-level (x-axis) and circuit-level (y-axis).Since the system-level simulation was performed during 8 discrete time steps, the firing time in the system-level is a discrete value.The firing times of the two simulations are not perfectly matched, but they show almost the same tendency, which means the proposed SNN shown at the circuit level works pretty much like that at the system level.Indeed, the proposed SNN has reached 94.9% accuracy for networks having 128 hidden neurons at the circuit-level.This accuracy is only 0.1% lower than the 95.0% accuracy in a system level simulation.The reason for the slight decrease in accuracy is that the off current in the synaptic array is not considered at the system-level.Also, calculating the weighted sum through discrete time steps in system-level simulation can cause a difference from actual circuit operation.

Power Measurements
In this section, we estimate the power consumed by the TTFS network at the circuit-level and compare the results with that of the rate-encoded network.The biggest advantage of TTFS encoding method is that it requires much fewer pulses compared to the conventional rate encoding method, which results in lower power consumption.TTFS and rate-encoded networks, each with the same number (128) of hidden neurons are simulated for 100 randomly picked MNIST data sets, and the total time steps for each image are set to 8. Fig. 8 shows the amount of power consumed by each block in the proposed SNN.As categorized in section 3.2.1, the entire network consists of synapse array (SA) and neuron circuits, and specifically, the neuron is composed of a current mirror (CM), a circuit for integrate and fire (IF), and a refractory period generator (RPG).Fig. 8 (a) depicts the power consumed in the inference process of TTFS network.The entire network consumes 353.6 μW, and it is observed that most (~ 90%) of the power is consumed by the components in the 1st layer.In particular, I&F block accounts for a remarkable proportion of power consumption.This is because not only power is consumed to generate the pulse, but also subthreshold leakage current flows due to the membrane voltage of the neuron below  ℎ  .In the I&F block depicted in Fig. 5 (d), even if   does not exceed  ℎ , M14 can be finely turned on if   is a positive value.This creates a leakage path through M14 and M15, allowing current to flow even the neuron is not fired.Since the number of spikes in TTFS network is small, this standby power occupies a relatively large portion as much as the power required to generate spikes.Improving the structure of I&F circuits to deal with this issue can be a topic for further study.In the circuit-level simulation, each input spike of the rate-encoded network is filled from the last time steps [46].Compared to the TTFS encoding method, the rate-encoding method requires much more pulses to represent an image, so the currents in the synapse array and current mirror are very large.Likewise, the number of spikes generating in each layer is greater, so the power consumed by I&F block is larger.Unlike TTFS network, the rate-encoded network does not require a refractory period generator, but the power that can be saved is very small (~2%).It is obtained that the entire network consumes 1240 μW of power, which is 3.5 times more than that of the TTFS network.
The power consumption ratio of the two networks increases as the total time steps per each image increases.Fig. 9 (a) shows the required number of spikes and consumed energy as a function of time step.The solid line represents the average value of the spike numbers required to compute an image at the system-level.The required spike number is the sum of spikes generated in all layers.The number of pulses in the TTFS network is only counted until the winner neuron of the output layer fires, and the number in the rate-encoded network is counted until the final time step.When the input is converted to a spike rate, the number of spikes required to express the same values increases as the time step increases.On average, if the total time steps are 4, only 49.5 spikes are required, whereas 30793 spikes are needed when the time step reaches 256.On the other hand, the number of spikes in TTFS encoding method is nearly constant at about 162 regardless of the total time steps.Therefore, as the resolution of input data increases, the difference between the required spike numbers of the two networks increases.
Meanwhile, the dashed lines represent the average energy required to compute an image as a result of circuit-level simulation.Since the time required to compute the image depends on the time step, the energy is compared between two networks.On average, the rate-encoded network consumes 5.65 nJ of energy per image at a time step number of 4 and 372 nJ at a time step number of 256.On the other hand, TTFS network consumes 2.16 nJ at a time step number of 4 and 24.6 nJ at a time step number of 256 to compute one image.As the total time step of TTFS network grows, the number of spikes is not changed, but the consumed energy is increased.This is because the amount of energy consumed by the leakage path in I&F block is proportional to the time for processing an image.Rate-encoded networks are also affected by this leakage, but the relative proportion of the leakage in total energy consumption is less than that in TTFS network due to a large number of spikes.Hence, the consumed energy of the rate encoded network is almost proportional to the required spike number.Meanwhile, TTFS, network uses a small number of spikes, tends to increase the consumed energy even if the required spike number is constant.Nevertheless, the superiority of TTFS network in terms of power-efficiency is increased as the time step increases compared to the rate encoded network.Finally, the ratio of power efficiency of TTFS networks to rate-encoded networks reaches 15.75 at a time step number of 256.
The TTFS network also has an advantage in terms of the latency, the time it takes to infer the answer.Fig. 9 (b) shows the latency of the TTFS network and rate-encoded network as a parameter of the number of total time steps.The latency of the TTFS network was calculated as the average value of the time until the emission of the first spike at the output layer.On the other hand, the rate-encoded network can only make a decision at the last time step, so the latency of the network is equal to the value of the total time step, as indicated by the dashed line.It is observed that the TTFS network with 128 hidden neurons can make a decision about 5 times faster than rate-encoded network of the same size.

Conclusion
In this study, we have evaluated the performance of the SNN consisting of NOR-type asymmetric FG synaptic devices and neuron circuits at system-level and circuit-level.Input data was encoded as the time of the input spikes (time-to-first spike: TTFS), and the network was trained by temporal backpropagation which is a learning method suitable for networks applying TTFS encoding method.The neural network with 512 hidden neurons showed a competitive accuracy of 96.90 % for the cropped MNIST data sets.We also investigated the impact of the non-ideal characteristics of the synaptic array and neuron circuits on accuracy.These results can be a guideline that informs which level of the variation is allowed in TTFS network.In addition, we proposed a neuron circuit for inferencing temporal data and modeled the synapse device to demonstrate the operation of the full network.Simulating an SNN with 128 hidden neurons in SPICE gives 94.9% accuracy for 1000 MNIST data sets, which is almost no degradation compared to the system-level simulation.We also analyzed the power consumed in the inference process by each block in SNN.When using

Fig. 1 (
Fig. 1 (a) depicts a schematic diagram of SNN encoded by TTFS method.The firing time of the th neuron in the th layer is defined as    .Input neurons emit a single spike only at their own firing time.   () represents the input spikes generated by the th neuron of th layer at time t in the form of voltage pulse.Input pulses are multiplied by weights and integrated by the non-leaky IF model.As expressed in eq.(3), when the membrane voltage ( ,

Fig. 2 (
Fig.2(b) provides the measured   -  curves of the NOR-type flash memory device.50 repeated erase pulses (  = −3 V,   = 5 V, duration = 100 μs) are applied, and the threshold voltage of the device decreases.The inset of Fig.2 (b)shows the change of conductance when the device is under the read conditions.The behavior of these synaptic devices is similar to the long-term potentiation (LTP) of the synapse in the nervous systems.

Fig. 4 (
Fig. 4 (a) -(d) show the degradation of the accuracy as the variation in the synaptic array gets worse.Compared to the rate-encoded network, TTFS network is vulnerable to the variation since a single neuron can contribute only one spike in the inference process.In TTFS network, even if only one spike disappears (or even one false spike occurs), it makes a big error in the overall weighted sum.In particular, it is observed that the network with a small number of hidden neurons shows severe degradation of accuracy due to the importance of each neuron.Therefore, synaptic arrays of TTFS networks should be finely controlled so that the variation to be minimized.For example, device-to-device variation can be reduced by precision tuning using the read-write-verify scheme in the weight transfer process[42].

Fig. 8 (
Fig.8(b) represents the power consumed in the rate-encoded networks.In the circuit-level simulation, each input spike of the rate-encoded network is filled from the last time steps[46].Compared to the TTFS encoding method, the rate-encoding method requires much more pulses to represent an image, so the currents in the synapse array and current mirror are very large.Likewise, the number of spikes generating in each layer is greater, so the power consumed by I&F block is larger.Unlike TTFS network, the rate-encoded network does not require a refractory period generator, but the power that can be saved is very small (~2%).It is obtained that the entire network consumes 1240 μW of power, which is 3.5 times more than that of the TTFS network.

Figure 1 .Figure 2 .
Figure 1.(a) Schematic illustration of a fully connected multi-layer neural network.   and  ,  represents the firing time and the membrane voltage of th neuron of th layer, respectively.  −1 is the weights between th neuron of (-1)th layer and th neuron of th layer.(b) The relationship between the firing time and the membrane potential of the neuron through Monte Carlo simulation.If the firing time of a neuron is not   or   , the firing time tends to decrease as the membrane voltage (weighted sum) integrated into the neuron increases.

Figure 4 .Fig. 5 (
Figure 4. Evaluation of the (Left) TTFS and (Right) rate-encoded network as a parameter of (a) device-todevice variation in the synaptic array, (b) firing threshold variation in the neuron circuits, (c) the stuck-at-off ratio in the synaptic array and (d) neuron circuits.

Figure 6 .
Figure 6.(a) Pulses fed into the input neuron shown in the time domain.(b) (Top) Evolution of the membrane voltage of hidden neurons.(Middle) Generated output pulse and (Bottom) refractory period by the neurons in hidden layer.(c) Evolution of the membrane voltage of output neurons.The answer predicted by the network is the class of output neuron 1.All results are simulated at circuit-level.

Figure 7 .Figure 8 .
Figure 7. Raster plots of the spike timing of the (a) input, (b) hidden, and (c) output neuron in the 2-layer (400-128-10) SNN for randomly selected test data '7'.The x-axis represents the time in the system simulation, and the y-axis represents the index of each neuron.(d) Comparison of the firing time of winner neuron in the system-level (x-axis) and circuit-level (y-axis) simulation.(e) Raster plots of the hidden neuron when simulated in the circuit-level for the same network size and data as (b).(f) Evolution of the membrane voltage of output neurons in the circuit-level simulation.

Figure 9 .
Figure 9. (a) Comparison between TTFS and rate-encoded network in terms of spike number (system-level) and consumed energy per image (circuit-level).Consumed energy was measured at various time steps on 10 randomly selected MNIST test sets.The simulation was conducted at various time steps.(b) The latency of TTFS network to make a decision.The dotted line indicates the total time step which is the time required for the rate encoded network to predict the answer.

Table 1 .
Parameters used in the system-level simulation Figure 3. (a) Training curves of the TTFS network as a parameter of the number of neurons in the hidden layer.The dotted line represents the accuracy for training sets, and the solid line represents the accuracy for test sets.(b) The recognition rate of the TTFS network as a parameter of the total time steps.