A New Deep Spiking Architecture for Reconstruction of Compressed Data in Cognitive Radio Networks

Cognitive Radio (CR) offers a spectrum sharing solution to handle the massive amount of devices operating in the same spectrum. In this work a sub-Nyquist compressive sensing technique is proposed that allows secondary users to sense and utilize idle spectrum. Reconstruction of compressed sparse data is achieved through a dual stage sophisticated reconstruction algorithm. The reconstruction uses a classical fast Orthogonal Matching Persuit (OMP), followed by a new spiking deep Residual neural Network (ResNet) architecture. The proposed architecture is obtained through a novel distributed conversion technique that is proposed to convert deep architectures to a spiking neural networks. The reconstructed data is compared in terms of Peak Signal-to-Noise Ratio (PSNR), Mean Square Error (MSE) and Structural Similarity (SSIM) to the compressed data and the ground truth. Super Resolution Convolutional Neural Network (SRCNN) and a Deep ResNet are also used for reconstruction. The proposed algorithm outperforms SRCNN and the unconverted ResNet, specially at low Channel SNR (CSNR). In addition, the proposed algorithm results in a 68% reduction in both storage and energy requirements, which makes it suitable for implementation on User Equipment (UE).


I. INTRODUCTION
The emergence of the Internet of things (IOT) and more recently the Internet of everything (IOE) [1] has resulted in a surge in the demand over network resources. On the other hand, limited spectral resources call for an efficient paradigm to handle the massive amount of devices operating in the same spectrum. Cognitive Radio (CR) offers a spectrum sharing solution by exploiting the idle spectrum bands, that are not being used by Primary Users (PUs) at a given time, to be utilized by other active devices called Secondary Users (SUs).
The spectrum sharing in CR has four stages as shown in Fig 1. The first step in spectrum sharing [2] is spectrum sensing, which is locating the idle spectrum channels The associate editor coordinating the review of this manuscript and approving it for publication was Marco Martalo .
(not occupied by PUs). Spectrum allocation is based on the detected idle frequency bands, where the SUs are distributed according to optimal method to efficiently exploit the idle holes in spectrum. Next, the distributed SUs are ready to access the spectrum holes provided that the primary signals are absent since the PUs have the priority to access their own spectrum bands any time. Finally, the spectrum handoff stage comes when the SU should leave the spectrum hole it occupies for one of three reasons. The first reason may be that the PU is getting ready to occupy its band, the second reason may be that the geographical location of SU is changed, the third reason may be that the spectrum hole is not suitable for the amount of data the SU needs for transmission.
Spectrum Sensing (SS) techniques are classified as coherent or non-coherent detection [3]. The advantage of coherent detection techniques such as matched filter detection and cyclostationary detection is their accuracy in spectrum detection, but the disadvantage is their implementation complexity. Non-coherent detection methods, such as energy detection, Wavelet detection, and Compressive Sensing, do not require any prior knowledge of the signal. They have the advantage of being simple to implement and operate, but at the expense of less immunity against noise.
SS techniques are classified into wideband and narrowband techniques as shown in Fig 2.; where energy detection is the most widely used narrow band SS technique. Many works relied on the energy detection technique because it is the most basic in terms of hardware complexity and operation [3], [4], [5], [6]. Narrow band techniques utilized in cooperative spectrum sensing suffer from delay which results in a low throughput and utilization of the spectrum. Wideband techniques resolve the aforementioned issues.
Special attention is given to approaches that utilize sub-Nyquist sampling such as Compressive Sensing (CS) [7], [8] which provides an important advantage of spectral efficiency, since data can be transmitted at a lower sampling rate than the Nyquist rate (sub-Nyquist rate) [9]. CS may also be applied for data transmission after spectrum allocation with high accuracy as well as spectral efficiency. Data such as images or voice signals whose length can be reduced while keeping important information unaffected, is called compressible or sparse [7], [10]. CS exploits the sparse nature of the data to reduce the sequence length N produced by Nyquist rate sampling to a new smaller length by sampling the data at a sub-Nyquist rate resulting in a sequence length M, where M<N [7], [11]. The main benefit here is the reduction in the bandwidth required to transmit the sequence. In this paper, we investigate a spectrum sensing algorithm for detecting spatial dimension holes using CS. This detects transmission opportunities even if the band is already filled, by taking into account the sparse order of the data.
The sparse representation of the data provides incomplete samples which can successfully represent the data be used to accurately reconstruct the original signal. First the data is transformed into the sparse representation using a measurement matrix ϕ [12], which has 2 types, random and deterministic. The random measurement matrix may follow a Gaussian or Bernoulli distribution, while Toepltiz and Circulant [12] are examples of a deterministic matrix. There are methods for designing measurement matrices. [13] designed an optimal measurement matrix. Also [14] introduced the machine learning as tool for measurement matrix design. because the deterministic matrix has the advantage of saving memory but may jeopardise the low coherence condition required for lossless information recovery. The type of measurement matrix used affects the reconstruction results. In [12], the authors showed that random measurement matrix gives superior results compared to a deterministic one.
After the signal is transmitted in the reduced sparse form, methods such as iterative greedy algorithms [15], [16], [17] [9]. are used to solve the problem as a Linear programming problem [18]. Denoised-based Approximate Message Passing (D-AMP) has been proposed in [19] for reconstruction, which combines highly developed denoiser and compression algorithms but suffers from high complexity. Recently deep learning has emerged as a novel technology to efficiently learn data representation. It may be exploited in CS to reduce time complexity of reconstruction, since once the deep learning network has been trained it operates much faster than any other optimization algorithm (greedy algorithm family).
Recently, deep neural network has been an effective tool for learning mappings for image restoration, and introduced to CS reconstruction part. In [20], the reconstruction part used algorithm formed of 2 stages, the in initial stage is relying on simple least squares problem that uses Fast OMP to reconstruct a rough form for the desired signal, the importance of this initial stage is to avoid utilization of fully connected layer in the upcoming part. The second stage utilized deep learning through a simple structure called Super Resolution Convolutional Neural Network (SRCNN) [21]. SRCNN had a small number of parameters, but its main disadvantage is slow convergence [22].
A more powerful architecture is the Residual Neural Network (ResNet) [23], which gives better results than using SRCNN with faster convergence. When deep architectures approach convergence, a degradation in performance arises due to a growth in training error as more layers are added. Hence the Residual Dense Net (RDN) in [24], which is a simple block-wise ResNet architecture, does not give a sufficient resolution enhancement over SRCNN. In [25], the authors introduced the concept of channel attention, where the weights are scaled according to feature importance, which improved performance but at the cost of a huge increase in the number of parameters. Densely Residual Laplacian Network (DRLN) [26] mainly contributed the dense connection, which use the previously computed features and Laplacian attention to weigh the features at multiple scales according to their importance. In addition, inter-dependencies between features was utilized for faster convergence relative to SRCNN. Enhanced Deep Super Resolution (EDSR) [27] provides a performance equivalent to DRLN but with a larger number of parameters.
Even though DRLN has the least number of parameters and the best performance, it is a dense architecture, which, although quite trainable, leads to a large number of operations during execution. Going through the dense architecture during reconstruction to produce the output will require high power and storage capabilities that may not be available in handheld devices. Spiking Neural Network (SNN) [28] comes as a promising solution for these problems.
Neuromorphic computation has proved to be a strong candidate in the next AI generation since it reduces the processing complexity of traditional Analog Neural Networks (ANNs). Deep architectures, such as the ResNet, need huge processing power and storage during implementation, due to the floating point weights. Deep SNNs reduce storage and power demands due to their event-driven operations rather than continuous operations that ANNs suffer from.
The main difference between (ANN) and SNN is that, in ANN data is passed through the layer as a whole with as the activation layers work with a continuous range for the data. In SNN, the input data is encoded in the form of spike trains through time steps, as in [29]. The gates of the activation layers compare the incoming potential with some threshold, then it either fires a spike or be off. This model for activation layer is called Integrate and Fire (IF) model [30] and reduces the number of operations and hence power consumption of the whole network [31].
In SNNs, communication between neurons is done by broadcasting trains of action potential (spike trains) to downstream neurons. These individual spikes are sparse in time; thus each spike has high information content. One of the advantages of SNNs is that they are sensitive to temporal characteristics of information transmission that occur in biological neural systems (brain) [32]. But the nondifferentiability of the spikes dictates a complex training process such as Spike-Timing-Dependent-Plasticity (STDP), which cannot be directly applied to deep SNNs. As the SNN goes deeper the spiking amplitudes gets smaller which makes it hard to deal with. The most applicable method for applying the SNNs to real life problems is the ANN to SNN conversion [33]. A pre-trained ANN may be transformed to a SNN by converting the activation and pooling layers into spiking activations, then analyzing the threshold values which affect the weights using either weight normalization or threshold balancing [31].
In [32] and [34] a method for conversion has been proposed that worked for traditional Convolutional Neural Networks (CNNs) and one block ResNet. Reference [35] is one of the primary works that discussed the details of deep ResNet or Visual Geometry Group (VGG) conversion depending on the position of Rectified Linear Unit (ReLU) activation and the convolutional layers. The authors in [36] applied the backward residual technique to enhance the conversion results and applied that for a simple block ResNet and also for VGG, yet it could not manage a dense connection ResNet architecture. Reference [37] applied a simple construction for SNN, that takes a simple CNN or one block ResNet architecture. It contributes the use of Explicit Current Control (ECC) which pumps extra spikes with weight normalization to enhance energy efficiency while reducing accuracy loss. All the previous work was to convert networks into SNNs that are used for classification. However, conversion of dense architectures used for prediction or reconstruction has not been discussed.
The aim of this work is to introduce a reconstruction method based on an initial layer of simple OMP algorithm to reconstruct a rough form for the desired image, then apply deep learning to enhance the image resolution.
This enhancement in the resolution is similar to the work in [20] but the SRCNN architecture is replaced with Deep ResNet architecture to improve the Peak Signal-to-Noise Ratio (PSNR). The trained Deep ResNet architecture is then converted to SNN to reduce complexity and energy consumption.
The Contributions of this work are: -Introducing weight quantization to reduce memory requirements to be suitable for the SUs as handheld devices. -Introducing a novel distributed conversion method to convert deep ResNet architectures with skip connections to deep SNN, without a significant loss in accuracy, this has an impact on CRN performance in real-time applications.

II. SYSTEM MODEL
One of the powerful wideband spectrum sensing techniques is Compressive Sensing (CS). It has the advantage of sampling wideband signals at a sub-Nyquist rate, allowing the Analog to Digital Converter (ADC) requirements for the SUs to be relaxed. Based on the assumption that the spectrum is underutilized (for example, in a suburban or rural area), CS can be used to approximate and recover the detected spectrum, making it easier to sense sparse primary signals in the wideband spectrum. As a result, CS offers a promising option for recovering wideband signals faster and facilitating wideband sensing with a lower computational complexity. Compressive Sensing is proposed for compressing sparse data before transmission to improve spectral efficiency through reducing the sampling ratio. In this work, the data used is in the form of images to illustrate how CS affects the quality of the data after reconstruction. However, the reconstruction algorithm proposed in this paper may be used with any type of data.
In this model, we propose a compressive sensing module formed of the sensing part and the reconstruction part, to be added to the system as shown in Fig 3. In the following subsections we discuss each part in detail. The first part in the compressive sensing is the sensing part, where the signal is projected on the suitable domain to be treated as sparse signal and then compressed to be ready for transmission.
The reconstruction algorithm starts with simple recovery Orthogonal Matching Pursuit (OMP) algorithm obtained to get a rough form for the reconstructed image. Next a deep architecture is used to improve the resolution of reconstructed image. Fast OMP is used to avoid the need for fully connected layers to reconstruct the image from scratch [16]. In [38], the authors tried to implement deep learning architectures to reconstruct the compressed data without OMP, but their reconstruction algorithm could not overcome channel noise and fading effectively. In [39], an Autoencoder was proposed to be used solely for reconstruction, but it suffered from the computational overhead of using fully connected layers.
The proposed deep learning architecture is ResNet, since ResNet outperforms SRCNN in terms of accuracy and speed of convergence [24]. After the architecture is trained till convergence, it is converted from ANN to SNN. The conversion reduces the number of bits required for weight representation, thus a significant reduction in storage and energy requirements is achieved without a significant loss in accuracy.

A. SENSING PART
The main idea of compressive sensing is to recover the signals from fewer samples than required by the Nyquist rate [10]. This is assuming signals are sparse or compressible in the frequency domain by some transform.
The first step in the sensing part is the sparse representation, where the signal appears to be sparse when projected on suitable basis. In this work, the basis matrix R N ×N is a Discrete Cosine Transform (DCT) matrix.
Next the signal x is sampled using C M ×N (M < N ). The compressed transmitted signal is formulated as: where A is the channel fading coefficients matrix and n is the channel noise. A random Gaussian measurement matrix is used, as it yields better results, due its low coherence, compared to other types [12]. During transmission Orthogonal Frequency Division Multiplexing (OFDM) is used for modulation to mitigate channel fading, since we investigate wireless fading channels.

B. RECONSTRUCTION PART
In this work, the signal recovery algorithm is comprised of two stages. The first step is a simple greedy algorithm to avoid the use of fully connected layers due to their high computational complexity. The second step is a deep ResNet architecture that is used to construct a more accurate signal.

1) FAST OMP
Sparse signals such as audio or images can be well approximated by a linear combination of few elements of some redundant basis (dictionary learning). Fast OMP is a linear mapping process from the measurement vector y to a real signal, such that y c (a column of y) has K-term representation over the measurement matrix , since x c has only K nonzero terms, then y c is just a linear combination of K-columns from .
The input for this algorithm is a column of compressed signal y c and the measurement matrix (a random matrix but is same one that has been used for encoding), fast OMP takes 2 * K rounds to recover the signal. Initially a residual vector is defined to be equal to y c, , then the residual is refined by calculating the inner product between the residual vector and the measurement matrix, the detailed steps are mentioned in [20].

2) DEEP ResNet ARCHITECTURE
In this work, the proposed architecture is ResNet [26], the main advantage of this architecture is the multi-level representation such that it divides the image into main features and small features. Each type of features is handled by different parts in the network. It relies on reusability of computed features and the method of residuals which gives better results with a lower number of parameters, compared to SRCNN [20].
In [26], The laplacian attention is proposed to boost and exploit the relationship between features that are essential for super resolving the images. A Global descriptor is used to produce the attention adaptively according to the relative importance of the features, specifically the high frequency type (sub-band). The Laplacian pyramid weights reflect the sub-band features of high importance progressively in each DRLM.
The proposed Deep ResNet is formed of a feature extraction layer which is a convolutional layer. The output of the first layer is passed through cascading residual in residual blocks, and finally the data is passed through the reconstruction layer to produce the image as shown in Figure 4. The cascading blocks include dense connections to improve the performance. Dense connections may be used to capture complex patterns, instead of having to use non-linear models for the representation. There are three types of skip connections in this architecture: Long Skip Connection to pass the main features that have already been computed in the feature extraction part, Medium skip connection to make use of the features computed for each block, and the Shortcut Connections(SCS) that can be called dense connections which enhance the performance of the building residual unit in the block.
The authors in [26] introduced the impact of using the Laplacian attention concept to enhance the feature learning, especially the sub-band features. This architecture is then converted into SNN.

3) CONVERSION TO DEEP SPIKING NEURAL NETWORK
The authors in [37] proposed a CNN-SNN conversion method through obtaining a thresholding mechanism to quantify the accumulated current into spikes in SNN. However, the allaround conversion method in [37] is not suitable for converting deep architectures with skip connections. Hence, this method will be used for converting individual blocks in this work, which will then be combined to construct the spiking deep architecture. And through the equations it bridges between the activation in CNN and the accumulated current in SNN instead of spiking rates as in previous works [31]. This is obtained through current normalization (CN) and Thresholding for Residual Elimination (TRE).
To achieve CN, weight normalization has been utilized for both weights and biases for the trained Deep ResNet as follows, for layer n, where V n th is the threshold potential for layer n, κ n is the amplification factor for layer n, λ n is the maximum activation value for layer n, W n ij is the weight between the neuron j in layer n − 1 and neuron i in layer n and b n i is the bias of neuron i in layer n. κ n is used to amplify stimuli for the next layer, whereas λ n is used to normalize the weights to avoid an explosion or decay in their values. CN normalizes the current of layer n, I n i (t) as: where T is the maximum time Step. VOLUME 11, 2023 TRE algorithm pumps small amounts of current at each neuron to get rid of the residual information that increases the error. The bias term for layer n is updated according to: where, η ∈ [0, 1] and a typical value of η is 0.5. Bias value updating takes place two times. the first time, through the initial step of weight normalization (Equation 5), and the second time, through the spiking activation, during spike generation, the bias is then updated according to Equation 7.
Given the deep ResNet architecture, we convert this trained architecture to a SNN through a novel distributed conversion technique to simplify the process. Initially, the input data should be encoded to convert the input image pixel values into a spike train across time step [40].
In the proposed distributed conversion method, we start with converting the smallest building residual unit into a linear Spiking model which relies on converting the Convolutional layer. The ReLU activation is converted into an Integrate Fire (IF) neuron, which is a spiking activation layer that produces spikes instead of producing continuous values.
Weight quantization [41] has been applied to reduce the required storage since the weights from the trained ResNet are represented using half precision (32-bits), whereas the proposed SNN uses weights encoded in only 10 bits. The weights from the ResNet are quantized before the conversion into SNN. There is some loss of precision due to reducing the number of bits. However, this quantization shortens the time steps required by the spiking neurons to represent the activations.

III. PERFORMANCE EVALUATION A. IMPLEMENATATION
We choose the UC Merced Land Use Dataset [42] and Set 5 Dataset [43] to train and test our deep learning regression. Before sensing, the images are converted from RGB to grayscale. There is also an experiment performed on RGB images to evaluate whether the architecture will work on 3-layer images. We have two deep learning architectures for comparison: -Super Resolution CNN [20]: a primary stage of Fast OMP followed by a simple CNN to enhance the image accuracy. -Deep ResNet Architecture [26]: a primary stage of Fast OMP followed by a Deep ResNet architecture to enhance the image accuracy. In the sensing part, the sampling ratio is set to 0.7 and the wireless fading channel is Rayleigh fading. BPSK and OFDM are used to transmit the data. We evaluate all CS reconstruction algorithms with CSNR varied from 10dB to 30dB, where the noise is Additive White Gaussian Noise (AWGN). The number of iterations for Fast OMP is set to 40. The sensing part and Fast OMP stages are implemented using Matlab 2018b on a processor intel core i7, 8.00 GB RAM. In the training process, learning rate is set to 10 −4 , batch size to 16 Fig.5 shows the reconstruction performance on an airplane image in terms of PSNR using different deep learning architectures with varying CSNR. The three reconstruction algorithms are robust against channel noise and fading. Analog Deep ResNet gives the best results with high CSNR (>15dB). We notice that the spiking Deep ResNet performance is comparable with SRCNN, hence the benefit from using spiking Deep ResNet is faster convergence at the training process and a much lower energy consumption, storage and runtime during operation. At a CSNR of 10 dB spiking Deep ResNet outperforms the other architectures which means that it is the most robust to noise and fading.     Fig. 7 shows another airplane image reconstruction where Fig.7(b) shows the image after fast OMP iterations with PSNR 24.9 dB and Fig.7 (c) shows the image after the Spiking Deep ResNet with a PSNR of 26 dB and MSE decreases from 206.28 to 163.05 with respect to ground truth mage. SSIM is slightly improved from 0.807 to 0.809 with CSNR set to 15dB. Table 2 shows the SSIM improvement for each reconstruction algorithm, where the performance of SRCNN and  Performance has also been tested on RGB images of the same dataset. The reconstruction has been performed separately for each layer (Red, Green, Blue). The reconstructed images, using Analog ResNet show an improvement of 5-6 dB over the degraded images, as indicated in Fig. 8. The MSE also decreases after reconstruction of the degraded images at a 10 dB channel. The performance deteriorates using the proposed SNN for reconstruction of RGB images, as illustrated in Fig. 9, where the PSNR increases by 3-4 dB compared to the degraded image.
The results on Set 5 dataset show comparable performance to the results of the UC merced dataset. Fig. 10 shows an example of the results on Set 5. Fig. 11 compares the performance of the analog ResNet and the proposed SNN architectures in terms of the achieved PSNR against CSNR.
Through weight quantization, the number of bits required for representing weights has been reduced from 32 bits (half precision), which are usually utilized, to only 10 bits without a significant loss in performance. This reduces both energy consumption and utilized storage by 68.72%, as the number of bitwise operations is reduced. The number of operations  required to run the reconstruction algorithm is reduced from around 170,000 MOps to only 53,000 MOps.

IV. CONCLUSION
In this paper, the main objective is improving the CRN through suggesting compressive sensing as a data transmission technique which is a sub-Nyquist based algorithm. We mainly worked on improving the reconstruction algorithm through relying on Deep Learning which is a powerful tool to alleviate the fading channel impact. The proposed reconstruction algorithm is formed of two stages the first stage is a simple OMP algorithm that recovers a rough signal with its original size. The purpose of using this layer is to avoid utilizing fully connected layers through the Deep learning part. The second stage is a more accurate reconstruction based on a deep learning algorithm. A Deep ResNet architecture is proposed for its faster convergence compared to SRCNN. The ResNet architecture is converted to a SNN through a novel distributed conversion method. The use of a SNN improves storage, processing and energy requirements for the hardware implementation of the reconstruction algorithm. Since the reconstruction is intended for SUs in CR network, which use battery operated devices, energy efficiency is of paramount importance. We evaluated the quality of reconstructed data through PSNR versus the CSNR, MSE and SSIM. Results show that the PSNR when using spiking Deep ResNet for reconstruction is comparable to SRCNN with a much lower storage and energy requirement. In addition, Spiking Deep ResNet is more robust against the low CSNR, with a lower MSE in the reconstructed data and without affecting SSIM. Results on RGB images show an improvement using the Analog ResNet. However, performance of the SNN is worse than the analog architecture since sparsity is not consistent among image layers, thus performance varies from one layer to another. Finally, the spiking Deep ResNet architecture significantly reduced the number of operations and storage requirements through weight quantization.