s-TBN: A New Neural Decoding Model to Identify Stimulus Categories From Brain Activity Patterns

Neural decoding is still a challenging and a hot topic in neurocomputing science. Recently, many studies have shown that brain network patterns containing rich spatiotemporal structural information represent the brain’s activation information under external stimuli. In the traditional method, brain network features are directly obtained using the standard machine learning method and provide to a classifier, subsequently decoding external stimuli. However, this method cannot effectively extract the multidimensional structural information hidden in the brain network. Furthermore, studies on tensors have show that the tensor decomposition model can fully mine unique spatiotemporal structural characteristics of a spatiotemporal structure in data with a multidimensional structure. This research proposed a stimulus-constrained Tensor Brain Network (s-TBN) model that involves the tensor decomposition and stimulus category-constraint information. The model was verified on real neuroimaging data obtained via magnetoencephalograph and functional mangetic resonance imaging). Experimental results show that the s-TBN model achieve accuracy matrices of greater than 11.06% and 18.46% on the accuracy matrix compared with other methods on two modal datasets. These results prove the superiority of extracting discriminative characteristics using the STN model, especially for decoding object stimuli with semantic information.


I. INTRODUCTION
N EURAL decoding is achieved by analyzing the neural signals pattern collected using the noninvasive device to evaluate the brain's activation patterns in response to specific visual stimuli.Subsequently, these patterns are used to deduce the external stimulus categories inversely [1].One vital step in this process is establishing the mapping relationships among the neural activation patterns, neural signal patterns, and external stimuli.Previous research focused on the brain individual-area model to establish this mapping relationship (such as a single brain area or channel signals) [2].However, the study of this mapping from the network model on the level of the entire brain or the local system has become a research hotspot [3].
Brain network structure data typically involve various dimensions, such as time and space, and indirectly represent potential characteristics, such as visual stimuli and participant characteristics [4].Some studies have shown that this brain connection patterns have a spatiotemporal structure and are the carriers between neuroimaging data and stimulus information [5].However, in the traditional data-processing method, the brain network structure is directly input into a classifier for the extraction and classification of the brain network features, during which some multidimensional structural characteristics of the brain network are lost.Therefore, building a strong representation ability model to extract network features is essential.
In the field of neural signals, some studies have introduced tensor models and network features into neuroimaging data analysis [3], [6], and, some results of the brain network model have shown that the tensor-based model outperforms the matrix-based model [7].The matrix-based model encodes the data correlation only in the low-rank dimension information (only in space or time) and neglects strong correlations between different modes, i.e., spatial and stimuli modes [7], [8].Tensor-based model encodes the data correlation as a tensor model, which is consist of the structured data retainning the high-order statistical characteristics of original data, internal structural information, and the correlation between data dimensions.Furthermore, some researches have introduced a multi-view information-constrained tensor network to extract the brain network pattern characteristics from EEG [9] or fMRI data [10] and distinguish the patients from participants.However, several shortcomings exit in the brain network model based on the tensor form, such as neglect of the sparsity in the network, local optimization, and some prior information about the constraint relationship between stimulus features and networks [10].
Among previous neural decoding methods, one study used visual stimulus information to build the neural coding and decoding model [11].It employed the characteristic neural data to optimize the classifier parameters under the aid or constraint of the stimulus information, which involving the adversarial or poisoning attack [12].This model considered the prior information of visual stimuli and the coupling information pattern between brain signals, increasing the effect of the decoding model above that of other models.Based on this, this research constructs the stimulus-constrained tensor brain network (STN), a tensor brain network model based on visual stimulus constraints.During the construction, we introduced the idea of compactness of the stimulus category according to the brain network pattern into the multivariate analysis framework.
This study's primary contents are as follows.1. construction of the s-TBN model.2. construction of the brain network patterns of different visual stimuli under fMRI and MEG modality datasets and adoption of a semi-supervised learning classification framework to verify the model's effectiveness.3. analysis and interpretation of the model's parameters.

II. METHOD
This paper proposes a tensor brain network model based on visual stimulus constraints coupled with multivariate pattern analysis to decode visual stimulus categories.In this section, the paper introduces four aspects.Firstly, we define the brain network pattern datasets under visual stimulation.Second, we introduce tensor decomposition technology and build a tensor brain network based on stimulus constraints.Finally, we introduce an optimization algorithm to solve the model.A diagram of the model framework is shown in Fig. 1.

A. Definition of the Graph Structure in the Brain Network
Studies have indicated that the connection matrix across brain regions can represent the brain network patterns [13].One common approach to studying brain networks is through the use of atlases [14].Brain atlases provide standardized anatomical reference maps that divide the brain into distinct regions based on structural or functional criteria.Therefore, We introduced the atlas-based workflow to construct the brain network patterns across the whole brain.
Definition 1 (Brain network): The connection matrix A i ∈ R m×m is set to represent the brain network pattern G i , it can be defined as follows: where p, q = 1, . . ., m. m indicates the number of brain regions.v p,t represents the neural signal in v p brain region across time t.µ p represents the mean value of the neural signal.w is the length of the time window.The atlas-based workflow of constructing a brain network is to estimate the connection matrix across brain regions according to the specific brain anatomy template.Our study adopts the Destrieux whole brain template to obtain the signal value within the separate brain region.For the length of the time window w, we choose the most available values according to the previous studies.
Our study adopts the graph to represent the structure of the brain network data.First, D = {G 1 , . . ., G n } represents a series of brain network patterns, where n indicates the number of visual stimuli of the same kind.The paper supposes that the brain network structure under visual stimulation shares a series of vertex sets V , representing a specific brain anatomy template to divide the brain into m brain regions.A brain network G i can be represented by an adjacency matrix A i ∈ ℜ m×m .
Definition 2 (Graph): The set G = (V, E) represents a graph, where V = {v 1 , . . ., v m } represents a set of vertices and E ⊆ V × V represents a set of edges.
This paper adopts a semisupervised learning framework to train the brain network data, i.e., only a part of the data is labeled, and some are unlabeled.For example, in the brain network set of the D, the lth sample is labeled with Y ∈ ℜ l×c , where c is the number of class labels.Each brain network belongs to only one category, i.e., if G i belongs to the jth category, Y (i, j) = 1.This study simplifies the labeling by labeling the labeled data as D l = {G 1 , . . ., G l }, the unlabeled data as D u = {G l+1 , . . ., G n }, and the total dataset as D = D l D u .

B. Tensor Brain Network Decomposition
In this paper, the output of the graph structure is represented by a tensor and the structure characteristic of the brain network is learned in the space represented by the tensor.First, the brain network data under visual stimulation, i.e., {A i } n i=1 , were combined into a large partial symmetric tensor structure X ∈ ℜ m×m×n , representing all graph information G. Therefore, the graph information G is incorporated into the model by embedding X into the model.Subsequently, tensor technology is used to analyze the tensor brain network.
Definition 3 (Partially symmetric tensor): An m order of tensor X ∈ ℜ I 1 ×•••×I m is a partially symmetric tensor if it can be represented by the tensor product of m vectors on the mode of i Where • represents the tensor product.x (1) andx (m) both represent the vector which tensor mapping into specific mode.
Definition 4 (CP decomposition): For any tensor X ∈ ℜ I 1 ×•••×I m , its CP decomposition is to decompose the tensor into the sum of several ranks, and its form is shown in Formula 3. Where , k is the number of factors, and ≡ indicated that both sides are identical despite the change in variables.X in Eq.( 3) is the same as that in Eq.( 2) We combined the output of brain network data into a tensor structure and assumed that this tensor structure involved three information dimensions, such as time, space, and visual stimuli.The construction process of brain network data was  CP decomposition of the tensor brain network, where the signal of ≡ indicated that both sides are identical despite the change in variables (refers to Equation 3).
designed to time and space information, the spatiotemporal information is represented by the connection information between the vertices in graph structure.Therefore, we wrote the CP decomposition form of the third-order tensor brain network into the following formula 4.
where B (s) ∈ ℜ m×k is the vertex factor matrix, C ∈ ℜ k×k×k is the indicative tensor, and S ∈ ℜ n×k represents the factor matrix for the stimulus.The physical meaning of S represents the hidden neural information on one of the modes regarding the decomposition of the X , and this mode refers to the external stimuli information.In a previous studies, the patterns of brain activity were found to be similar when the stimuli patterns were similar [15].Therefore, the factor matrix S can be used to identify brain activity.The schematic diagram of CP decomposition is shown in Fig. 2.

C. The Tensor Brain Network Model Based on Stimulus Constraints
The following three problems must be solved while constructing the s-TBN model: (1) learning the brain network's high-order structural characteristics during representation learning, (2) adding visual stimulus information as an auxiliary variable to learn the brain network data, and (3) integrating the learning of the classifier into representation learning.This paper has solved the first problem by constructing tensor brain network data.Next, the paper will focus on solving the remaining two problems.
The stimulus information can assist in analyzing the brain signal data.Previous studies have highlighted that in the hidden space of auxiliary variables [16], the brain network structure's similarity under visual stimulus is consistent with the stimulus feature space's similarity.When the two visual stimulus features are similar, the characteristics extracted from the brain network structure are also similar.Therefore, we defined an objective function to constrain the distance of the brain network structure based on visual stimulus characteristics.min where Z is the kernel matrix and Z (s) (i, j) represents the similarity between network structures in the stimulus feature space.S(i, :) represents the stimulus factor matrix for the index of the stimulus picture i.Here, the boundary information for the visual stimulus features can be effectively used to discover the discovery of the meaning of hidden factors.For simplicity, Formula 5 is rewritten as follows: where tr (•) represents the trace of a matrix, L Z (s) is the Laplacian matrix, derived from the similarity matrix Z (Z (s) = D Z (s) − Z ), and D Z (s) is the diagonal matrix.The nonzero elements of the diagonal matrix are the sum of the row vectors of the matrix, specifically Furthermore, we adopt orthogonal constraints on the stimulus matrix factors to discover hidden discriminative factors and obtain higher accuracy and interpretable results.
Regarding the third problem, i.e., to learn and classify brain network structure data, we assumed a mapping matrix W (s) ∈ ℜ k×c between the visual stimulus S and the label Y .Therefore, the third question can be described by the ridge regression problem as follows: min where D = [I l×l , 0 l×(n−l) ] ∈ ℜ l×n , and ∥W (s) ∥ 2 F controls the capacity of W (s) , and the parameter γ controls the influence of W (s) .This research adopts the hidden factor S on the stimulus feature modality as a feature by combining the framework of semisupervised learning to find the discriminative features related to classification from the original data.In the classification learning framework, this paper embeds the hidden feature S and the classification parameter of learning framework W (s) into the same model to be cross-trained.The hidden feature S is learned from the tensor decomposition model.Here, combined with a partial symmetry tensor and tensor decomposition technology, a brain network data X and visual stimulus category information Y coupling was indirectly put in the visual stimulus modal space for training and learning.
Finally, this study's proposed the s-TBN model can be described by the following optimization problem: min Where α, β, and γ separately control the three parameter values of stimulus information, classification loss and regularization, respectively.

D. The Solving Process
Recently, studies have introduced the Alternating Direction Method of Multiplier (ADMM) to solve the CP tensor model [17].This method converges to the optimal solutions and achieves much higher accuracy than the proximal gradient algorithm using fewer iterations.The s-TBN model is mainly to solve the nonconvex orthogonal-constrained optimization problem involving the CP tensor model.Therefore, we used the ADMM to solve the s-TBN model.Fixing the parameters S and W (s) and solving B (s)  In Formula 9, setting paramaters S and W (s) as constants, the objective of solution focuses on the first part of the equation.Indeed, X is a partially symmetric tensor, and Formula 9 involves the fourth-order term of B (s) ; hence, directly solving the optimal solution is difficult.By employing variable substitution technology, we use the paramater F (s) to represent the paramater B (s) and treat it as a constraint term.Thus, the original function about the Formula 9 is transforms into minimizing Formula 10. min Where F (s) is an auxiliary variable.The augmented Lagrangian function of Formula 10 can be written as follows: Where U ∈ ℜ m×k is the Lagrangian multiplier, µ is the penalty multiplier, and the parameter µ is adjusted according to the conference [18].To find the optimal solution of B, we rewrote Formula 11 into a convex function.The specific calculation process is as follows: First, the tensor decomposition can be written as an equivalent form according to modal one.
Second, the tensor decomposition formula in Formula 10 can be further written as.
For Formula 11, by adding one 1 µ ∥U ∥ 2 to the last two items, we obtain the following equation: As the last item in Formula 14 is irrelevant to B (s) , combined with Formula 13, the minimum value of parameter B (s) can be transformed into an augmented lagrangian function as follows: Thus, the problem is simplified to a problem of the convex function's optimal solution B (s) , and its optimal value is at the function's inflection point.The specific solution derivation process is as follows: By combining the B item in Formula 16 and moving the unrelated items to the right of equals sign, the following equation is obtained: Finally, the optimal solution B (s) obtained in this study is as follows: Simultaneously, the optimal solution of another auxiliary variable F (s) is the Formula 19 Where P = S ⊙ F (s) ∈ ℜ (m * n)×k , and X (2) is the matrixization of the tensor X on mode two.Furthermore, this paper adopts the gradient descent principle to optimize the Lagrangian factor.The specific process is shown in Formula 20.
Fixing the parameters W (s) and B (s) and optimizing S Parameter S is the factorization matrix of tensor's mode three on the visual stimulus characteristic direction.Therefore, we set the objective optimization function as follows: Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
Where G = F (s) ⊙ B (s) ∈ ℜ (m * m)×k and X (3) ∈ ℜ n×(m * m) is the matrix on tensor mode three.The objective function is an optimization problem with orthogonal constraints.
For Formula 21, it is often to be as N-P hard problem and is challenging to solve S.Many studies have proposed algorithms to solve this orthogonal constraint problem [19].The curvilinear search is the majorly used algorithm because it can be applied to finding an effective step size and can guarantee that the iterations converge to a stationary point quickly [20].In curvilinear search algorithm, the gradient function of the objective function is the crucial step in solving S. Therefore, we calculated the derivative of L(S) regarding S.
Fixing the parameters S and B (s) and solving W (s) The final part is solving the parameter W (s) , which represents the process of classification learning.Based on Formula 23, the main function is the least squares function under the quadratic constraint, and it is a convex function, therefore, there is an optimal solution.
The process of solving this parameter is that the objective is zero after deriving W (s) , and the inflection point is the optimal solution W (s) .Finally, W (s) is solved using Formula 24.
Last, Algorithm 1 shows the process of solving the tensor brain network based on visual stimulus constraints.

III. EXPERIMENTS
This paper adopts two collected datasets to evaluate the model's effect: MEG data and fMRI data.Many studies have proven that these two neuroimaging datasets models can construct a stable brain network pattern [21].

A. fMRI Data
Six Chinese participants were included the study (three females, mean 24±2 years).All participants were righthanded, had normal hearing and had normal or corrected to normal vision.All participants participated in an emotional stimulus picture test system, indicating that they could recognize emotions.Furthermore, all participants provided written informed consent.The Beijing Normal University Review Board approved the experimental procedures.

B. MEG Data
Nineteen Chinese participants were included the study (10 females; mean 23 ± 2 years).All participants were righthanded, had normal hearing and normal or corrected-to-normal vision.All participants provided written informed consent.Experimental procedures were approved by the Peking University Institutional Review Board.
The stimuli comprised 640 gray-scale images (300 × 300 pixels, visual angle 7.92 • × 7.92 • ) from four categories (160 images per category), including faces with neutral expressions, scenes, animals, and tools.All images were matched for mean luminance and contrast using the SHINE toolbox.The face stimuli, selected from the Chinese affective picture systems, included 80 unique male and female neutral faces.The stimuli were all outdoor scenes, including mountains, countryside scenes, streets, and buildings, with 40 unique pictures of each type.The animal stimuli included mammals, birds, insects, and reptiles, comprising 40 items, each with four exemplars.Finally, the tool stimuli included kitchen utensils, farm implements, and other common indoor tools, comprising 40 items, each with four exemplars.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

IV. RESULTS
This paper first gives the results of the STN model on multiple datasets and the decoding effects of other decoding models on the same dataset.We analyzed the effect of the STN model's hyperparameters on the decoding results.

A. The Effect of Model
This study divided the two modalities' data into 14 data sets and then conducted experiments on these data sets one by one to verify the effect of decoding the categories of visual stimulus in the s-TBN model.First, the MEG datasets were divided into seven sets, including Faces vs. Animals (F-A), Faces vs. Scenes (F-S), Faces vs. Tools (F-T), Animals vs. Scenes (A-S), Animals vs. Tools (A-T), Scenes vs. Tools (S-T), and four visual stimulus datasets.The fMRI data were divide into seven datasets, including Disgust.vs Fear (D-F), Disgust vs Happiness (D-H), Disgust vs Neutral (D-N), Fear vs Happiness (F-H), Fear vs Neutral (F-N), and four emotion sets.Second, we compared the decoding effect of three decoding models: the traditional classification model based on brain network features, the classification model based on the tensor brain network, and the classification model based on neural networks.Finally, we used three evaluation indicators to measure the model performance of the model: accuracy, recall, and F1 scores.
1) The Results of Decoding the MEG Data: For constructing a brain network pattern in the MEG data, we adopt the results of the references [26] to construct the optimal brain network pattern under four visual stimuli.The decoding results were the average of all subjects, and each subjects' results were obtained from the average of the ten-fold cross-validation.Table I presents the decoding results of its various models on MEG data.The values in parentheses represent the variance values between the subjects.
To measure the decoding effect of the s-TBN model effectively, this paper adopts the widely used and high-efficiency classification models, such as Support Vector Machine (SVM) and Random Forest (RF), to extract brain network features and identify the different brain network patterns.These decoding models have strong applicability and do not require too many samples; therefore, they are often used in visual decoding and achieve good results [27].In this study, the kernel parameters in the SVM model are a linear kernel, the other parameters are default.In addition, these models were implemented with scikit-learn on Python 3.6.
To prove that this study's s-TBN model could decode the visual categories effectively, we compared the decoding models constructed based on the tensor, such as alternating least squares (ALS)and Rubik.The ALS model is a tensor technology based on the alternating least squares algorithm [28].It can effectively and quickly extract the tensor decomposition components, and does not require excessive data samples, so ALS is typically used to extract tensor brain network features.The Rubik model is a tensor decomposition model proposed based on prior knowledge constraints.It can effectively overcome the effects of noise and missing data to extract tensor components.Moreover, it can perform parallel computing on large-scale neuroimaging data quickly; therefore, the Rubik model is one of the reference templates typically used to construct knowledge constraint tensor models [29].The ALS and Rubik models were implemented on MATLAB, and their parameters refer to the setting values in the original literature [30].
Finally, the deep learning model is the latest method for extracting brain network features.In most neural network classification models, LSTM and EEGNet are smaller than other deep network models.Some researchers have indicated that the large-scale and deep-structured neural network models can obtain superior classification accuracies but need larger datasets to train [31].However, our datasets are smaller than these datasets (samples are at the thousands level).Therefore, the small-scale network classification models are more suitable for our experimental data.LSTM and EEGNet are the most representative models for RNN and CNN models, respectively, which are the base units of constructing other deep network models.Finally, we chose these two-classification models for comparison because they have more advantages than other models.For example, LSTM can fully mine the time series and semantic information about neuroimaging data compared to other standard network models.Furthermore, EEGNet is specially designed to process EEG or MEG data and achieves significantly better decoding results than other models [32].
Table I shows that the average two-classification decoding accuracy of the s-TBN model is 83.80%, and the decoding rate of the four categories is 64.32%.These two decoding rates are significantly higher than those of the other seven models.The experimental results show that the tensor brain network model can effectively extract more discriminative brain network features when the model adds stimulus feature constraints, significantly improving the effect of neural decoding.
2) Results of Decoding fMRI Data: During the fMRI data classification induced by emotional face pictures, constructing brain network patterns was implemented on the MATLAB platform.
First, the window length to construct the brain network must be determined.Considering the time delay characteristics of fMRI data, this article adopts 60 s duration under 10 similar visual stimuli, where the duration of a single fMRI experiment included 2 s seconds of stimulus presentation plus 4 s of fMRI time series moving backward.Second, The Destrieux whole brain template was used to extract the signal value across the entire brain area.Finally, the brain network patterns were obtained by calculating the time domain correlation between brain areas.The entire implementation process was through the DPABI software.Table II shows the averaged emotion decoding results from fMRI across subjects, and the values in parentheses represent the variance values across the participants.Here, we adopted the cross-validation of the leave-one subject out to obtain the final average decoding results.Furthermore, we adopted the same comparison method on MEG data to verify the STN model on fMRI data.
From the results presented in Table II, the s-TBN model performed the best on three indicators compared with other methods.These results indicate that adding prior information constrained-stimulus features to construct the tensor model can help extract further distinguishing structural features, especially for advanced cognition, such as decoding emotional activity states, and the effects are obvious.
Tables I show that for the fMRI dataset, the maximum decoding rate of the s-TBN model for two classification tasks is 95.16% and the maximum four classification decoding is 64.32%.For EEG/MEG data, the highest decoding rate of the STN model for two classification tasks is 96.79%.Therefore, combining the comparison results of Table II on the three decoding models, the STN model's decoding performance exceed those of most models.These comparison results also show that the decoding effect of the proposed STN model in this paper is dominant.

B. Analysis of Model Parameters
Our decoding model primarily involved four parameters: k, α, β, and γ .Here, k is the number of factors for tensor decomposition and directly influences the reconstruct effect of the original tensor data [33].Therefore, the size of parameter k affects the decoding rate.Parameters α and β control the relationship between the label information for the visual stimulus and the pattern information on the brain network; the more appropricate the description of that relationship, the better the construction of the decoding models for visual stimulus.Thus, the size of these parameters directly determines the validity of the decoding model.Finally, parameter γ controls the processing of classification learning.The regularization parameter of the variable W (s) was used to constrain the mapping relationship from the information space to the visual stimulus space.Therefore, the decoding rate is affected by the size of the parameter γ .In this study, we analyzed the influence of specified parameters on the model by fixing other parameters and adjusting the target parameters to observe their influence on the STN model's decoding rate.
Hyperparameter k is the number of components in the tensor decomposition process's stimulus modality.It is also the number of feature dimension extracted from the coupled space  between brain network pattern features and visual stimuli features.In this study, the influence of parameters was analyzed by selecting the range to analyze the decoding change rate.The range is from 0 to 50 components with a step size of 5.
The decoding changes present in Fig. 5 show that when the value of k is small, the model's decoding rate is not ideal on two modality data.However, when k is approximately 20, the STN model shows an improved decoding effect on the MEG datasets.For the fMRI datasets, the STN model shows an improved decoding effect when the k is approximately 40.
Hyperparameter γ is the regularization parameter of the variable W (s) in the STN model, and it was used to constrain  the mapping relationship from the brain network feature space to the visual stimulus space.We set the adjustment range of the parameter γ as {10 −2 , 5 × 10 −2 , • • • , 10 2 }, and analyzed the influence of model's decoding rate on multiple datasets.However, Fig. 6 shows that the STN model's decoding change rate is not very obvious on the different values of parameter γ .
Hyperparameters α and β were introduced to the s-TBN model when the model solved the visual stimulus factor matrix.α controls the similarity of the brain network patterns under the visual stimulus, β controls the classification model between the visual category and the corresponding brain network patterns.These two parameters jointly constrain the model's variable W (s) .Therefore, we joined the adjustment process of these two parameters.The adjustment range {10 −3 , 10 −2 , • • • , 10 3 } and the model's decoding rate are shown in Figs.7 and 8.
Indeed, we also demonstrated the convergence of our optimization algorithm by tracking the changes in the objective algorithm values.The changes in the objective function of our algorithm regarding the increase in iterations were present in Figure 1 in Appendix section.The results show that the objective function values on 12 datasets declined and converged to a local minimum value, proving the effectiveness and stability of the proposed optimization algorithm.

V. DISCUSSION
We proposed the s-TBN model to decode neural activity patterns under different visual stimuli from multiple MEG and fMRI datasets.Our study will be applied to the identification of biomarkers or other indicator that allow for the early detection of neurological disorders, such as teenagers with depression or social anxiety disorder.At the same time, it can also be extended to the field of brain-computer interfaces in biomedicine.Overall, the study dissects the neural mechanisms of brain perception at the level of network systems, with its findings better facilitating human understanding of the brain's processing of visual information.
The STN model's average binary classification decoding rate is 83.80%.Furthermore, the four two-classification decoding rates are higher than the average of other models.The average of four classification is 64.32%, compared with the highest decoding accuracy of other models, its decoding ratio increased by 11.06%.In the decoding effect of twoclassification datasets, the average decoding of the face and the other three visual stimuli are higher than the decoding rate between the other two stimuli, and the average decoding rate of the face is 7.52% better than the other two classifiers.These results indicate that the s-TBN model is highly likely to extract the characteristic pattern to distinguish between the face and other visual stimuli.These results correspond with conclusions in neural decoding [34], i.e., the brain activity pattern evoked by face stimuli is easy to distinguish.However, for the decoding effect between two visual stimuli, the two decoding results are lower than other models.This result might be due to EEGNet having stronger extracting discriminative features for scene stimuli than the s-TBN model on these two MEG datasets.Moreover, EEGNet is specially designed to process EEG/MEG data and achieves significantly better decoding results than the other models.However, the STN model gained more advantages than the EEGNet model on the MEG datasets.
In the fMRI datasets, the average two-classification decoding for the s-TBN model is 86.67%, 18.46% higher than the highest decoding rate of other models.Furthermore, the average decoding of four classification is 65.19%, 21.92% higher than the highest decoding rate of other models.One of the inadequacies of this study is that small sample size of six participants may weaken the results.However, the decoding results are remarkably consistent across participants, as shown by the small variance values across participants (all being less than 0.06).Furthermore, our sample size is comparable to that of previous studies [35].Indeed, the differences in brain structural network among people of different ages may vary greatly.However, in this study, the experimental design involves tasks related to object and emotion recognition, which are cognitive behavioral experiments with a certain level of difficulty.Therefore, it requires participants with strong visual and decision-making abilities.Hence, the participants ultimately selected for our experiment were relatively young in age.In the future work, we will extend the dataset and recuit middle-age and elderly participants.Finally, all decoding rates are significantly higher than those of the other models, showing that the s-TBN model has more substantial advantages in decoding emotional neural signals.This is because the tensor model, constrained by visual stimulus features, can easily extract graph structure features with discriminative semantics.This study's s-TBN model extracts the brain network characteristics by finding the optimal subgraph pattern from the original graph structure.Recently, brain network development rapidly, enabling many studies to use the entire brain network connectivity information to decode external stimuli and provide new ideas and directions for neural computing research [36].However, the complexity of brain network data and the lack of graph vector representation methods make it challenging to directly introduce the model to mine effective information.The most direct method is extracting graph-related features from the brain network structure, such as using the local weight coefficient of ROI in the brain network [37], topological core, and other features to study brain state [38].However, this method is often poor in interpretability.The other method is extracting the subgraph pattern's characteristics.The subgraph pattern is more suitable for the brain network pattern.For example, it can model the network connection pattern around the vertex and capture the changes in the local area.The subgraph pattern feature can be a weighted graph [39].For example, Kong et al. proposed a probability distribution model based on dynamic programming, which scores the subgraphs in each set of graph patterns [40].The subgraph can also have a side view introducing multiple vector constraints to find the optimal feature set of subgraphs for graph classification [41].We used the second method, i.e., from the perspective of visual stimulus features, to extract a set of subpicture structure patterns and used them to classify brain network structure data.The article's experimental results prove this method's superiority in neural decoding.
Furthermore, our model added visual stimulus constraints to the tensor decomposition model.Many studies have added constraint information to tensor decomposition to improve the model's effectiveness of the model.For example, Wang et al. [30] introduced knowledge-constrained tensor decomposition to calculate the representation analysis.Except for the low-rank hypothesis, these research results typically use the relationship between data or behavioral data as auxiliary variables to improve tensor decomposition quality.However, a problem faced by the current tensor calculation model is that some prior guidance and constraints are designed for a specific field, resulting in the failure of these methods in other fields.Therefore, it is crucial to introduce prior knowledge to analyze the tensor brain network.For example, in the STN model, vital priori hypothesis is that if the characteristics of the stimuli are similar, the brain network patterns are similar.In the follow-up study of this model, we can add some constraints to further analyze the effect of decoding.For example, specific brain regions have been proven to be involved in processing a specific cognitive process.Hence, to preserve the adjacent areas around these brain regions and find other vertices simultaneously, the known brain regions can be used as a mask to restrict the model so that the brain areas that match the mask can be effectively discovered.For the subsequent models in this study, we can use the FFA brain area, known for processing human faces, the amygdala, involved in emotion processing; and other known brain areas into the model as local constraints.

VI. CONCLUSION
Our experimental results show that the s-TBN model can extract the discrimination pattern of the brain network and decode the visual stimulus categories.Combining the tensor decomposition, which was constrained with prior visual information, this paper uses graph knowledge and tensor structure to represent the brain network patterns and construct the tensor brain network model.The s-TBN model's learning involves discrimination of sub-brain network patterns being extracted from the original brain network.Subsequently, the subnetwork pattern is set as a feature into the classification frame to decode the visual stimulus categories.The experimental results show that the subnetwork pattern extracted from the STN model has Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
a strengthened ability to represent brain activity pattern under different visual stimuli, significantly improving the accuracy of neural signal decoding, especially for emotional decoding with abstract semantics.

Fig. 1 .
Fig. 1.A diagram of the stimulus-constrained-tensor brain network (STN) model.Here, Y represents the stimulus label, W ▷s◁ represents the mapping relationship, and X ▷3◁ represents the factor matrix on stimulus mode.

Fig. 2 .
Fig. 2.CP decomposition of the tensor brain network, where the signal of ≡ indicated that both sides are identical despite the change in variables (refers to Equation3).

Fig. 5 .
Fig. 5.The change in the decoding rate of the model for k on two modal datasets.

Fig. 6 .
Fig. 6.Decoding rate change of the model for γ on two modal datasets.

TABLE I AVERAGE
DECODING RESULTS OF THE STN MODEL AND COMPARISON MODEL ON THE MEG DATA

TABLE II AVERAGE
DECODING RESULTS OF THE STN MODEL AND COMPARISON MODEL ON FMRI DATA