Deep2s: Improving Aspect Extraction in Opinion Mining With Deep Semantic Representation

Syntactical rule based approaches for aspect extraction, which are free from expensive manual annotation, are promising in practice. These approaches extract aspects mainly through the dependency relations in the surface sentence structures. However, deep and rich semantic information hidden in sentences which can help improve aspect extraction, is difficult for them to capture. In order to address the problem, this paper first proposes to employ Logic Programming to explore the feasibility of deep semantic representation, then proposes Deep2S, a hybrid rule-based method to improve the performance of aspect extraction. Deep2S integrates deep semantic representation such as Abstract Meaning Representation (AMR) with syntactic structure. It can take advantage of the syntactical rules to obtain dependency relations in the surface structure as well as the semantic rules to capture deep semantic information. Our experiments are conducted on eight popular review datasets using two evaluation metrics. Experimental results demonstrate the usefulness of deep semantic representation and the ability of Deep2S to improve the performance of aspect extraction in opinion mining.


I. INTRODUCTION
Aspect extraction, also named opinion target extraction, is one of the most important tasks of opinion mining or sentiment analysis [1]- [3]. It focuses on the identification of product aspects or attributes from online reviews. For example, we can extract ''battery'' as aspect terms from the user's reviews in Example 1 and Example 2.
Example 1: ''The camera has a good battery.'' Example 2: ''The battery of the camera is good.'' In the task of aspect extraction, syntactical rule based approaches such as double propagation (DP) [4] have been proven to be promising in practice. This is because they are usually domain independent and free from expensive manual annotation. Syntactical based approaches mainly utilize the surface syntactical information of dependency trees The associate editor coordinating the review of this manuscript and approving it for publication was Shiqiang Wang . to extract aspects. For example, from Figure 1 (b) and (c), we can obtain the dependency relations (i.e., ''nsubjdobj'' and ''nmod'') between ''battery'' and ''camera'' in Example 1 and 2. But these approaches cannot capture deep and rich semantic knowledge hidden in user's review sentences, such as semantic relation ''domain'' which means ''battery'' is an aspect or attribute of ''camera'', as shown in Figure 1 (a). If we know ''camera'' is a product, we can extract ''battery'' as its aspect. It is intuitive that semantic information can help improve the performance of aspect extraction.
To explore the usefulness of semantic knowledge in aspect extraction, deep semantic representation (e.g., Abstract Meaning Representation, or AMR [5]) which abstracts away from syntactic idiosyncrasies, is adopted to capture deep and rich semantic information from user's reviews. Due to the diversity and flexibility of natural language, the same or similar reviews are often described in different sentence structures or expressed in casual manner [6]. It is obvious that two different sentences with the same meaning often have the unique AMR graph or the same AMR subgraph. For example, Figure 1 (a) shows the common semantic structure of AMR graph of Example 1 and 2. In AMR graph, some words which do not contribute to the meaning of a sentence, such as function words (''to'', ''the''), are omitted from the sentence. We can see that the depth between ''battery'' and ''camera'' in AMR graph of Figure 1 (a) is one while in dependency tree of Figure 1 (b) is two. This means semantic representation has deep semantics. Furthermore, the semantic relation ''domain'' between ''camera'' and ''battery'' can be captured by AMR subgraph, which means semantic representation has rich semantics.
To improve the performance of aspect extraction, a hybrid rule-based approach with the capacity to capture surface structure and deep semantics called Deep2S is proposed. Deep2S is based on Answer Set Programming (ASP) [7] framework -a variant of Logic Programming. Since the nonmonotonicity and scalability of ASP, this paper uses ASP rules to represent syntactical rules and semantic rules. Specifically, DP and rule selection-DP (RSDP) [8] are employed for syntactic rule pattern to acquire surface structure information. And AMR is used as semantic rule pattern to capture deep and rich semantic information.
The major contributions of this paper could be summarized as follows: (1) We first propose to employ ASP to implement semantic structure based aspect extraction by adopting AMR graph to represent the semantic relations between aspects and opinion words. Then, we propose Deep2S, a hybrid rule-based approach to improve the performance of aspect extraction by combining syntactic and semantic rules. (2) Deep semantic representation such as Abstract Meaning Representation, is explored to capture deep and rich semantic information in customer reviews. This paper hypothesizes that the semantic relations between aspects and opinion words or other entities can be captured by AMR subgraphs.
(3) In our experiments, we evaluate our proposed method on eight popular aspect extraction evaluation datasets.
The experimental results have shown that deep semantic representation is useful, although its performance is not as good as syntactic structure of dependency grammar. Moreover, our method Deep2S which integrates shallow structure and deep semantic information, can improve aspect extraction. Our code is publicly available at https://github.com/njirene/Deep2S. In the remainder of this paper, section 2 mentions related work. Section 3 introduces Deep2S model. Subsequently, section 4 presents experiments. Finally we conclude with a summary and some future research directions in section 5.

II. RELATED WORK
In this section, we first introduce applications of syntactic structures, and then semantic structures in NLP tasks.
With respect to aspect extraction in opinion mining, there are two major approaches, one is rule-based approach and the other is deep learning approach. The former is mainly based on syntactic or dependency structure of each sentence to design some rules for extracting aspects [4], [8], [16], [18]- [22], while the latter is mainly based on syntactic or dependency structure of each sentence to yield embeddings or learned features [17], [23]- [28].

1) RULE-BASED APPROACH
Rule-based approaches which are usually unsupervised and free from expensive manual annotation, are widely adopted in aspect extraction.
Qiu et al. [4] proposed a double propagation approach to perform opinion lexicon expansion and target extraction tasks given a set of seed opinion words. This approach employed VOLUME 8, 2020 a dependency grammar to describe the relations of opinion targets and opinion words.
Liu et al. [8] proposed an automated rule set selection approach (RSDP) to automatically select a subset of high quality rules from a set of syntactical extraction rules based on dependency relations.
Liu et al. [16] utilized DP approach as baseline and improved the syntactic rule-based method by similarity-based and association-based recommendations.
Wu et al. [22] proposed a hybrid unsupervised method to perform aspect term and opinion target extraction by combining chunk-level rules based on dependency relations with machine learning methods.
As we all know, there are a lot of works focusing on syntactic rules which are proven to be helpful to improve the performance of natural language processing applications. However, little research has been done on whether semantic rules can contribute to applications such as aspect extraction in opinion mining. In this paper, we will validate whether semantic rules are useful for aspect extraction.

2) DEEP LEARNING APPROACH
Some studies have shown that deep learning approaches can achieve better performance.
Yin et al. [23] utilized the embeddings of words and dependency paths as features in CRF to extract aspects.
Wang et al. [17] encoded high-level features such as the information of dependency paths into RNN to extract aspect terms and the opinions.
Ye et al. [24] proposed dependency-tree based convolutional stacked neural networks to capture the syntactic features of sentences for aspect extraction.
Li et al. [11] exploited multi-order dependency-based word embeddings which indicated that the dependency-based context can capture most of the stable and valuable syntactic features and partial semantic information.
Luo et al. [29] proposed a bidirectional dependency tree network to extract dependency structure features from the given sentences. This method constructed a bidirectional propagation mechanism on the dependency tree to enhance the tree-structured representation.
However, deep learning methods require a great number of data to train models, and are usually uninterpretable. In this paper, we focus on the rule-based approach. Our work is most related to the DP [4] and RSDP [8], both of which are based on syntactical rules. There are two differences between our work and the two works. The first one is that their work only extracts aspects modified by opinion adjectives, while our work can also extract aspects modified by opinion verbs, adverbs, as well as aspects without opinion words. The second difference is that we consider them as our bases and propose to incorporate semantic rules with syntactic rules to improve the performance of aspect extraction.

B. SEMANTIC STRUCTURE BASED APPROACHES FOR NLP TASKS
Semantic structure based dependency parsing aims to produce graph-structured semantic dependency representations of sentences instead of tree structured syntactic dependency representations.
Abstract Meaning Representation as a form of deep semantic representation, is a semantic representation where the meaning of a sentence is encoded as a rooted, directed, acyclic graph. AMR mainly focuses on content words, some surface words such as function words (''to'', ''the'') which do not contribute to the meaning of a sentence, will be omitted from a sentence. Recently, AMR has been utilized as some NLP tasks, such as entity linking [30], event extraction [31], [32], text summarization [33]- [36], machine translation [37].
Pan et al. [30] adopted AMR to represent semantic information about entity mentions for entity linking.
Li et al. [31] improved event detection by adding AMR features on the basis of the commonly used features (e.g., lexical, syntactic, and entity information). In this task, AMR was adopted to capture the deeper semantic contexts of the trigger words and represent the relations between the event trigger word and entities.
Rao et al. [32] used a deep semantic representation based on AMR to extract events in biomedical text and hypothesized that an event is an AMR subgraph. Based on the assumption, event extraction task is considered as a graph identification problem. And a supervised neural network-based model is trained to identify the event subgraph.
Dohare et al. [34] generated the story AMR graph using co-reference resolution and Meta Nodes for automatic summarization.
Song et al. [37] utilized a graph recurrent network (GRN) to encode AMR graphs without breaking the original graph structure, experimental results on an English-to-German dataset show that the structural semantic information from AMR can improve neural machine translation.
Compared to the conventional dependency parsing, AMR captures deeper and more semantic meaning [36]. To our knowledge, no existing work has exploited semantic structure of AMR for enhancing aspect extraction. To fill in this gap, this paper employs ASP rule based method to explore the usefulness of semantic structure in aspect extraction. That is because the existing studies all adopted neural networkbased methods, no studies use rule-based method to explore the usefulness of AMR in NLP tasks. Thus, we employ deep semantic representation based on AMR to capture deep semantic information.

III. Deep2S MODEL
In this section, we first introduce problem definition where important concepts about Deep2S model are formulated. Then, we describe the entire framework of Deep2S. Finally, we give a detailed introduction of the four steps in the framework, namely, ASP fact generation, semantic rule generation, syntactical rules used in our model and ASP answer set computing.

A. PROBLEM FORMULATION
In this subsection we formulate the problem to be investigated. To illustrate AMR can capture deep and rich semantic information, we start with some basic definitions.
Definition 1 (Identity Tree): For any sentence S, ∃ S, S is sentence space, for ∀ S ∈ S, S =< w, r >. w is a sequence of words, w = w 1 , . . . , w n . r is a set of relations existing between words, r = {r ij |i = w i , j = w j }. There exists mapping f : S → T and its inverse mapping f −1 : T → S, T is tree space. If satisfying T = f (S) and S = f −1 (T ), where tree T ∈ T , and T =< V , E >, V is a set of nodes in T , E is the set of edges in T , then f is the identity mapping of the sentence S, and T is the identity tree of the sentence S. That is, specify that each word w i in S has one-to-one correspondence with a node v in the tree T , v ∈ V , and the relation between words, r ij has one-to-one correspondence with the edge e ij in the T , e ij ∈ E, and vice versa. can be taken as identity tree, since they can be mapped oneto-one with sentences respectively, and vice versa. Consider tree as a special case of graph, but AMR graph is not taken as an identity tree, because AMR graphs cannot be mapped one-to-one with sentences, as is shown in Figure 2  (1) if depth(T ) < depth(T ), namely, the depth of the tree T is less than the depth of its identity tree T ; (2) there exists words w 1 , w 2 , which are mapped as nodes represents the path from node A to node B in identity tree T . Namely, node A in tree T is taken as the starting node, and its path to the designated node B is less than that of the identity tree T .
For example, ''The camera has a good battery.'', the depth of AMR graph (in Figure 2 (c)) is less than that of syntactic tree (in Figure 2 (a)). This means that semantic representation has deep semantics.
then T is considered to have rich semantics over T , if one of the following conditions is met: (1) if enabling the word w i (e.g., buyer) to abstract into an ontology z (e.g., person in Figure 2 (d)) [38], z = g(w i ), z ∈ V , then this node z is considered to have rich semantics; (2) if enabling the implicit semantic relation r ij in S to abstract into ontology z, z = g(r ij ), where z ∈ E , then the edge z is considered to have rich semantics.
For example, ''The camera has a good battery.'', the semantic relation ''domain'' between ''camera'' and ''battery'' is considered as ontology, which can be captured by semantic parser, not by syntactic parser. This means that semantic representation has rich semantics.
Definition 4 (AMR Graph): An AMR graph G = (V, E) is a rooted, directed, acyclic graph, where V is the set of nodes, or equivalently the vocabulary of entity, and E is the set of edges. Each edge in E connects one entity v i ∈ V with another entity v j ∈ V via a semantic relation type e ij ∈ N . N is a set of semantic relations. Each edge in E is denoted as Definition 5 (AMR Subgraph): We hypothesize that semantic relation between aspects and opinion words or other entities is an AMR subgraph. Let M be a set of semantic relations conforming to this condition, which is a subset of N . Let

B. ASP BASED FRAMEWORK FOR ASPECT EXTRACTION
Answer Set Programming originates from non-monotonic logic and logic programming. It is a logic programming paradigm based on the answer set semantics [7], [39], which offers an elegant declarative semantics to the negation as failure operator in Prolog. An ASP program consists of rules of the form: where each l i for i ∈ [0. . . n] is a literal of some signature, i.e., expressions of the form p(t) or ¬p(t) where p is a predicate and t is a term, and not is called negation as failure or default negation. For instance, amr(A, mod, O) is VOLUME 8, 2020 an atom with the predicate amr, and three terms, one constant mod and two variables A and O, respectively. A rule without body is called a fact.
Our proposed model Deep2S is based on ASP framework for extracting aspects. The entire ASP based framework consists of the following steps: (1) Parse review texts, develop algorithms to extract related information and represent them as ASP facts; (2) Construct semantic rules and represent them by ASP rules; (3) Use syntactical rules as base and represent them by ASP rules; (4) Compute the answer set of the logic program resulted from the first and the third steps to perform shallow aspect extraction, and then from the first and the second steps to perform deep aspect extraction using an ASP solver like clingo. 1 By integrating both syntactical-based and semanticbased rules, Deep2S finally extracts aspect terms from the answer set.

C. ASP FACT GENERATION
In this paper, we hypothesize that the semantic relations between aspects and opinion words or other entities can be captured by AMR subgraphs. In order to obtain AMR subgraph, an algorithm called AMR-SE (short for AMR Subgraph Extraction) given in Algorithm 1 is developed. AMR-SE includes two main steps: Step 1 (Lines 1-3): This step is to obtain AMR graph. Line 1 initializes AMR subgraph G s as empty. AMR-SE starts with reading each review sentence S from the training data D. For each sentence S, AMR parser is used to automatically generate meaning representation, namely, AMR graph G = (V, E) (lines 2-3). The time complexity of parsing is O(n), n is the number of the training data D.
Step 2 (Lines 4-18): This step is to identify AMR subgraphs from the AMR graph. AMR subgraph may contain the candidate aspects. After obtaining AMR graph G = (V, E) of a sentence (line 4), we identify AMR subgraphs conforming to semantic relation set M. For each e ij in E (line 5), we do the following steps. If the edge e ij belongs to semantic relation set M, the triple (v i , e ij , v j ) will be added into G s (lines 6-7). Then we search the edges connecting with the nodes v i and v j , respectively. If the edge e ih connecting with the node v i belongs to M, we insert the triple (v i , e ih , v h ) into G s (lines 8-10). If the edge e jk connecting with the node v j belongs to M, we insert the triple (v j , e jk , v k ) into G s (lines [11][12][13]. After adding all nodes and edges related to semantic relation set M into G s , we can obtain AMR subgraphs of the candidate aspects (line 18). The time complexity of identifying AMR subgraphs is O(m), m is the number of AMR subgraphs. Therefore, the total time complexity of Algorithm 1 is O(nm).
After identifying AMR subgraphs from AMR graph, we can represent them as ASP facts. For example, in 1 https://potassco.org/ Algorithm 1 AMR-SE(D, M) Input: Training data D, semantic relation set M Output: AMR subgraph G s 1: G s = (); //Initialize G s as empty 2: for each sentence S in D do 3: parse S into AMR graph G = (V, E); 4: for G = (V, E) of S do 5: for each e ij in E do 6: if e ij ∈ M then 7: insert if e ih ∈ M, when e ih connecting with v i then 9: insert end if 11: if e jk ∈ M, when e jk connecting with v j then 12: insert (v j , e jk , v k ) into G s ; 13: end if 14: end if 15: end for 16: end for 17: end for 18: Output G s as AMR subgraph.

D. SEMANTIC RULE GENERATION
This subsection explains in detail how to generate semantic rules. In order to obtain semantic rules, we first construct the seed rules manually, then propose an algorithm to extend them and obtain the final rule set, and finally detail some examples of the final semantic rules. In the following, we start with several related key concepts used in semantic rules. The first concept is semantic relation. Since AMR can capture the notation of ''who is doing what to whom'', AMR graph is adopted to describe semantic relation of a sentence. It uses approximately 100 relations. And a dozen of them are used in this paper, whose meaning is shown in Table 1. These semantic relations are illustrated by an example of ''I recommend unreservedly this camera to any potential buyer.'' in Figure 2.
The second concept is general word. When AMR parser parses a sentence into its AMR graph, some named entity types such as person, thing (which are the same with general words in [20]) are parsed as concepts. These concepts have relations (e.g., arg1) with other concepts in the AMR graph. In order to improve the performance of aspect extraction, we adopt the non-monotonicity of answer set programming to exclude general words.
The third concept is trigger word. Motivated by the fact that there are some special trigger words such as ''buy'', ''purchase'' which are usually adopted to pair with aspects to express user's reviews, we build a trigger lexicon and use the relations (e.g., arg1) between trigger words and aspects to extract aspects.
The last one is part of speech (POS). POS tags of words are very important for aspect extraction task. For example, aspects are always nouns/noun phrases and opinion words are often adjectives, which have been widely adopted in previous work [4], [20], [40]. We thus impose some constraints of the POS tags on aspects and opinion words.

1) SEED RULE CONSTRUCTION
In light of syntactical rules in DP [4], this paper constructs manually four types of seed semantic rules based on the development datasets. The propagation mechanism in DP is still utilized in semantic rules in the process of aspect extraction. Table 2 displays the four types of seed semantic rules.
R1: extracting the aspects using the given seed opinion words.
R2: extracting new opinion words using the extracted aspects in R1.
R3: extracting new aspects using the extracted aspects in R1.
R4: extracting new opinion words using the extracted opinion words in R2 and the given opinion words.
In the table, column 1 is the rule ID, column 2 is the input and output of rules, and column 3 is the four types of seed rules. In these rules, end if 10: end for 11: Rank r k ∈ ER by presicion; 12: for each rule r k ∈ ER in descending order do 13: insert r k at the end of FRS; 14: end for 15: Output FRS as the final rule subset.

2) FINAL RULE SELECTION
After constructing the seed semantic rules R, the next step is to extend R and obtain the final extraction rules. In order to do this, an algorithm named DeepSem is proposed, as shown in Algorithm 2. It consists of three main steps: Step 1 (Lines 1-4): This step is to obtain the extended rules ER. Line 1 initializes the extended rules ER and the final rule set FRS as empty. For each rule r i in the seed semantic rules R, we expand it by replacing relations REL (including ''NR'', ''Asem'', ''Osem'', ''Sems'' and ''Semf'') in r i with semantic relations M , replacing NN with noun (nn) and noun phrase (np) and replacing JJ with adjective (jj) and adjective phrase (jp) or discarding JJ, and finally obtain the extended rules ER (lines 2-4).
Step 2 (Lines 5-10): This step is to identify the rules which can extract the candidate aspects from the extended rules ER. For each rule r j in ER, perform aspect extraction using an ASP solver based on ASP facts F and rule r j (lines 5-6). If rule r j cannot extract any aspect, delete r j from ER (lines 7-9). Thus the remaining rules can extract one or more candidate aspects.
Step 3 (Lines 11-15): This step is to rank the rules based on their precision. For each rule r k in extended rules ER, compute its precision, follow the descending order to rank these rules and insert r k at the end of FRS (lines [12][13][14]. Then we can obtain the final rule subset. In the following, we detail the semantic rules to extract the aspects in Figure 2 (c) and (d).
Rule1 1 : Rule1 1 means ''if a word A, whose POS is noun (nn), has the semantic relation mod with an opinion word O, and has the semantic relation domain with H , then A is an aspect'', which can be formulated by the following ASP rule: aspect  Figure 2 (c), we can extract the aspect ''battery'', as there is a relation mod between ''good'' and ''battery'' and a relation domain between ''battery'' and ''camera'', ''good'' is an opinion word. Rule1 2 : Rule1 2 means ''if a word A whose POS is noun (nn) is not a general word, a word H is a trigger word, it has the semantic relation arg1 with its direct object A and has the semantic relation manner with an opinion word O, then A is an aspect''. This rule can be represented as an ASP rule: aspect  Figure 2 (d), we can extract the aspect ''camera'', as there is a relation arg1 between ''recommend'' and ''camera'' as well as a relation manner between ''recommend'' and ''unreservedly'', ''recommend'' is a trigger word, and ''unreservedly'' is an opinion word.

E. SYNTACTICAL RULES
In this work, syntactical dependency relations between opinion words and opinion targets are adopted to extract opinion targets. Syntactical rules from DP [4] and RSDP [8] are used as the base in our work. Detailed information about these rules can be found in [4] and [8].
In the following, we detail the syntactical rule for extracting the aspect in Figure 2 (a). For example, one syntactical rule from DP [4] is that if there is an adjectival modifier relation (amod) between A and O, where A is a noun and O is an opinion word, then A is identified as an aspect. This can be represented as an ASP rule  Figure 2 (a), we can extract the aspect ''battery'' as there is a relation amod between ''good'' and ''battery'', ''good'' is an opinion word, and ''battery'' is a noun.

F. ASP ANSWER SET COMPUTING
After obtaining ASP facts and ASP rules, we compute the answer set of the logic program using an ASP solver. Then the aspect terms are extracted from the answer set.
The following is an example to illustrate how our logic framework works for aspect extraction. For the sentence ''The camera has a good battery.'', the POS facts of its words and the syntactic facts between words are automatically acquired by Stanford Parser, 2 and the semantic facts between words are automatically obtained by JAMR 3 [41]. These facts are represented by program P 1 . Then, let P 2 be the program of the ASP rules. Based on rules Rule1 1 , Rule2 1 and facts f 1 − f 14 , we run an ASP solver like clingo to compute the answer sets of P = P 1 ∪ P 2 . Finally, the solver outputs a unique answer set including all atoms in P 1 and the aspect will be extracted from the sentence, namely, aspect(''battery'').

IV. EXPERIMENTS
We now evaluate the proposed method, first introduce the experiment datasets, evaluation metrics and comparative methods, then detail experimental results.

A. DATASETS
Three publicly available aspect-annotated corpora are used. The first one is from [40]  . The third one is from [8], including reviews of three products: computer (D9), wireless router (D10), and speaker (D11), which are used as the development datasets to construct the seed semantic rules manually. The seed opinion words of the first and second corpus are provided by [40] and the original annotated datasets, respectively. Table 3 shows the detailed information of each review dataset. For the first corpus, due to the small size of each dataset, we have followed the same way of cross domain test as adopted by [8]. Namely, for D1 to D5, leave-one-out cross validation is utilized. For example, the annotated data from D1 to D4 is used for selecting rules, then the selected rules are used to test D5. According to [8]'s statement, the rules selected through this way can be proven to be domain independent. For the second corpus, the train-test split is the same as the original dataset, as shown in Table 3.

B. EVALUATION METRICS
We have adopted precision, recall, and F 1 -score as our evaluation metrics. For the comparison, we choose the same two ways used by [8] to compute the extracting results: (1) based on multiple occurrences of each aspect term, and (2) based on distinct occurrence of each aspect term.
In customer reviews, users often review an opinion target many times, for example, the target ''camera'' is frequently mentioned in camera dataset. For (1), if any occurrence of ''camera'' is extracted, it means that all occurrences of ''camera'' are identified. If no one occurrence of ''camera'' is identified, it means that all the occurrences have been lost. For (2), if any occurrence of ''camera'' is identified, it is regarded as one extraction. If no one is identified, it is regarded as one loss. Both of the above evaluation metrics have their own importance. In (1), the most frequent opinion targets are considered with respect to their frequency and it means that identifying (or losing) the most frequent opinion targets is identifying (or losing) the most important opinion targets. And in (2), all the opinion targets have the same importance.
As evaluation measures for aspect extraction, we employ true positive (TP), false positive (FP) and false negative (FN) values to calculate precision, recall, and F 1 -score. To calculate these values, let A be a set of extracted aspects and let T be the set of manually annotated aspects. Hence, TP is For (1), we follow the same precision, recall and F 1 -score metrics as adopted by [8].
where f i is the term frequency of a i , E(a i , A) (or E(a i , T )) equals to 1 if a i is an element of A (or T ), otherwise E(a i , A) (or E(a i , T )) equals to 0. For (2), the standard evaluation metrics are adopted as follows, F 1 -score is the same as in (1): To validate the performance of our proposed method Deep2S on aspect extraction, we compare Deep2S against the syntactical rule based baselines (e.g., DP, DP+, RSDP, RSDP+) as well as the state-of-the-art and most recent baselines (e.g., TF-RBM, CNN-LP, R-DNN). We compare with DP, DP+, RSDP and RSDP+ because we utilize DP, DP+ rules and RSDP, RSDP+ rules as the part of our method. DP and DP+ refer to the original double propagation algorithm in [4], and utilize 8 aspect extraction syntactical patterns. The difference between them is that the former uses 8 dependency relations (i.e., mod, subj, s, obj, obj2, pnmod, desc and conj), and the latter adopts 18 dependency relations (i.e., amod, advmod, nsubj, csubj, xsubj, dobj, iobj, prep, conj, dep, cop, mark, nsubjpass, pobj, acomp, xcomp, csubjpass, and poss), both of which are defined in Stanford Parser.
RSDP and RSDP+ perform aspect extraction by selecting the optimal rule subset from all the rules of DP and DP+ using the greedy algorithm, respectively. The two baselines are proposed in [8].
DeepSem utilizes the final semantic rules to extract aspects.
Deep2S-DP combines DeepSem and all the rules of DP.

Deep2S-DP+ combines DeepSem and all the rules of DP+.
Deep2S-RSDP combines DeepSem and all the rules of RSDP.
Deep2S-RSDP+ combines DeepSem and all the rules of RSDP+.
TF-RBM is a sequential pattern rules-based aspect extraction approach proposed in [6].
CNN-LP is a supervised approach which combines deep convolutional neural network and linguistic patterns to improve the aspect extraction performance, which is proposed in [42].
R-DNN is a hybrid unsupervised approach which combines chunk-level linguistic rules and deep neural network for aspect extraction, which is proposed in [22]. Table 4 shows the comparison of precision of our proposed method Deep2S (e.g., Deep2S-DP, Deep2S-DP+, Deep2S-RSDP, and Deep2S-RSDP+), compared with the syntactical rule based baselines (e.g., DP, DP+, RSDP and RSDP+), the semantic rule based baseline (e.g., DeepSem), the sequential pattern rules-based baseline (e.g., TF-RBM), as well as the baseline combining rules and neural network (e.g., CNN-LP). Similarly, Table 5 shows the comparison of their recall and Table 6 shows the comparison of their F 1 -scores. For each method, column 1 labeled ''mul'' refers to the first evaluation metric, namely, based on multiple occurrences of each aspect term, and column 2 labeled ''dis'' refers to the second evaluation metric, namely, based on distinct occurrence of each aspect term. All these methods utilized the same resources, as we have followed, thus, comparison of these methods is achievable.

D. EXPERIMENTAL RESULTS
In our experiments, DP, DP+, RSDP and RSDP+ are the bases, our approaches (e.g., Deep2S-DP, Deep2S-DP+, Deep2S-RSDP, and Deep2S-RSDP+) combine DeepSem with them. From the three tables, we observe that our approaches perform more consistently for all evaluation metrics of both multiple and distinct evaluations and produce better results than the bases. In Table 4, for both D1 to D5, and D6 to D8, Deep2S-RSDP gets the best average precisions. In Table 5, for both D1 to D5, and D6 to D8, Deep2S-DP+ gets the best average recalls. In Table 6, for both D1 to D5, and D6 to D8, Deep2S-RSDP+ gets the best average F 1 -scores. It is obvious that our approaches outperform the bases significantly, which proves the semantic rules can mine deep and rich linguistic information hidden in user's review sentences. This semantic information can help improve the performance of aspect extraction. For example, in Figure 1 (a), the semantic relation ''domain'' which means ''battery'' is an aspect or attribute of ''camera'' can be captured to help extracting aspect. From Table 4, Table 5, and Table 6, we can see that the average results of Deep2S-DP and Deep2S-DP+ are higher than those of DP and DP+. And the average results of Deep2S-RSDP and Deep2S-RSDP+ are higher than those of RSDP and RSDP+. Compared with the corresponding four bases on two evaluation metrics, for D1 to D5, the F 1 -scores of Deep2S-DP, Deep2S-DP+, Deep2S-RSDP, and Deep2S-RSDP+ improve by 4.4% and 4.2%, 1.5% and 5.1%, 5% and 6.3%, 2.8% and 4.6%; for D6 to D8, improve by 1.2% and 1.6%, 0.2% and 0.9%, 2.5% and 3.5%, 1.4% and 1.7%, respectively. This means that our methods adding semantic rules are markedly better than the bases which only use syntactical rules in shallow structure.
In order to validate where the performance gain is coming from, we conduct the experiment of DeepSem. As we all know, the improved performance may be affected by the two factors -one is the use of semantic information, and another is the additional number of rules. From the three tables, we observe that for D1 to D5, the average precision results of DeepSem are higher than those of DP and DP+, while the average recall results of DeepSem are higher than DP and lower than DP+. This is expected because DP+ has more rules than DeepSem. The average F 1 -scores of DeepSem are higher than both of them. For D6 to D8, the average precision results of DeepSem are higher than DP and DP+ while its recall results are lower than both of them, and the average F 1 -scores of DeepSem are slightly less than them. This shows that DeepSem may capture the deep and rich semantic knowledge of those review texts such as grammatically incorrect sentences or excessively long sentences.
These three tables show the comparison of our best method Deep2S-RSDP+ with the sequential pattern rulesbased approach (i.e., TF-RBM) and the approach combining linguistic rules with neural network (i.e., CNN-LP). Deep2S-RSDP+ performs better performance than all of them in terms of recall and F 1 -scores but drops only a little in precision below CNN-LP. The precision of Deep2S-RSDP+ decreases 2.4% compared with CNN-LP but recall improves by 6% and the F 1 -score by 2%. As compared to TF-RBM, Deep2S-RSDP+ shows almost the same in precision but 0.7% and 3.2% improvement in recall as well as 1.2% and 1.9 % improvement in F 1 -scores, respectively. Another recent approach combining linguistic rules with neural network (i.e., R-DNN) is employed to compare with our best method (Deep2S-RSDP+), as shown in Figure 3. We can observe that Deep2S-RSDP+ outperforms R-DNN significantly in both D6 and D7. Specifically, Deep2S-RSDP+ performs better performance than R-DNN in terms of precision and F 1 -scores but drops a little in recall. Compared with R-DNN, the recall of Deep2S-RSDP+ decreases 7.2% and 2.5% but precision improves by 12.9% and 23.2% and the F 1 -score by 2.4% and 10.0%, respectively. We believe one of the key reasons is that our method uses semantic rules to capture deep and rich information such as ''domain'' which R-DNN may be not.
From Table 4, Table 5, and Table 6, we can see that the precisions of Deep2S-RSDP and Deep2S-RSDP+ are dramatically higher than those of Deep2S-DP and Deep2S-DP+ on every dataset, respectively. In particular, for D1 to D5, the average precisions of Deep2S-RSDP and Deep2S-RSDP+ improve by more than 14% on both evaluation metrics. For D6 to D8, the average precisions of Deep2S-RSDP and Deep2S-RSDP+ improve by more than 10% on both evaluation metrics. And the final F 1 -scores of Deep2S-RSDP and Deep2S-RSDP+ are markedly better than those of Deep2S-DP and Deep2S-DP+ respectively. Although there is some loss in recalls, this is expected because Deep2S-RSDP and Deep2S-RSDP+ have less syntactical rules. These results are in accord with [8]. We further see that the recall of Deep2S-RSDP+ is improved on almost every dataset compared with Deep2S-RSDP, although the precision of Deep2S-RSDP+ decreases slightly on some datasets. The average F 1 -score of Deep2S-RSDP+ is improved compared with Deep2S-RSDP. This shows that Deep2S-RSDP+ is able to take good advantage of the additional syntactical rules (in RSDP+), maintaining a high recall with a little loss in precision. We also observe that the overall recalls of Deep2S-DP and Deep2S-DP+ are higher than Deep2S-RSDP and Deep2S-RSDP+, this is expected because Deep2S-RSDP and Deep2S-RSDP+ have less syntactical rules.
In summary, our experiment results show that both semantic rule based method and syntactic rule based method provide valuable information for aspect extraction. Our best method Deep2S-RSDP+ on two evaluation metrics outperforms the state-of-the-art syntactical rule based method RSDP+, and the sequential pattern rules-based method (e.g., TF-RBM), as well as the methods combining rules and neural network. This result indicates that the semantic rules can be utilized in conjunction with the syntactic rules to obtain the best performance. This is consistent with previous study [43], where semantic structures were combined with syntactic structures through embeddings for DDI extraction. Since the explainability and scalability of rules, our proposed method can combine syntactic structures with semantic structures by rules. We can conclude that the proposed approach can capture shallow structure and deep semantic information, and can improve the performance of aspect extraction.

E. ANALYSIS
Semantic structure-based methods are hypothesized to have less rules, since the sentences with the same meaning but different syntactic structures such as Example 1 and Example 2 have the same semantic representation of AMR graph, as shown in Figure 1.
To validate this hypothesis, we compare semantic structure-based method (e.g., DeepSem) with syntactic structure-based methods (e.g., DP, DP+, RSDP, RSDP+). Note that DP is the general rule while DP+, RSDP, and RSDP+ are the refined rules of DP. The datasets used in our experiment have more or less the same rule number, so we take the dataset D1 as an example. As shown in Figure 4, the rule number of DP is 8, fewer than that of DP+ (521), RSDP (36), RSDP+ (270), respectively. We can see that the refined rules (e.g., DP+, RSDP, RSDP+) require more rules than the general rule (e.g., DP). DP+ has more rules, it achieves higher recall and lower precision than RSDP, RSDP+, as shown in Figure 5.
Note that RSDP, RSDP+ and DeepSem use the same greedy algorithm to select the final rule set. We can observe that DeepSem has less rules than RSDP, RSDP+ and it achieves higher recall than both of them. This may be because DeepSem can capture deep semantic relation (e.g., ''domain'') between concepts (''battery'' and ''camera'') in the different sentences such as Example 1 and Example 2 which have the same semantic structure.  DeepSem achieves lower precision than RSDP and RSDP+, but higher precision than DP and DP+. The results using semantic structure alone are not promising as we expected. One reason might be that the semantic structure discards some surface linguistic knowledge such as some surface words and dependency relations which may be important for extracting aspects. And another reason might be due to the limitations of semantic structure parser such as JAMR which cannot deeply capture noun-noun or noun-adjective relations. And these noun-noun or noun-adjective relations are useful for aspect extraction.

V. CONCLUSION AND FUTURE WORK
In this paper, we first propose to employ ASP to explore the usefulness of semantic structure of AMR in aspect extraction. Although its performance is not so good as syntactic structure of dependency grammar, the experimental results demonstrated that ASP could be used effectively and concisely for the study of semantic structure. Then we propose a hybrid rule-based approach with the capacity to capture surface structure and deep semantic for improving the performance of aspect extraction in opinion mining. The proposed methods are capable of employing syntactical rule-based approach to capture surface structure information, and exploiting semantic rule-based approach to capture deep semantic information. Compared with the baselines, our proposed method produces better results on two evaluation metrics. For example, our method outperforms the best syntactical rule based method by 2.8% and 4.6% for D1 to D5, 1.4% and 1.7% for D6 to D8, respectively. We also compared them with the sequential pattern rules-based baseline and the state-of-the-art statistical based baselines. The experimental results demonstrated their superior performance. Our findings suggest that semantic structure is a powerful semantic formalism which holds potential for aspect extraction task.
In our future work, we plan to employ semantic rules to extract implicit aspects and validate the usefulness of deep semantic information using the datasets in other languages like Chinese. We also plan to explore dependency based embeddings and semantic based embedding to extract aspects. Moreover, we will explore the distinct compositional strategies to combine semantic and syntactic structures as well as part of speech.
XUELIAN LI received the B.S. degree in information and computing science and the M.A. degree in applied linguistics from Nanjing Tech University, Nanjing, China, in 2010 and 2013, respectively. She is currently pursuing the Ph.D. degree in computer science and technology with Southeast University, Nanjing. Her search interests include opinion mining, information extraction, natural language processing, artificial intelligence, and semantics.
BI WANG is currently pursuing the Ph.D. degree in computer science and technology with Southeast University, Nanjing, China. His current research interests include reinforcement learning, natural language processing, and artificial intelligence.
LIXIN LI received the B.S. degree from Soochow University. He is currently pursuing the M.S. degree in computer science and technology with Southeast University. His current research interests include sentiment analysis, natural language processing, and artificial intelligence.
ZHIQIANG GAO was born in Hebei, in 1966. He received the B.A. and M.D. degrees from Southeast University, and the Ph.D. degree from the Department of Mechanical Engineering, Tsinghua University, in 1995.
After graduation, he was worked for National CAD Application and Training Network-Nanjing Center, as well as the Department of Mechanical Engineering, Southeast University. As a Postdoctoral Researcher, he went to Japan to do researches on digital city and artificial intelligence with the Department of Social Informatics, Kyoto University, in 2000. He has been an Associate Professor with the Department of Computer Science and Engineering, Southeast University, since October 2002. In December 2007, he joined the Institute of Web Science (IWS) to study Ontology Learning and Natural Language Processing (NLP), as well as large scale reasoning. He is interested in almost everything, such as material science and manufacturing, numerical simulation of heat transfer and fluid flow, CAD, CG, visualization, VR, game theory, agent oriented programming, semantic Web, knowledge capture, web mining, military simulation, as well as philosophy, traveling, basketball, and Taijiquan. He was concentrated in human-like and conversational agent, multi-agent systems, agent oriented programming and machine learning. In 2003, a project was launched about interactive multi-agent system used for shooter training simulation, in which agents are hosted by Augmented Reality. He has to construct 3D buildings and terrains, design and debug scenarios, and recognize behavior of shooters in the real world. Although, this project is more complicated than designing ordinary CG games, it is much more interesting and challenging. He is also involved in 973 projects for semantic grid. Recently, he is responsible for two NSFC (National Science Foundation of China) projects, which are related to ontology learning and inductive logic programming. Speech recognition, text understanding, and image recognition are faced with the same puzzles, and natural language processing is one of the best test bed for various innovative AI ideas. Machine Learning may play a key role in these fields in the future. Additionally, semantic annotation of 3D environments for multi-agents to plan and act rationally is an important branch of Semantic Web. He firmly believes that a new era for search engine based on deep NLP, Machine Learning, and large scale reasoning is coming.
QIAN LIU was born in 1986. She received the Ph.D. degree from Southeast University. She entered the Nanjing University of Posts and Telecommunications, in 2016. Her research focuses on artificial intelligence and natural language processing, with applications in opinion mining and relation extraction. Her ongoing research focuses on complex non-monotonic information extraction tasks.
HANCHENG XU received the M.A. degree with Xi'an International Studies University and the Ph.D. degree with Shanghai International Studies University. He completed his Postdoctoral ship with Heilongjiang University. He is currently a Professor with the International Studies College, National University of Defense Technology. His research interests are computational linguistics and traditional Russian language studies and teaching.
LANTING FANG received the Ph.D. degree from Southeast University, Nanjing, China, in 2018. She is currently a Lecturer with the School of Cyber Science and Engineering, Southeast University. Her current research interests include NLP, information extraction, social media, and sentiment analysis.