A Multi-Objective Differential Evolutionary Method for Constrained Crowd Judgment Analysis

,


I. INTRODUCTION
Crowdsourcing-based annotation [1]- [3] systems have already been successfully established as an effective tool for solving different complex real-life problems in cost-effective and time-efficient manner. It is demonstrated that proper utilization of non-expert human beings can reduce the overall cost of hiring the experts while maintaining the quality. Given a collection of opinions provided by crowd workers, producing an aggregated judgment from their diverse opinions is a challenging task. Due to the existence of multiple nonexperts, retrieving appropriate opinions in a proper way is very necessary, in order to filter the malicious crowd workers. In our day-to-day life, it is often required to obtain the decisions from crowd and finally reach into an aggregated The associate editor coordinating the review of this manuscript and approving it for publication was Gang Li . decision from these set of opinions. For example, in several real-life scenarios, it is needed to obtain the information about the public demands for some specific resources in a city. But gathering this information in a rapid way as well as receiving the actual public perceptions over the necessity of these resources, are difficult tasks. However, this problem can be addressed very easily if this is outsourced to the general crowd.
During the last couple of years, vast online communities i.e., social media play a larger role to express public opinions [4], [4]- [6]. On the other hand, soliciting crowd opinions for proper decision making and planning is found to be very interesting to the researchers [7]- [10]. A spectrum of algorithms has been developed to find the aggregated judgment from multiple crowd opinions [11]- [14], however, the questions tackled by the crowd in these situations are consisting of a single component. To explain this with an VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ example, for sentiment analysis task from tweets [15], [16], the possible options of a particular tweet can be 'Yes', 'No', or 'Skip', etc. The question considered in this situation is basically consisting of unit component and there exists no sub component in it. But in various real-life applications, it can be observed that the posted question can have multiple components instead of a single component. Moreover, there can be certain constraints among the components and these constraints are needed to be maintained by the crowd workers while registering their opinions. Meanwhile, majority of crowdsourcing based work solve the question with single component, so minimal research has been performed addressing these issues. These types of issues create a new avenue of research to apply in various real-life problems having question comprising multiple components. Moreover, if the number of components becomes large then it leads the aggregation problem to be more difficult maintaining the constraint satisfying criteria.
In constrained judgment analysis [17], the main distinctions from the normal crowd opinions are that there are some pre-defined options sets similar like 'Yes', 'No' and 'Skip' in normal scenario. However, this problem becomes difficult when there is no perfect knowledge about the options available. For example, consider a situation where an organization is trying to allot a few resources in multiple places (as an example k places) in a locality. Finding out the appropriate locations according to real public necessity along with the proper demographic information in a very limited time is not a simple task. Moreover, obtaining the flavor of exact public choices or gathering information towards public inclination requires a huge amount of time and cost. Apart from that, a specific distance between any two facilities (e.g., post offices, fire stations, ATMs) is needed to be maintained as professional planning. It is due to that the objective of the organization is to make the resources available to the great extent of people as much as possible. Thus, maximizing the area coverage and minimizing the overlapping zones between the coverage of any pair of resources are two of the main criteria in designing professional planning. Therefore, obtaining all the information regarding public choice is a hectic task. So, this task can be addressed in easier manner if this problem can be outsourced to the general crowd and the opinions are obtained from them promptly. Here, the constraint is to maintain a specific distance by the crowd workers for any two facilities.
To exemplify the above situation, with the vast advancement of social media, someone can post a question online requesting 2D coordinates of k best possible locations in an area for providing some facilities subject to some constraints. Here the constraint is the distance between any pair of facilities must be greater than a certain threshold value. Thus, the crowd opinions are of the form of k 2D coordinate values and each 2D coordinate is termed as a component. However, the main challenging job is to produce the aggregated opinions from the constrained opinions. Similarly, in this current research this constrained crowd judgment problem is studied by analyzing two case studies employing the power of crowd. The first case study is to identify the best possible locations for installing three ATM counters at Ulsan National Institute of Science and Technology (UNIST) Campus while open innovation from public is employed in order to reach into a prompt and quality consensus decision. Here mainly, the question posted is seeking three 2D coordinate values from crowd for the three possible locations by showing a grid map of the campus. Moreover, there should be certain distances between any two locations, thus, it serves as a constraint to the crowd workers to be maintained while providing their responses. Here, each of the 2D locations is considered as a component, thus, the posted question comprises multiple components which are not considered in traditional crowd-based judgment analysis problems. On the other hand, the second case study is to identify three best possible locations in India to set up three study centers of a Top-ranked US university and the opinions are collected by demonstrating a grid map of a state of India [17]. Thus, the aim is to find an aggregated decision from multiple constrained crowdsourced opinions.
As the generalized version, the problem can be extended to any k components instead of three while the 2D coordinate values can be multi-dimensional. In Section III, the generalized version of the constrained crowd judgment analysis is discussed in more formal way. Here, in this model exploiting collective intelligence from crowd in open innovation context is important because every crowd worker has his/her own perception according to their thought. So, imposing competitive scenarios to select a winner from crowd workers and designing a proper reward mechanism is not needed here. Importantly, this constrained judgment analysis problem can have wide applications in other domain also like healthcare and travel tourism industry [18]. To illustrate, in travel itinerary planning, obtaining crowd knowledge prior to the journey can be very effective. In this situation, along with receiving multiple travel plans (i.e., sequence of appropriate visiting places) from crowd, satisfying the time constraint between any two subsequent visiting places within a minimal budget is very important in order to produce an appropriate tour planning. Hence, modelling proper constrained judgment analysis can be very effective not only in the smart-city planning but also in other various domains.
The state-of-the-art crowd judgment analysis models [6], [12]- [14], [16], [19]- [27] basically compute the consensus decision from multiple crowd workers. But in those cases, the option sets were not so complicated. The general judgment analysis algorithms as well as simple majority voting cannot be applicable here in this constraint judgment setting. The reason is that due to the existence of multiple components of a question, a simple majority voting through componentwise cannot be performed there. This is since componentwise majority voting cannot guarantee the satisfiability of the constraints. Moreover, as the option set is not provided (only the ranges of each coordinate values for each component are available), hence like other state-of-the-art methods we cannot apply the weighted voting because it needs to find the weight of each option. But in our case, the options sets are not defined accordingly. Again, the existing probabilistic models cannot be applied there as it needs to compute the posterior distribution of each option considering the accuracy of workers. Thus, limited research has been performed to tackle the crowd opinions which contain multiple components with constraints for a single question. This motivates us to derive a method that can tackle the constrained judgment analysis of crowd and solves a real-life problem to make a proper city planning. A recent work [17] dealt with the problem of constrained crowd opinions and found better solutions from crowd opinions. In this work, a grid map was demonstrated and the opinions about the appropriate locations to install different facilities are solicited from crowd. In this problem, as there is no perfect option set available so only the ranges were considered. However, to apply posterior distribution of each option it was needed to define the option set here. To define the option set here Bayesian binning [28] was used, although this method assumes some bins to define the option set which can change the crowd solutions from their original form to keep the similar kind of opinions in the same bins. Moreover, the various aspects like considering overlapping area between any pair of facilities, and penalty value causing pointing out a location in a prohibited place (like inside lake, forest, etc.) are not considered here.
In this current research, a multi-objective formulation of this problem is discussed, and we solve it using Nondominated Sorting Genetic Algorithm (NSGA-II) [29] based differential evolution algorithm (DE) [30], [31]. DE is considered to be one of the powerful and simple stochastic global optimization algorithms without requiring much resources [31]. The strategy specifically employing the differential mutation operator is very effective in discovering many promising solutions and thus it enhances the capability of searching new solutions reducing the search space [30], [32]. Thus powerfulness of DE algorithm compared to other metaheuristics is due to the ability to combine different solutions having a difference factor rate of chosen individual solutions. Most importantly using DE, the stage of defining option set (as discussed in the previous paragraph) is bypassed to produce the solutions directly from the crowd opinions. In this way, it reduces the extra computational burden of performing Bayesian binning for defining the option set. Another important point is that in presence of multiple objectives, optimizing one objective function may often lead to degrade the quality over other objective functions, so a compromise is needed. Hence in this work, we optimize the two objective functions simultaneously and try to reach into a good tradeoff between the objectives. The first objective function is the sum of three individual sub functions based on the crowd opinions. The three individual sub functions are (i) coverage area enclosed by the possible locations pointed out by the crowd workers, (ii) overlapping zones by the pair-wise locations, and (iii) the penalty value incurred due to pointing out the location in any prohibited place. Here the first sub function is the maximization function and the second and third sub functions are of minimization nature. Maximization of the first sub function means that the resources are distributed in homogeneous manner so that maximum people are served from those resources. On the other hand, we consider another objective function i.e., the deviation of each individual crowd solution from the mean of the crowd opinions in order to remove any solution which is too much dissimilar than the average opinion. This second one objective function is to minimize the deviation of the solution from the mean opinion of all the original crowd opinions.
In this context, if the number of crowd becomes very small then it becomes highly difficult to obtain a final decision. Again, optimizing one objective function can degrade the quality in terms of other objective functions in unacceptably manner, hence optimizing multiple objectives concurrently is necessary. Multi-objective optimization [33], [34] is a variant of the class of problems under the multiple criteria decision making (MCDM) problem where there are several alternatives and they are evaluated by specific attributes. Another class of problems under MCDM is multiple criteria evaluation (MCE) [35]- [37] and here in presence of limited number of alternatives, the best alternative is chosen based on a single objective function depending on multiple criteria. Thus, the difference of our work from MCE class of problems is that here multiple objectives are optimized simultaneously to make a better trade-off. Furthermore, in this proposed model, several solutions are generated guided by the crowd and finally a set of near optimal non-dominated Pareto front solutions are found from which the decision makers can choose the suitable final solution. There is limited research that focuses on utilizing multi-objective DE in order to obtain better judgment analysis from constrained crowd opinions. We apply the proposed model in two real-life crowd judgment datasets and in-depth analyses have been performed. The experimental analysis exhibits that the model is effective to derive better solutions from original crowd opinion thus provides decision makers to prepare proper planning within less time and feasible budget. Finally, the general applicability of this constrained crowd judgment model in other areas is discussed.
The main contributions of this work are described below.
• We propose a multi-objective method of a recently introduced crowd judgment analysis problem termed as 'constrained judgment analysis' [17]. Here, the two main objective functions are simultaneously optimized to find out a set of non-dominated solutions.
• It is demonstrated that the need of defining option sets are not required here. Interestingly, we also show that we can generate useful solutions even when the number of crowd workers (especially satisfying the constraints) are very limited.
• The effectiveness of the proposed model is demonstrated by two case studies where the crowd opinions are collected for smart city planning. In both cases the derived solutions after applying the proposed method are of VOLUME 8, 2020 superior quality than any of the individual original crowd solutions. The rest of the paper is structured as follows. In Section II, we provide the description of state-of-the-art methods dealing with crowdsourcing based annotations. We describe the problem formulation in Section III. Section IV is devoted to present the methodology used to solve this problem and we demonstrate the experimental analysis and result in Section V. Finally, Section VI concludes the paper mentioning some future research works.

II. RELATED WORKS
Since the last decade, crowdsourcing based annotations are found to be very effective to solve different real-life problems within limited time and cost. A spectrum of algorithms have already been devised in order to obtain better decisions from crowd opinions [1], [12]- [14], [35], [38]. Among different popular crowdsourcing platforms, Amazon Mechanical Turk (AMT) is one the most popular platforms to solve different real-life tasks by employing vast human power. Although there are multiple benefits to solve a complex problem by utilizing the vast human knowledge available in market. However, there are certain risks to collect their opinions owing to several important issues like reliability, uncertainty and presence of spammers, etc.
Due to the vast advancement of social network and internet availability, outsourcing a problem to anonymous crowd instead of internal sourcing to designated workers found to be advantageous for various firms [24]- [27]. There can be different forms of crowdsourcing based innovation such as, collaborative, competitive, and complementary innovations. Additionally, crowd labour market can act as intermediaries to match skill and task as third party. Still, managers of these firms need to be reasonably cautious before posting the job to numerous strangers and hence proper understanding about the suitable form of crowdsourcing is mandatory. In this article [39], authors offered a couple of guidance to select the most suitable form of crowdsourcing environment according to their business policies and budget. Nevertheless, the independent crowd responses are often unsatisfactory and inconsistent, therefore, introducing collaboration among them and forming small teams to solve tasks is found to be interesting [40]. Another work [41] explains that how perceptual diversity over task representation degrades the decision quality of crowd and a pre-defined rating scale is used to judge the quality of them. In this model, to elicitate the crowd response experts were employed and the decision on quality were evaluated based on multiple criteria. Therefore, the main motivation and the shape of crowd responses in these abovementioned works are different than ours (i.e., dealing with constrained crowd opinions without any defined option sets), thus the challenges to aggregate them should be addressed in different way.
As mentioned in a nutshell at introduction, majority of the state-of-the-art research solve the crowd opinion problems that deal with the opinions having binary or multiple opinions [16], [42]. In binary opinion problems, there exist two options like either 'Yes' or 'No', whereas in the multiple opinion datasets the options are like 'Yes', 'No', 'Skip', 'Unsure', 'I can't tell', etc. However, there exist several crowd workers who maliciously provide wrong answers and try to manipulate the process. Thus, filtering out these crowd workers and concurrently retrieving the best possible answer is really challenging. In spite of numerous research [11]- [13], [19], [43], [44] dealing with the crowd opinions there are various complex real-life problems where these types of methods are not applicable. As an example, there are several urban city planning scenarios where crowd opinions are highly needed to be aware of personalized recommendation regarding installation of any facility in an area [7]- [10].
In traditional crowdsourcing based judgment analysis models, the posted question has only one component so there is no question of any relationship between any pair of components. However, in this current type of problem, the question may comprise several components each describing the different possible locations of the facility. Moreover, there is a relationship between a pair of components because proper planning ensures that there will at least a certain distance between any two facilities. Thus, these certain types of constraints are needed to be strictly followed by the crowd workers. Interestingly, the main challenge is to find an aggregated judgment from the diverse crowd opinions which comprise multiple components [17]. The reason is because there is a rare chance of repetition of same opinions for each component (i.e., for any particular facility). So, we cannot apply majority voting there to find the aggregated judgment. Besides it, the component wise majority voting can be applied but this does not ensure the satisfiability of the constraints. Thus, there is an enormous scope to introduce new models to address these complex types of issues.
Recently, one work reveals the qualitative and quantitative results specially for document relevance judgment task to demonstrate how crowd workers behave changing their strategies to finish the task efficiently [45]. In this work the authors mainly focus to observe at which task-level work strategies crowd workers are found to be more effective instead of preferring the reputation of requester or population bias. This work uses a 4-level relevance scale (e.g., (i) Not relevant (ii) Partially relevant (iii) Relevant (iv) Highly relevant) to provide judgment score of each document in order to distinguish the crowd workers' behavior. Moreover, the crowd workers were rewarded by a certain amount of payment hence it is not voluntary. In our model the opinions collected from crowd in voluntary manner. Therefore, the wide difference between this work [45] with our proposed work is motivation. Along with that, this work employs a predefined option set i.e., a 4-level relevance score scale which is not available (i.e., no defined option set) in this present work that is challenging. In addition to that, constrained opinions of crowd workers are not considered here. Similarly, in other work [38] collecting rationales behind the crowd relevance judgment while soliciting their responses is further studied in order to characterize the judgment disagreement between crowd and experts. In this work also, the same 4-point judging scale as discussed above is used and crowd opinions are not like constrained opinions with multiple components. Besides it, in most of the crowd opinion-based work, it is not mandatory that all the crowd would provide feedback for all the questions. It is observed that some specific crowd workers tend to response some specific questions, hence, they can be clustered based on their attempted feedback. In this work [46], a biclustering approach is provided to deal with independent crowd judgment analysis problem but all the responses here are from a predefined option set consisting of binary or multiple opinions. Again, in this work the questions considered have a single component with some specific predefined options (i.e., either 'Yes', 'Skip' or 'No'). Thus, there is no question of constrained opinions of crowd that clearly demonstrates the novelty our work from it.
From the perspective of DE algorithms, the focus of the current research is to avail the effectiveness of DE in deriving better solutions by reducing the search space while the constrained opinions are solicited from crowd. Although various DE algorithms are proposed [32], [47], [48], however, in this current context, the objective is to utilize DE algorithm from multi-objective perspective to tackle the crowd judgment analysis problem. Due to the various challenges present in crowdsourced opinions, therefore, the various steps of the proposed DE are needed to be designed carefully considering the factors specific to problem. Interestingly, these stateof-the-art DE algorithms do not consider the constrained opinions of crowd in order to solve a real-life problem. As explained before, due to the unavailability of the predefined option set (as discussed in I. INTRODUCTION), we cannot apply DE directly on the original crowd opinions. Moreover, as there is no predefined ordering of the various components (i.e., possible locations) so label correspondence is also needed. Thus, the problem formulations as well as different steps along with the objective functions are focused to solve this new domain of constrained crowd judgment analysis task and thus it differs from the abovementioned state-of-the-art literature. In general, the initial populations are created based on random solutions, whereas in this proposed model the initial population are generated randomly but based on the supervision of crowd response. Furthermore, it is always required to relabel, and scale of the solutions evolved during the overall process of the algorithm in order to restrict the generation of infeasible solutions and it is explained elaborately in subsequent sections. Therefore, in this work, motivated by these issues, we address the problem from multi-objective perspective to consider multiple objectives simultaneously and solve it using a multi-objective differential evolution (DE) algorithm.

IV. PROPOSED MODEL
In this section, the proposed differential evolution-based method is presented. The different steps of the proposed model for the aforementioned problem is discussed in detail hereafter.

A. RELABELING OF INCONSISTENT CROWD SOLUTIONS
Here, we first introduce the main challenges due to the presence of multiple diverse crowd opinions while there is no predefined option set, rather, only the ranges of coordinate values are presented. The objective here is to bring the correspondence among the corresponding crowd workers solutions to make standardization between the components.
In the proposed model, the opinions are solicited from crowd workers about the best k possible locations by VOLUME 8, 2020 demonstrating a 2D grid map. Therefore, each of the crowd workers provides opinions that consist of 2D coordinate values of k locations. Crowd workers provide the solutions from their own perceptions and knowledge. Now, as there is no such predefined ordering of the locations to the crowd workers, therefore two different crowd workers' solutions might look different even if they are same. For example, in Fig. 1, the solutions provided by the crowd workers 1 and 2 appear to be different although they represent the same solution. Note that, the first location given by crowd worker 1 is same as the second location given by crowd worker 2. Again, the first location of crowd worker 2 is basically the same solution as the third location given by crowd worker 1. Therefore, proper relabeling is needed to be done to bring the correspondence between these solutions.
To keep proper correspondence between these solutions, first the most appropriate solution among all the crowd solutions is selected. Then it is chosen as a reference solution and the other solutions are relabeled based on the reference solution. To select the most appropriate solution, the first objective function is considered as it has multiple factors and the solution with maximum value is chosen. To perform the relabeling, each of the components of any particular solution is compared with the first component of the reference solution based on Euclidean distance. Now the component for which the distance becomes minimum, is selected as the first component of that solution. Then rest of the other components, except the first component (as the first one is already standardized), of the solution are compared with the second component of the reference solution. The same process is repeated and the component having minimum distance is chosen as the second component of the particular solution. The process is repeated for all the components and in this way, they are standardized. This same process is repeated for all the solutions and after this process is over, all the solutions become ready for further processing.

B. PROPOSED MULTI-OBJECTIVE FORMULATION
This section illustrates the utility of NSGA-II [29], [49] for generating a set of near Pareto-optimal solutions. Thereafter the proposed methodology is described in step by step manner.
Genetic algorithm [49], [50] is an effective heuristic search algorithm that follows the concept of Darwinian evolution. In this heuristic, a population is randomly made, and it is generated by the chromosomes. These chromosomes are basically the candidate solutions and the parameters related to the search space are encoded by the chromosomes. An objective function is used to measure the goodness of the solutions and this function is needed to be optimized. Different genetic operators like crossover, selection and mutations are applied for evolving subsequent generations. The algorithm stops when the convergence criteria is met, or the maximum number of generations is reached.

It involves finding a vectorx
. . , x * n ] T of decision variables satisfying some equality and non-equality constraints while optimizing a vector functionf (x) = [f 1 (x), f 2 (x), . . . , f r (x)] T subject to some constraints. In this formulation, constraints denote the feasible region that contains the competent solutions. Herex * represents the optimal solution. In case of a minimization problem, Pareto optimality is defined as: A decision vectorx * is considered to be Pareto optimal if and only if there exists nox such that ∀i : i ∈ Differential evolution (DE) [30], [51] is a widely known population dependent global optimization process. The optimization step of DE begins with a population of individual solutions of the underlying optimization problem. It enhances the quality of the solutions in iterative manner by adopting two operators namely, mutation and crossover. The traditional simple DE starts with a population created randomly with real-valued encoded chromosomes. Here the chromosomes signify the candidate solutions for the above-mentioned process, and it improves the candidate solutions in an iterative way. In every generation, the current vectors (i.e., the candidate solutions) are replaced by the new vectors with better objective values. Finally, the process terminates when the convergence criteria is met. DE is arguably considered to be one of the most powerful stochastic real-valued optimization techniques. The comprehensive discussion on DE is available in [30], [51], [52] which has been applied in different problems like feed forward neural network [53], electromagnetic [54], clustering problems [55], economic load-dispatch problem [56], etc. In this work, instead of traditional simple DE, we are proposing a multi-objective version of DE where two main objective functions are simultaneously optimized for finding a trade-off between these two in order to obtain better solutions from constrained crowdsourced opinions. The different phases of the proposed model are discussed hereafter.

1) ENCODING SCHEME
The crowd workers are needed to select k locations for the corresponding facilities according to their wide viewpoints. As each solution consists of 2D coordinate value in twodimensional space, therefore, for k locations there are 2k real values present in the chromosome. In this type of problem, a grid map specifying the coordinate values is demonstrated to the crowd workers in order to obtain their opinions. Basically, X min , X max , Y min , and Y max are shown to crowd workers to collect their possible opinions. So, each of the crowd solution is confined within the specified ranges between the two coordinates (i.e., X and Y coordinate). The idea behind bounding this value within this range is because some random solutions may fall outside of the range of possible locality (already mentioned at the time of collecting opinions) and thus it may evolve some infeasible solutions. The encoding scheme of chromosome to keep information in allocating k resources for 2D coordinate values is shown in Fig. 2.

2) INITIAL POPULATION
The solutions obtained from the crowd workers while demonstrating the grid map are treated as the base solutions of the initial populations. In addition to that, in order to reduce any kind of bias towards any particular solution some random solutions are also generated. These random solutions are basically guided by the crowd workers' solutions. While producing the random solutions, in order to avoid generation of infeasible solutions, the maximum and minimum value for each coordinate of all the locations are considered. Then, the random solutions are generated by bounding these values within these specific ranges. To bring the correspondence between the labels of the newly generated solutions the label correspondence method (as described in Section IV-A) is applied on it.

3) MUTATION
In DE, the mutation operator is applied first and thereafter the other genetic operators like crossover, selections are employed. In this step, each chromosome passes through the mutation process and an individual is mutated based on some other individuals randomly selected from the population. In this strategy, for each gene the mutation is performed using the following formula.
Here, X r 1 ,G , X r 2 ,G and X r 3 ,G are three randomly selected individuals, where r 1 = r 2 = r 3 = i. The i th mutation vector is expressed as Here, F ∈ [0, 1] is the extension or contraction factor which governs the step of exploration direction.

4) CROSSOVER
Crossover is a probabilistic technique that interchanges genetic information between any two parent chromosomes. Here, a fixed crossover probability p c is used. In this context multi-point crossover is performed by using a binary mask.

5) SELECTION
This stage selects the chromosomes for further breeding inspired by the concept of ''survival of the fittest''. According to the binary tournament selection strategy, tournaments are played multiple times to choose the suitable chromosomes from the initial population. However in this current scenario, crowded binary tournament selection strategy is operated as mentioned in NSGA-II [29] to maintain the diversity of new population.

6) CHOICE OF OBJECTIVES
In this crowd judgment problem, there are mainly two objective functions. The first objective function basically comprises three sub functions. These sub functions are not dependent on the original crowd workers (while computing the values of these functions) once their opinions are collected and hence, they are coupled in a single objective function. This coupled function is treated as the first objective function. Here, the first sub function is the coverage of the polygon area enclosed by the k points of facilities in a particular crowd solution. This is a maximization function as greater the polygon area means that better the solution. The reason is that greater the polygon area reflects that the possible k facilities are well distributed such that maximum people are covered by these crowd solutions. The second sub function depends on sum of the overlapping areas covered by any two locations of facilities. This function is a minimization function as minimization of overlapping zone also signifies the possible facilities can reach as much as people. Although, we may think of that the coverage function and overlapping area are seemed to be similar, however, they are distinct. For example, if there are three facilities and even though they are non-overlapping, but the area enclosed by the three locations can be different due to their distances from each other. Finally, the third sub function is the penalty incurred by the solutions for indicating possible locations inside some prohibited places. This situation happens when some one provides location that is inside a lake, forest or sports ground causing to generate an infeasible solution. Thus this crowd response suffers with a penalty value, so this function is also a minimization function. Therefore, the first objective function is the summation of these three sub functions.
On the other hand, the second objective function is the deviation of the solution from the mean of all original crowd solutions. Hence this objective function value is dependent on the crowd solutions and this objective function is employed with an aim to remove any kind of bias towards a particular solution. Therefore, this second objective function should be minimized as any solution which is very much diverse with respect to the mean solution may not be a good solution, rather it should be treated as a noisy solution. Finally, these two objective functions are optimized simultaneously in an iterative manner. The mathematical formulation of the objective functions is expressed below.
Let the encoding scheme of a chromosome C symbolizing k different locations be {x 11 , x 12 , x 21 , x 22 , . . . , x k1 , x k2 } in 2D space. Now, suppose the area enclosed by the k locations in C is given by A and the total overlapping area by the k possible locations is denoted by O. For some specific restricted places within a range, the solutions incur a penalty value and suppose the total penalty value is denoted by P. The objective 1 is defined as max To find the overlapping area enclosed by any two facilities i and j located at (x i1 , x i2 ) and (x j1 , x j2 ), we need to consider three cases based on their radii, i.e., (i) when there is no overlapping, (ii) when there is overlapping and (iii) when one circle fully contains the other one. The formula for this is mentioned below.
Suppose, d ij is Euclidean distance between the points (x i1 , x i2 ) and (x j1 , x j2 ) then the distance d ij denotes the metric to check whether the distance between the centers of the two locations is greater than the sum of their radii considering r 1 and r 2 as the corresponding radii. Hence if t > 0 then the overlapping area a ij for the two points located at (x i1 , x i2 ) and (x j1 , x j2 ) is calculated using the formula defined below in Eqn. 2.
Therefore, the total overlapping area for n th crowd solution having k components (i.e., for k locations) is denoted as A = (k−1) i=1 k j=i a ij , where i, j be the corresponding facilities. The third sub function is penalty P for pointing out a location in some prohibited places. Although coordinate values of the prohibited places are not exposed to the crowd workers while obtaining their responses. Nevertheless, it is expected these should be easily realized for a worker from the demonstrated grid map, which can be a combination of land, forest, lake, etc. Therefore, the penalty value is obtained while anyone provides the option installing a facility inside the lake, forest, etc. Let the penalty value incurred by the i th component of k th solution is denoted as p k i and this is applicable for a certain range such that a < x k i1 < c and b < x k i2 < d where (a, b) and (c, d) are the two coordinate values (i.e., minimum and maximum values of the 2D coordinates surrounding the place) of a prohibited area. Thus the total penalty value P is calculated as P = m k=1 p k i , where m is the number of facilities i.e., components. Thus the overall objective function 1 is To describe the second objective function, suppose, the mean solution M is represented as {m 11 , m 12 , m 21 , m 22 , . . . , m k1 , m k2 } for k components (i.e., k facilities). To calculate the Euclidean distance δ between a reference chromosome having coordinate values for k locations i.e., {(x 11 , x 12 ), (x 21 , x 22 ), . . . , (x k1 , x k2 )} and M is δ = (x 11 −m 11 ) 2 +(x 12 −m 12 ) 2 +. . .+(x k1 −m k1 ) 2 +(x k2 −m k2 ) 2 , then the second objective function F 2 is to minimize δ i.e., F 2 = min |δ|. Here, the constraint is the distance between any two facilities which must be greater than a userdefined threshold value α. Therefore, it can be expressed that if d ij be the distance between any two facilities i and j then ∀i, j, d ij > α.

7) REPAIRING SOLUTIONS
In every generation, during the different processing steps a chromosome traverses through different genetic operations like crossover and mutation. Thus, the different cell values of it are transformed to a new set of values. However, according to the crowd opinions there are some possible range of values which are the feasible regions. So, altering these values to new values may cross the limit of the feasible coordinate values perceived by the crowd workers. Hence, it is very necessary to repair the values within a specified range guided by the crowd workers. Before the repairing procedure, the new solutions are relabeled using the original crowd solutions (as mentioned in section IV-A) to make the correspondence between them. This scaling procedure is described below.
If S = {s 1 , s 2 , . . . , s m } be the coordinate values (e.g., either X or Y ) of m workers (while collecting opinions from crowd at the beginning of applying the algorithm) for a specific facility (i.e., either 1 st ATM counters or other) and A = {a 1 , a 2 , . . . , a m } be the vector with m values for the same specific coordinate (e.g., either X coordinate or Y coordinate) for the location obtained after applying different genetic operators. Then we need to scale down the values of A based on the values of S to prevent the generation of some values which surpass the maximum and minimum range of coordinate values guided by the crowd workers. Thus, the formula to adjust the values of each a i of A to a new value a i is written below.
In this way, all the coordinates values for each of the possible locations are repaired. Finally, these solutions are combined with parent population utilizing the concept of nondominated Pareto front and crowding distance. The algorithm is terminated after a certain number of iterations while a particular objective function is converged. The workflow of the overall process is depicted in Fig. 3.

V. EXPERIMENTAL DESIGN AND RESULTS
In this section, the two synthetic datasets used in this paper for the experimental purposes are described. The first dataset used is prepared here and the second dataset is collected from [17]. The experiments are performed in MATLAB 2013a and the environment is an Intel(R) CPU 2.4 GHz machine with 8GB RAM running Windows 10.

A. DATASET PREPARATION
In order to obtain the constrained crowd opinion dataset, we made an open online platform and the questions were posted there. As an example to prepare the first dataset, a grid map of Ulsan National Institute of Science and Technology (UNIST) campus is demonstrated in an online forum and the question is like that ''An organization wishes to install three ATM counters inside the campus of UNIST and what will be the possible three locations to install the ATM counters according to your own perspective?''. The constraint mentioned here is that there should be at least 20 unit apart between any two ATM counters. There are 24 crowd workers who responded in the online survey and out of them 3 persons have provided their opinions only for one ATM counter. On the other hand, another one has registered but provided no opinion for the question. As a consequence, these 4 noisy workers are removed from the opinion set and it can be observed that there are 20 crowd workers who were considered for further computation. Among these 20, 2 crowd workers violated the corresponding constraint. The grip map of UNIST showing the coordinate values along X and Y axis is demonstrated in Fig. 4.
After posting the question, the opinions are collected from the crowd workers after a certain time-period. Here while attempting the question, the crowd workers are also needed to satisfy the constraint. Moreover, as the opinions are collected from the individuals, therefore, it reflects their personal perception over the actual demands and suitability of it from the geographical point of view. This particular region contains a diverse type of natural places including lake, forest, mountains and sports grounds as well. Therefore, some crowd workers (mainly the local residents) find their difficulties resolved if they individually provide their opinions. For example, a few crowd workers may feel that there is a need of an ATM counter adjacent to the bus stop or other important places. However, the opinions may differ from each other. On the other hand, many spam crowd workers can be present there in order to make some malfunctioning by providing a large set of opinions without having a perfect knowledge over the locations. Therefore, in that situation, there is a possibility of violating the constraints.
In order to prepare the second dataset [17], an open online platform was made to solicit crowd opinions over some questions. For example, a grid map of a particular state of India was posted as an image in this platform. Then a question is provided mentioning that ''A top-ranked US university wishes to open three extension centres in the state and what will be best appropriate locations for this?''. It is also mentioned that there should be a distance of at least 30 units between any two locations. In this dataset, 20 crowd workers provided their responses and 18 among them satisfied the constraint. Here also to provide the better opinions the domain knowledge regarding the geographical characteristics along with their perceptions are required. Additionally, due to the presence of spammers the constraints condition can be violated.
In this type of dataset, the opinions collected from the crowd workers basically reflect the actual demand of the facility in that location. Some workers may choose the location nearest to their hometown instead of thinking the global perspective. Again, some workers may try to annotate the question within a very short time to establish themselves as experts so they can violate the constraints. In this way, a diverse set of crowd opinions were obtained. Note that, in this proposed model, we can generate a large pool of other solutions guided by the crowd workers and refine it based on the different objective functions. Therefore, it can be useful while the number of crowd workers is less, and the algorithm is applied over a large pool of newly generated initial solutions. VOLUME 8, 2020 B. PARAMETER SETTING The number of components of the questions are fixed for all the crowd workers for these two datasets. In the proposed algorithm crossover rate is fixed at 0.9, mutation rate is 0.01, extension factor is 0.5, population size is 100 and number of generations is 80. These are selected empirically.

C. STUDY ON THE DATASET
In the experimental phase, the opinions are received from the crowd workers and they provide their best possible opinions voluntary according to their own perception. In this way, for both the datasets, we obtained 24 and 20 crowd workers opinions, respectively and noisy workers (who provide their opinions only for one component instead of all) are removed from that. Then, these opinions are processed further in the later phases. Although we obtained these number of crowd opinions, we can generate any number of random solutions guided by the crowd workers and improve the solutions in step-wise manner. For each of the dataset here, we generate another 80 and 82 random solutions, respectively. To generate these random solutions, the maximum value and minimum value of each of the components perceived by the crowd (i.e., for any particular coordinate of a possible location) are taken into consideration and these random solutions are generated based on these values. In the subsequent phases, the solutions are relabeled based on a reference solution to bring correspondence between them. A constrained satisfying solution from crowd with highest value with respect to the first objective function can be chosen as the reference solution. To make the standardization of the crowd solutions all the other solutions are relabeled using the reference solution. Furthermore, while applying the other genetic operators we need to repair the chromosomes (as described in section IV-B7), otherwise this can lead us generating a large number of out-of-range infeasible solutions. Thereafter, the proposed multi-objective DE algorithm is applied, and the non-dominated solutions are obtained. Finally, all these solutions can be treated as the final solutions. In city planning, the preference of the facilities may change over the variation of ages. We collected the demographic information like age and gender of the crowd workers. As most of the crowd workers for this first dataset are students, therefore, the age range varies between 20-30 years. The histogram plot of age and the pie chart revealing the contributions from different gender are demonstrated in Figs. 5 and 6.
The non-dominated rank-1 and rank-2 solutions after various generations for the first dataset are plotted in Fig. 7. The solutions are generated after 40, 60 and 100 generations. Here, after applying the algorithm the non-dominated rank-1 and rank-2 solutions are found out and then the constraint satisfying solutions are filtered. After that, the solutions are plotted fixing the first objective function along the X-axis, while keeping the second objective function along the Y-axis. As mentioned before, the first objective is a coupled function with coverage, overlapping zone, and penalty value,  while the second objective function is the deviation of a particular solution from the mean solution. In this regard, as shown in Fig. 4, the penalty zones are considered to be the areas inside the lake, sports ground, forest, etc. The penalty value is kept as 0.5 for pointing out the location in any prohibited place. The same non-dominated rank-1 and rank-2 solutions for the second dataset are plotted in Fig. 9. The solutions plotted here are produced after 40, 80 and 100 number of generations. Finally, the solutions which satisfy the constraints are selected. It is noticed that although the algorithm begins with 100 number of responses guided by the crowd workers, at the last phase the actual number of constraints satisfying solutions is reduced to lesser than 100. In this process, the elitism operation is automatically performed while combining the parent population with the child population at each generation. In addition to that, in this proposed model, we include the best fit chromosome by relaxing the objective 2 and preserve that solution holding the highest value in respect of first objective function for the next generation. Furthermore, while combining the parent 87656 VOLUME 8, 2020   population with child population, we restrict to consider only the constraint satisfying solutions in order to ensure a better solution in terms of objective functions as well as feasibility.
To analyze the performance of the proposed model, in terms of multiple objectives as well as constraints, we compare the generated solutions with the original crowd solutions. Since there is no ground truth available so we can consider all the crowd solutions for the comparison purpose. The solution which has higher value in terms of first objective function and lower value for second objective function that solution can be treated as the good solution. To the best of our knowledge, there is no available constraint judgment analysis model (without specifying any kind of binning) which tries to improve the constrained opinions of crowd, thus we cannot compare this with other existing algorithms. However, we can test the performance of the proposed model with respect to all the crowd solutions. The quality of individual crowd solution when compared to the whole set of crowd workers for both of the datasets are reported in Tables 1 and 3. To measure the performance of the proposed method, 6 best solutions (when compared to all the original crowd solutions) after applying the proposed model are selected and the two   Tables 2 and 4 for the two datasets, respectively. The best solution is chosen based on the first objective because it can be realized that the first objective contains three important sub functions, while the second one contains only one. So, the first objective should be prioritized more. It is seen that in Table 1, the best solution given by crowd in terms of first objective has the value 1.1997 and the second objective function has value 0.0008. Importantly, we can understand that a solution with less deviation from mean crowd solution confirms that it is not too much diverse than all of the original crowd solutions. It can be noticed from Table 2, one solution produces 2.3841 as first objective, whereas 0.0006 as second objective. Thus, this obtained solution surpasses the original crowd solution in terms of both objective 1 and objective 2. On the other hand, this same solution also outperforms the solution 4 (having lowest value in terms of objective 2) of Table 1 in terms of first objective in a great extent (i.e., (2.3841 -0.5750 = 1.8091)), although the second objective is very close to the original crowd solution. Meanwhile, for population 80, there is a solution having objective values 1.1974 (first objective) and 0.0003 (second objective), thus VOLUME 8, 2020  it outperforms the solution 4 of Table 1 which has minimum value in terms of objective 2. Moreover, all these solutions are constraint satisfying solutions and many better solutions can be obtained from these crowd solutions for better customization later. Thus, this shows the effectiveness of the proposed model on tackling this type of problem with a better trade-off of these two objectives. From Table 3, it can be observed that for the second dataset the best individual crowd solution (in terms of objective 1) is solution 1, however, there are many solutions obtained after applying the algorithm (as shown in Table 4) which have superior values with a high difference in terms of objective 1. However, the value in terms of second objective is better in original crowd opinions. Even though the margin of difference is high for first objective, whereas, it is very minimal for the case of second objective. Consequently, this shows the applicability and success of the proposed model in order to obtain better crowd judgment analysis. We have also performed the experiments by varying the extension/contraction factor within the ranges [0.5-0.8] to study the performance of the proposed method. Keeping a higher value near to 1 over this factor may cause premature convergence of the algorithm, thus we kept the value as 0.5 while showing the experimental results in Tables 2 and 4. The experimental results for the two datsets presented in Tables 5 and 6 while we execute it for different populations by varying the extension/contraction factor. It can be noticed that in most of the cases the superior quality solutions are obtained from the pool of solutions.
To further study the performance of the method over first dataset according to the results reported in Table 5, it can be noted that all the top-6 solutions generated for population size = 60 and generation number = 100, have greater objective 1 value than any of the original crowd solutions (as mentioned in Table 1). It is easily realized that among the original crowd solutions, the solution 1 in Table 1 has a good compromise between two objective values. We cannot treat the solutions 4, 5 and 6 of Table 1 as good enough although they have very low values in respective of objective 2, due to the poor quality values in terms of objective 1. There are many solutions which have far better values than them (i.e., solutions 1, 2 and 3 in Table 1) in terms of objective 1. Hence, the aim is to find a set of good solutions in terms of both the objectives and the decision makers can have multiple choices to select the final one among them. To study the performance we focus on Table 5, where we obtain a solution having objective 1 value 2.1315 and objective 2 value 0.0006 (for population size = 60, number of generation = 100, extension/contraction factor = 0.6) and this solution has also better values in both objectives than the solution 1 of Table 1. From Table 5 (for population size = 40, generation number = 40 and extension/contraction factor = 0.5), it is seen that many of the constraint satisfying solutions are of better quality in respective of both the objectives than any other original crowd solution for first dataset. To explain in more depth, in Table 5 (for population size = 40, generation number = 40 and extension/contraction factor = 0.5), even the least good solution (i.e., solution 6) has better objective values in both objectives than the good solutions mentioned in Table 1.
Similarly, to discuss the results for second dataset as reported in Table 6, we first notice the goodness of the original crowd solutions as mentioned in Table 3. It can be seen that the best solution according to first objective is solution 1. However, there exist some solutions (i.e., solutions 4, 5 and 6) which have better values in terms of objective 2 but they have poor values in terms of objective 1. Therefore, these three solutions cannot be treated as the promising solutions. From Table 6, after applying the proposed model we see that there are multiple solutions which have far better values with respect to objective 1 than any original crowd solutions as described in Table 3. Furthermore, they have very close values in terms of objective 2 with solutions 1, 2 and 3. As an example, the solution obtained in Table 6 (for population size = 40, generation number = 100 and extension/contraction factor = 0.5), has objective 1 value as 2.0445 which is greater than any of crowd solutions with a high margin as mentioned in Table 3. However, the second objective function is better in the original crowd solution but with a very narrow margin. As we cannot consider the solutions 4, 5 and 6 of Table 3 due to worst values in objective 1, therefore, solution 1, solution 2 and solution 3 can be treated as good. Hence it can be seen that the solution of Table 6 having far better value in objective 1 produces very close value 0.0006 with these solutions (i.e., solutions 1, 2 and 3 of Table 3) in terms of objective 2, thus, the margin of difference becomes very less. Interestingly, we obtain another solution i.e., solution 1 (for population size = 60, generation number = 100 and extension/contraction factor = 0.6) which has better value than the solution 1 and solution 2 of Table 3 in terms of both the objectives. Likewise, we can observe there exist multiple solutions that can be a good trade-off to balance both the objectives. Hence, after applying the proposed model over the different datasets in different experimental scenarios many promising solutions with a good compromise between two objectives are achieved. Thus, these experimental results demonstrate the well tradeoff between the different objective functions to reach final judgment as well establish the applicability of the proposed model.
We visualize the raw data obtained from crowd workers for both the datasets in Figs. 8 and 10. The first plots of both the figures show the original crowd opinions for the three locations (abbreviated as loc) over the two datasets. From first plot of Fig. 8, it can be seen that the crowd opinions are well distributed and diverse. From these solutions we generate a large pool of solutions and the rank-1 and rank-2 constraint satisfying solutions are plotted. From the middle and right plots of both the Figs. 8 and 10, it can be observed that some solutions are concentrated towards some specific locations after applying the algorithm due to filtering the solutions which are too distant from the mean solutions of original crowd opinions. VOLUME 8, 2020   As mentioned previously, to the best of our knowledge there is no other method dealing with the constrained opinions of the crowd workers (without considering binning) where the option sets are undefined. Hence, the method cannot be comparable with other state-of-the-art methods. Even if the binning is performed to define the option set, based on this, it may distort the original crowd solutions as some solutions are changed into other values based on the bin widths. Nevertheless, we try to compare the performance of the proposed model with respect to majority voting. Although we cannot directly apply majority voting over these constrained crowd opinions owing to the presence of diverse knowledge of crowd and there is a very less chance of repeating the same coordinate values for all the components (i.e., for all the ATM counters) even from two workers. This situation appears to be more difficult if the number of components becomes high as in that case there will be very less chance to repeat the same opinion for all the components even from two workers. Moreover, we cannot apply majority voting irrespective of all the components, rather, it can be applicable component-wise after some preprocessing. To apply majority voting in order to find aggregated judgment from constrained opinions, first we need to define some bins and convert the crowd opinions within the values of that ranges specified by the bins. Note that, in this process, the actual crowd opinions are changed to some nearest integer values depending upon the bin width. Hence, some original crowd opinions which were previously constraint satisfying after this phase may violate the constraints. Another important issue is to find the perfect number of bins for combining some crowd workers' responses in order to define the option set. In this work, we apply two kinds of binning procedures e.g., using (i) Square-root formula [57] and (ii) Doane's formula [57]. We cannot use Sturges' formula [57] as it performs poorly when the number of opinions is less than 30. The steps to perform the majority voting are mentioned below.
• First, the number of bins for all the crowd opinions of a dimension (i.e., either X or Y coordinate) for a particular component is decided. Note that, this same process is repeated for the other dimensions of that component and in this way, the number of bins for all the dimensions of all the components (i.e., for each X and Y coordinates of all the three locations) are calculated. Initially, the number of bins is computed using Squareroot formula. The formula for finding the number of bins is b = √ (n) , where n is the number of crowd responses. In this process, the number of bins is same for all the dimensions of all the components (i.e., for all the X and Y coordinates of the three locations) because the numbers of crowd responses are same. As another method, we also apply Doane's formula to find the appropriate number of bins in order to group the crowd workers' responses. The formula for finding the number of bins is expressed as follows: b = 1 + log 2 (n) + log 2 where b is the number of bins, g 1 denotes the third moment skewness of the distribution and σ g 1 can be defined as below. In such a manner, the crowd opinion dataset is made applicable for executing majority voting and it happens due to the merging of some crowd responses after applying this binning procedure. While performing binning, it is seen that for the first dataset using Square-root formula, the number of bins is 5 irrespective of all the dimensions for all the components. On the other hand, the numbers of bins are 4,4,5,4,4,4, respectively for all the 2 dimensions of three components (i.e., total six) when we apply Doane's formula on the same dataset. To apply binning on the second dataset the number of bins is same as 5 for all the components while using Square-root formula, whereas, it becomes 4 for all the dimensions of each component when Doane's formula is employed. After this binning procedure, we apply majority voting for each dimension of every component (i.e., for six cases) and check the constraint satisfiability of that solution. Then the two objective function values of that solution are computed, and the values are reported. For comparison purpose, the two scenarios of majority voting i.e., performing binning based on equal and unequal number of bins are demonstrated and these results over two datasets are reported in Tables 7 and 8. In our proposed method, as we obtain multiple constraint satisfying solutions, therefore, to pick the best one solution from them for both the datasets, we first select top-6 solutions based on the first objective as reported in Tables 2 and 4. Then we calculate the ratio of two objective functions values (i.e., objective 1 to objective 2) and the solution having maximum value is chosen as the final best solution. Finally, these values are reported in both Tables 7 and 8.
From Table 7 for the first dataset, it can be seen that first objective function value produced by the proposed solution is far better than the other solutions, whereas, the second objective is better in very minute margin by majority voting strategy using Doane's formula. However, we cannot treat that solution produced by majority voting as good because the  first objective function value has very poor quality and even it is worse than top-6 (chosen based on first objective) original crowd solutions reported in Table 1. Interestingly, the solution produced by the proposed method has maximum value in terms of first objective when compared to all the top-6 original crowd solutions (as mentioned in Table 1). Similarly, for the second dataset as shown in Table 8, the proposed solution performs good in respective of both the objective functions when compared to other solutions. The experimental results demonstrate that in both the cases the solutions derived by the proposed method outperform the other solutions obtained by other methods. Interestingly, as the proposed method generates multiple non-dominated solutions with better objective function values than the original crowd solutions (including other methods like majority voting with fixed or variable number of bins) in terms of both the objectives, hence the decision makers have a convenient choice to select any one according to their external criteria. Therefore, this shows the effectiveness of the proposed method to tackle this type of complex problem to produce promising solutions.

VI. DISCUSSION
In this current work, we focus on a real-life problem where the individual perception of human beings is needed to the decision makers for making an intelligent and prompt planning. Along with the pervasiveness of social media, it is now becoming easier to receive public opinions to address different common real-life problems. Although employing the crowd to solve these types of problems is easier but there exist plenty of challenges due to the involvement of malicious crowd workers. Hence, proper aggregation policy is always needed in order to derive the best possible response from the set of crowd opinions. In this work, we discuss a novel judgment analysis model where the question contains multiple sub parts (i.e., components). As described earlier, the question in the first dataset was about to know the suitable location of three ATM counters at UNIST. However, this model is also applicable in other complex scenarios as a generalized version. For example, the number of ATM counters can be anything beyond three, i.e., for any integer k, where k > 3 and thus the crowd opinions can be the 2D coordinate values of k locations. Importantly, in this problem, we consider 2 dimensional coordinate values, but this model can also be applicable for high dimensional environment as a generalized model. Note that, if the numbers of components and dimensions grow large then the aggregation problem maintaining the constraint becomes relatively complex as well. But in this proposed model all the crowd information can be easily encoded in the chromosome depending on the problems while designing the proper objective functions and thus it becomes highly effective to derive final judgment maintaining the constraint without specifying any kind of option set and binning.
The wide applicability of this research can be very further explained in other diverse domains including healthcare as well as travel and tourism. To illustrate, with the advent of digital social media now, it is becoming straightforward to receive alternative opinions from crowd for personalized medicines by posting health symptoms online [18], [58]. CrowdMed is one the popular and widely used platforms among these that utilizes the vast knowledge of human resources to solve healthcare issues [58]. In complex treatment scenarios, multiple medicines are needed to be consumed with perfect time ordering. There are also different side effects due to the change of ordering of the medicines. Hence, the multiple medicines prescribed by the crowd include some proper time sequences, therefore, maintaining the time gap (i.e., the constraint) depending on the composition of the drug is highly needed in order to maintain proper health. Thus, this can also be treated as a constrained judgment analysis problem and aggregating these opinions can be dealt with this type of model. On the other hand, in travel itinerary planning, travelers often depend on others' feedback to obtain prior knowledge about their upcoming travel plan and this is an ordered list of activities with specific time constraint. To exemplify this, one such list of activities can be 'spend 2 hours in the child museum', 'roam at the park for 1 hour' and then 'visit at the waterfall for 1 hour', etc. is one of the possible lists of activities. Hence, the complex constraints in this scenario are total travel time, time sequences and overall budget. In this proposed model, these kind of opinion structures can be encoded in the chromosomes while designing perfect objective functions along with other steps arising from complex crowd opinions. Thus, the model can be applied in other multiple areas where crowd opinions follow different structures. Hence, this demonstrates the generalization and establishes its wide application capabilities in different emerging research domains.

VII. CONCLUSION
Over the last couple of years, crowdsourcing based annotation played an important role in order to solve different complex real-life problems. Several approaches have already been introduced to tackle the different crowd opinions which deal with binary or multiple opinions. However, in this current framework, we utilize the enormous power of crowd with an aim to solve the real-life urban planning problem where a single opinion has multiple components. Moreover, there is a relationship between these components, hence these crowd opinions are termed as constrained opinions. In this current work, we formulate the problem in multi-objective setting and solve it employing the differential evolution algorithm to simultaneously optimize multiple objectives. Although there are large number of benefits of using crowd, there exist substantial challenges due to presence of non-experts malicious crowd workers. In this work, the objective is to find a better aggregated decision from all the crowd responses. In this respect, the coverage area enclosed by multiple facilities, overlapping zone and penalty received for any pair of facilities, and finally the deviation of a solution from mean solution are considered to be an important criterion. The effectiveness of the proposed method is shown by applying it over two crowd-based datasets. In the future, we try to consider other characteristics as quality metric criteria of the crowd workers with an aim to distinguish them in order to find better solutions. Moreover, the applicability of the method in solving other crowdsourcing-based complex case-studies can also be investigated as a future research. At the same time, it is also challenging to obtain the sufficient number of crowd opinions when the number of components grows large. Therefore, it can also be further studied that whether any rewarding scheme can be applied to motivate the crowd to be involved and obtain sufficient number of constrained opinions from them in a limited time.