Advances in e-Participation: A perspective of Last Years

The opinions of citizens are now being given ever-increasing consideration. Today, many government administrations have set up public participation processes as one more of the inputs required to make a decision on several aspects of governance. e-Participation initiatives make it easier for citizens to access such processes. At the present time, there is no clear and accepted field definition due to the wide diversity of theoretical proposals and the interdisciplinary nature of the initiatives, many of which have been developed ad-hoc. This paper reviews the present literature in the field of e-Participation by means of a systematic mapping of the research work carried out in the timeframe 2000–2019, together with some earlier relevant proposals in the area, with the aim of obtaining a conceptual guide to e-Participation components. This review analyses the findings and clusters the results into a conceptual e-Participation framework, which we call ePfw. The results show the diversity of the conceptualizations of many authors (25% on average) in the identification of tools, areas and levels in the field of e-participation and the almost null incorporation of fundamental aspects like trust, security, or transparency. We also found a lack of systems development (13.3%) that would prove and allow the proposed theories to be put into practice.


I. INTRODUCTION
Information and Communication Technologies (ICT) have emerged in the last decades as a force for the engagement of citizens in processes related with policy making, disaster response of governments, and the improvement of spaces for democracy [1]. Many countries have now implemented laws and regulations that allow governments to take actions that involve decision-making through participatory processes with citizens, under the assumption that the increased use of ICT calls on citizens and companies to actively engage in political debate and decision-making processes [2].
According to [2], the concept of e-Government includes governmental websites, social media channels, and other digital services [2]. These e-Services are available in a 24/7 schema, providing immediate access to information The associate editor coordinating the review of this manuscript and approving it for publication was Josue Antonio Nescolarde Selva . at any time [3]. Moreover, they provide improvements in features such as public access to information, democratic deliberation, collaborative environments and transparency. This also happens with public participation, a process through which needs, concerns and values are incorporated into the decision making of governments or corporations. There is no single definition for public participation. In [4] we can read that ''Public participation is the process by which public concerns, needs, and values are incorporated into governmental and corporate decision making. It is two-way communication and interaction, with the overall goal of better decisions that are supported by the public''. Another definition calls public participation ''the participation of various stakeholders in a collaborative process; they can be individuals, citizens' initiatives or common interest groups also known as organized public. Any participatory process should be open to all interested parties, like a wide audience'' [4]. The Federal Austrian Chancellery says that ''Public participation means the chance of all those concerned and/or interested to preset and/or stand up for their interests or concerns in the development of plans, programs, policies, or legal instruments'' [5]. The above definitions have several common aspects, like the diversity of stakeholders (decision-makers and citizens or participants), the collaborative nature of the decision-making, and the bi-directionality of the process. Public participation is a well studied area due to its key role in the global evolution that governments are involved in, aiming at improving the transparency and citizens' trust in their activities.
With the rise of e-Government, traditional public participation processes are combined with the use of ICT as a fundamental support for their stages [6]. The use of ICT tools within the public participation context led to the term ''e-Participation (electronic participation)''. In Macintosh's words, e-Participation means ''ICT-supported participation in processes involved in government and governance. Processes may concern administration, service delivery, decision-making, and policy making'' [7]. Although many authors consider e-Participation as exclusive to domains like e-democracy, e-governance, and e-government, it is known that ''its scope is much broader and encompasses citizens' participation in virtually any public service and not necessarily in the political, or governance related, field'' [8]. During the last two decades, there has been a significant increase in the number of projects related to e-Participation; some ad-hoc supporting tools have also been developed, thanks to funding from various government agencies; however, it is recognized that the research field is still highly fragmented [9] and it is necessary to develop models and frameworks that can reduce this fragmentation.
e-Participation has developed extensively over the last few years, giving rise to a variety of research and implementations, mainly carried out by governmental agencies [10]. Until now, several authors have made contributions with the objective of providing theoretical elements and somehow establishing a common conceptual language to support implementation of e-Participation initiatives. Furthermore, an increasing number of studies have been carried out in various areas, mainly related to social sciences (politics, psychology, sociology, economics, demography, etc.) and computer science (information systems and software engineering), confirming the interdisciplinary nature of this field. This diversity of theories, concepts and application domains has given rise to several bibliographic reviews that have attempted to characterize the state of e-Participation [8], [11]- [14]. Other literature reviews have focused on studying the role played by e-Participation in other e-Government subdomains such as policy making, administration or political perspective. Some current views show, for example, the collaborative functions of e-Participation with social networks [15], [16]. Some authors and organizations consider e-Participation processes as constrained to government-related topics [7], [17]. For instance, e-Participation is often confused with e-democracy by assuming that they both fulfill the same function; however, e-democracy is simply an area of application of the e-Participation principles, as suggested by [11].
The most recent surveys published in the area of e-Participation date back to research carried out up to 2011 [8], [11]. Since then many new contributions by researchers and practitioners have been published that have brought new ideas, methods and procedures to the field of study. In this paper, we perform a complete review of the status of e-Participation, incorporating the most relevant, recent and previous research (2000 to 2019), in order to complement the reviews made by others in the past [8], [11], [13], [14]. Our review aims at providing an updated document to new e-participation researchers and actors and compares the results with its most relevant predecessors. The growing public awareness and the increased participation in various initiatives promoted by agencies around the world justify this update.
To structure our review, we defined a theoretical framework, which we call ePfw, for e-Participation; the framework identifies the main components that interrelate in an e-Participation process. Starting from the idea of the relevance of the ''process'' in this set of elements, we make a conceptual study based on its phases, activities, technologies, methodologies and outcomes. An analysis is also provided of the existing e-Participation and evaluation frameworks in relation to the components and characteristics identified by ePfw. The paper aims to be a theoretical guide for technicians, researchers and practitioners who develop initiatives in the field of public participation, especially in e-Participation. Also, the review carried out in this document is technologyoriented with a view to the future development of technological solutions for e-participation based on the ePfw framework.
This paper is structured as follows: Section 2 describes the systematic mapping method and the criteria used to select the sources. Section 3 identifies the components of the new theoretical framework for e-Participation (ePfw) and describes its main component, which is the process. We also provide a review based on identified ePfwcomponents, such as the levels, areas, methods and/or techniques used, actors, roles and tools. Section 4 describes the existing e-Participation frameworks, a review of evaluation frameworks and an analysis of the characteristics (procedure and technology) identified in the articles detected by the bibliographic mapping. Finally, the conclusions and a roadmap for further research are given in Section 5.

II. METHOD AND SELECTION STRATEGY
The literature review presented here followed a systematic mapping method adapted from the model proposed by [18]. The method specifies the definition of a search protocol with three main phases, namely planning, execution and results.

A. PLANNING STAGE
The planning stage specifies the details of the search protocol that allow the articles to be collected systematically and consists of the following activities: VOLUME 7, 2019 • Establishment of the research questions: The basis of this work is the analysis of the existing relevant bibliography for e-Participation in order to obtain a theoretical framework with the artifacts that compose it and interact in this domain. Several works have studied the various existing contributions [7], [8], [11]- [14], [17], [19], [20]. These contributions have become the primary consultation elements for researchers in the area. However, the study of e-Participation has continued to be the source of new research and interesting findings that have not been compiled in an updated paper. For example, the study of aspects like trust in electronic participation is one of the current relevant topics [21], [22]. Due to the wide variety of e-Participation study areas, this research is based on Macintosh's vision [6] directed towards a specific area, i.e. the inclusion of ICT in participation (e-Participation). In other words, the aim is to obtain a theoretical and analytical framework that can serve as a support for the future development of technological solutions for e-Participation. In this context, our research questions are: • RQ1: What are the components that integrate an analytical and theoretical framework for e-Participation?
• RQ2: What theories, models, and frameworks are put forward in current literature to conceptualize e-Participation?
• Creation of the search string: the proposed protocol specifies the use of a search string. This research used an adaptation of the search string created by [14], also later used and validated by [8], [11], [23]: ''e-Democracy, electronic democracy, digital democracy, democracy AND internet, democracy AND information system e-Participation, electronic participation, e-Government AND participation, e-Governance AND participation e-Voting, electronic voting, internet voting e-Inclusion, digital divide AND participation'' [11]. For this study, specific aspects of democracy and electronic voting that do not come within the scope of this research were eliminated. Subsequently, the following relevant terms were added: ''framework, trust, and tools''. Resulting in the string: ''e-Participation OR (e-Government AND e-participation) OR (framework AND e-participation) OR (trust AND e-participation) OR (tools AND e-participation)''.
• Selection of the information sources: the search takes as primary sources several relevant digital libraries: Springer link, ACM Digital Library, IEEE Xplore Digital Library, Scopus, Web of Science and the specialized E-government reference library. The specified method is complemented by an open search in the various information sources. The most relevant academic journals in the field of e-Government were also selected to make a manual search: Government Information Quarterly (GIQ), Information Polity (IP), Electronic Journal of E-Government (EJEG), Electronic Government, An International Journal (EGaIJ), International Journal of Electronic Government Research (IJEGR), Transforming Government: Process, People, and Policy (TGPPP) proposed by [24] also used by [8], [11]. Besides, we review manually the proceedings of the most important conferences in the area: the Electronic-Government track at HICSS conference, the IFIPEGOV and e-PART conference, DG.O conference and ICEGOV conference.

B. EXECUTION STAGE
This stage was carried out in two phases. The first was through an automatic search (using the search string) in the various bibliographic sources of information. The second stage consisted of a manual search in relevant journals and conferences that were not taken into account by the automatic search. The execution of the search string based on the proposed method produced a large number of bibliographic items. To reduce this number, as well as to avoid overlapping with previous surveys we defined and applied the following activities: • Selection of primary studies: First, a review was carried out to eliminate duplicate papers of the same study in different sources. Additionally, several of the papers obtained by the systematic mapping method may be irrelevant to the investigation, even if the search terms appeared in either the title, summary or both. To reduce this problem, papers were selected manually, carefully reviewing the title and the abstract to preserve only those results that were relevant to the goals and research questions. The papers were then selected according to a defined series of inclusion and exclusion criteria.
• Inclusion criteria: The studies that met at least one of the following inclusion criteria were included: Research papers presenting examples or any empirical studies (e.g. study cases, experiments), on e-Participation. Since the most recent reviews had been published in 2012 [8], [11], we decided to focus on the work done from 2012 to 2019, as well as keeping relevant earlier contributions as part of the study. In order to propose a complementary vision to previous works oriented to the search for papers with the emphasis on theoretical content, we incorporated those that used ICT implementations for participation and, conceptual works that proposed new methods, models and theories for the area, without distancing ourselves from the method proposed in [6].
• Exclusion criteria: The studies that met at least one of the following exclusion criteria were excluded: introductory documents for special issues, books and workshops. Publications in any language other than English were excluded, which can be considered as a limitation of the study since e-Participation has a worldwide scope and important contributions could be discarded. We also discarded works published on workshop proceedings and poster papers. Since this review focuses on consolidated works and models, any type of work in progress was excluded, while those not related to research objectives or questions were excluded.
• Quality Assessment: In addition to general inclusion or exclusion criteria, it is considered critical to assess the ''quality'' of studies. Various aspects are defined in order to provide quality assessment of the selected studies.
• The study presents strategies to define e-participation frameworks or evaluation frameworks.
• The incorporation of ICT aspects in the models and proposals of e-Participation.
• The study has been published in a relevant journal or conference.
• The study has been cited by other authors

C. RESULTS STAGE
This stage gives the preliminary results obtained from the systematic mapping. Given the diversity of the papers found, a classification is made following the cross-sectional survey method. The following activities are carried out: • Data Extraction Strategy: This is based on raising a set of possible categories for each previously defined research question. With respect to RQ1, the results are classified according to the following categories: • e-Participation framework components • e-Participation process (include phases, activities, technologies, methodologies and outcomes) • e-Participation models With respect to the RQ2, the various papers found were classified according to the following categories: • e-Participation frameworks • e-Participation evaluation frameworks • Cases of real applications of e-Participation processes and e-Participation tools • Trust in e-Participation • Synthesis Method: We applied both quantitative and qualitative methods. The quantitative synthesis was based on counting the primary studies classified by each category defined from the research questions. We then counted the number of articles found in each bibliographic source per year. The qualitative synthesis was based on the inclusion of several representative studies for each category when considering the aspects defined in the quality evaluation.
• Conducting Stage. The application of the review protocol yielded the following preliminary results (see Table 1). As a result of the automatic search carried out in the various databases, a total of 735 works were obtained. After applying the first filter to remove duplicate items, the sample was reduced to 648. Next, the contents (especially the abstracts) were reviewed to obtain 273 works relevant to the subject. Subsequently, the various inclusion, exclusion, quality assessment criteria and categories established by the cross-sectional method were applied to this sample, selecting 92 articles related to the objectives and research questions of this survey. Subsequently, the automatic study was complemented by a manual search on 234 papers not considered in the automatic search. The same types of filter previously established by the methodology were applied and 23 articles were obtained. Finally, adding the results of the two search types, 99 articles were obtained for analysis. A summary of the various resulting works classified according to the categories previously defined by the cross-sectional method is presented in Table 2. 42% of the papers refer to cases of e-Participation processes and e-Participation tools, i.e. they are related to the purpose of this survey. 30% of works are about the e-participation process and the same percentage for articles related to e-participation models. 26% study or define the framework's components. 17% focus on generalities of the e-participation frameworks and 12.12% focus on the evaluation frameworks. Finally, 8.08% of the results concern the degree of trust in e-Participation.

III. YET ANOTHER e-PARTICIPATION FRAMEWORK: EPFW.
The comparative analysis of the proposals requires a conceptualization of the domain that unifies the diversity of sometimes overlapping concepts proposed in the different models. This conceptualization, called ePfw, was defined by analyzing and interpreting the existing proposals, adding the current needs and experiences with practitioners acquired through fieldwork. The wide diversity of applications in this domain has given rise to a variety of classification proposals, making it difficult to obtain a common language or terminology that can be a guide for this type of process, which was why we proposed the new ePfw theoretical framework for e-Participation. Table 3 shows the wide diversity of e-participation framework components proposed by different authors. A wide variety of works were found in the review that refer to e-participation frameworks, of which only 8 papers define specific components. The study also showed that most area works refer to those included in Table 3, with special emphasis on those by [6], [35].
Based on the analysis, in our opinion, the e-Participation process is the main component of this type of initiative. The process must incorporate in its phases and elements the VOLUME 7, 2019 various sequential tasks that compose the life cycle. Based on this background, in our view (ePfw), in agreement with [42], [42], policy-making process, procedural models, rules, duration, accessibility, dimensions, capabilities, programs and content development, planning and goal setting, participation and evaluation/outcomes are included in this category (process). This component (process) must act in accordance with the other components of the framework, primarily with the levels and areas of action.
The e-participation processes are framed in different levels of action according to their degree of influence in decision making. In accordance with [6], [35], [36], [38], [39], [43] the levels are part of ePfw. In addition, some participation areas are mentioned by [44], [50]. We currently know that e-participation processes are used to satisfy needs in a specific area. As participation has no limited areas, ePfw identifies as a component of the framework anything that is represented as a scalable entity. Similarly, [35], [39], [51], [60], [75] define the use of different participation techniques (some authors use the term ''methods'' with the same semantic connotation) in their proposals. Based on [45], ePfw uses the term method to include in its frameworks the various interaction techniques between actors that occur in an electronic participation process. Our analysis agrees with the work of [43] which defines a global set of a domain metamodel for the actors, levels, areas and methods.
The actors are another element represented in ePfw and are presented as an entity that includes all the stakeholders in the e-participation processes [34]. In the case of the actors, [44], [46] give instances of this actor category (e.g. government, people) as independent components, while our representation aims for a wider coverage. Technology is another component present in every e-Participation process [22]; all the cited authors agree on the incorporation of this feature in their proposals and identify tools, platforms, systems, etc. The proposal in [43] represents a wider vision, grouping this category with the requirements and reference models. ePfw also incorporates an element (roles), which, despite being identified, has not been proposed as a component of an e-Participation framework in previous framework proposals (see Table 3). As in e-government [33], there are several actors in electronic participation who can perform various roles.
Finally, the last column shows the percentage of works that define the various elements identified in their research. From highest to lowest we noticed that both the process and the tools were proposed by 100% of the authors. Also, 57.14% propose the actors as components of their frameworks. The component levels and methods are present in 42.85% of the works and the areas barely reach 28.75%.
Several research studies published in the last twenty years have included efforts to identify the components that make up a theoretical e-Participation framework. Most of the previously proposed components are incorporated, synthesized or reinterpreted to provide a reference set of concepts that will guide the discussion in the rest of this paper. Some of the components mentioned here have been widely used by researchers and are part of many of the theoretical frameworks and applications proposed. Based on the work carried out, we selected the components that make up a framework for e-Participation, as represented in Fig. 1. Each of these components will be described in detail in the following sections. In the tables below, a special column (ePfw) is added to represent our criterion in relation to bibliographical analysis and support the construction of the proposed framework.

A. THE e-PARTICIPATION PROCES
Several authors have proposed a series of phases that are achieved in the processes of public participation and e-Participation. In this section, we present the most relevant works chronologically (see Table 4). Each work has been assigned a code (P1, . . . , P3) for the public participation process phases and code (F3, . . ., F5) for the e-Participation framework proposals. These codes will be used in the subsequent analysis (see section IV).   The first three papers (P1, P2 and P3) present a very similar classification of the participation process based on three phases. In relation to the first phase of P1 (launching), the authors specify the importance of creating the environment to maximize the possibilities of successful execution of the processes. For [48] participation is based on analyzing success factors found. Next, the ''preparing'' phase aims at planning activities to ensure quality in the process. Finally, ''implementing'' executes the work taken in the two previous phases. In this context, the authors indicate ''which method is selected, how the process is designed and whether a competent facilitator steers the process and takes care of quality assurance'' [48]. A very similar approach is presented by [4], who specifies a phase prior to planning (decision analysis), which indicates the need for the existence of a new participation process, identifies the credibility characteristics of the decision-making process, and chooses the level of participation required (P2). In the same way, in the public participation guide of the [42] a process is established especially for environmental areas consisting of three phases (P3). This guide identifies a series of activities to be carried out in each of the stages. Following the same idea, the proposal by [43] is very similar to the previous one, except that the planning stage is divided into ''initiation and design'' and ''preparation'' (F5). In addition, the authors mention that the ''e-Participation project should be accompanied by continuous requirements management''.
In their framework of ICT exploitation of e-Participation (F4), Phang and Kankanhalli [51] define the importance of identifying the objective that an e-Participation initiative must cover, followed by a correct choice of techniques and ICT tools; they consider these aspects as key factors in the success of a process. In our opinion, as we consider that these three activities can be framed within a global planning phase, the model proposed by these authors is therefore incomplete. Islam and Business [47] propose a much broader classification with seven stages (F3). The authors consider that their proposal for a sustainable model with broad applicability to be ''designed to fit under any socio-economic conditions of a country and can be initiated both by public (state) and private agencies''. The first stage consists of a plan based on a national political agenda, with the objective of satisfying the questions ''what, why, whom, when, where, how'' [101]. Like the other authors referenced in this section, this paper also includes a planning phase. Subsequently, the following four phases emerge from a planning stage, starting with a ''contents development'' and the correct choice of processes and tools to be used to support the proposed content. Finally, the activities of promotion and participation are defined. The last phase corresponds to a ''post-implementation analysis'' with the objective of improving the processes according to the feedback received.
In our analysis (ePfw), we consider that an e-Participation process consists primarily of three global phases (planning, implementation and evaluation). Several authors suggest specifying a series of sub-activities from the first two, but these are activities specific to each stage only. In agreement with the rest of the authors, we show the importance of an evaluation stage in order to obtain the feedback that would allow us to improve the processes and raise the indexes of trust perceived in them. The mapping can be seen in Table 4.

B. e-PARTICIPATION LEVELS
Participation -and also e-Participation-processes can be categorized according to a hierarchy of participation levels. The levels determine the nature of the interaction between the different actors (namely governments, politicians, citizens, etc.) and the process, and have been defined according to the degree of involvement of each participant in the different stages of a public participation process. Although this revision is based on e-participation, we consider it necessary to mention the different proposals for levels of public participation, because they are the basis for the later definitions of e-participation. Several authors have established different participation level hierarchies, as summarized in Table 5.
''Non-participation'' is the level previous to participation [52], [56]; in this, the governments and organizations make the decisions without consulting the citizenship. The real beginning of a participatory process is done through the ''informative'' level [5], [52]- [56]; in this, the information flows in one direction (government -citizen); by itself, it is not considered public participation, for this reason, it must be complemented by one or several of the following levels. In the ''consultation'' level citizens' opinions are collected, but the influence of the participant in the decision making is low [5], [52]- [55]. ''Placation'' and ''active participation'' are considered levels with slight influence by [52], [53].
The ''collaboration'' in the decision-making process establishes a degree of influence of citizen participation [5], [52], [54], [55]. The following levels (engagement, empower, citizen control, etc.) are based on the high degree of influence of the participations in the decision making, in these, the contributions made are summarized and considered predominant [52], [54], [55]. Different authors [5], [53], [56] group all the collaborative levels in a single global level.
In the percentage column, shows that 100% of the proposals identify the level of information, followed by 83% of works that have been defined at the level of consultation. In the same way, the level of collaboration reaches a percentage of 66.66%, while participation, empower and involve are specified in 50% of the investigations.
In reference to the incorporation of ICT in participation, several authors have proposed the classification levels for e-Participation. First, Macintosh [6] based its work on the OECD participation levels [53] (the same levels were also used by [36]); ICTs are included with the aim of characterizing e-democracy initiatives. This work includes three levels, ''eEnabling'' with the information of the process and citizen predisposition to participate in a new initiative; ''eEngaging'' is the level of consultation and ''eEmpowering'' with high degree of incidence in decision making. Second, Tambouris et al. [35] propose the same levels of IAP2 [55] participation spectrum with the special incorporation of ICT. Finally,....Terán and Drobnjak [25] use the model in Tambouris et al. to add the concepts of Web 2.0 in order to include community-building processes and discussions between citizens and authorities. These last two authors subdivide the consultation and collaboration levels into other more specific levels. Table 6 shows the different levels proposed by VOLUME 7, 2019 each author and their respective descriptions. In our opinion, three levels are necessary for e-Participation; ''e-Informing'' always accompanied by a consultative or collaborative level, or even both. The ''e-consultative'' level is able to include the tasks of consultation, involving and discussion with the use of various e-Participation methods. As in the ''e-collaboration'' level, the e-Empowerment activities are included in the collaborative decision-making processes.
C. e-PARTICIPATION AREA Several authors [15], [26], [27], [32], [42]- [44], [54], [57], [60], [62]- [64], [76] mention the various areas in which e-Participation projects are applied. These areas are as diverse as the fields of application and vary according to each particular initiative. Moreover, new areas are created and added according to new needs and political, social, cultural circumstances, etc. Tambouris et al. [35] suggest an extensive classification of areas based on various parameters, serving as the basis for further classifications [27], [28]. Also, in some cases methods, techniques and phases of the process have been incorrectly represented as areas. Table 7 summarizes the different areas proposed by each author and their respective descriptions.
The ''community building'' area widely used in e-government may be applicable for e-Participation, for example in participatory policy modeling [32], [61], [102]. In our analysis, the ''collaborative environments'' can be encompassed by ''collaborative'' level of participation techniques and do not represent a particular area. Several areas (community informatics, citizenship education, cultural politics, inclusion/exclusion, service delivery) identified by [35], have been scarcely used by the various e-Participation initiatives [10], so that we consider that there is no need to define them as such. ''Discourse'' or ''political discourse'', is an area oriented to citizen dialogue on the part of the elected representatives [14], [46], [77].
Various ''citizen journalism'' initiatives have emerged, especially those supported by ICT, which have become a predominant factor of success in this type of process [81]. In this context, Saebø et al. [14] define ''eActivism'' and ''ePetitioning'' as areas, although we consider that these are activities can be covered by the ''citizen journalism'' area.
Another proposed area is ''mediation''; we can observe the development of online moderation systems and applications in deliberative cases, in which it is combined with environmental or voting projects [26], [31], [48], [66], [104]. Several processes for making decisions on collaborative decisions are developed within the ''spatial planning'' area, examples are the development of ad-hoc systems for neighborhood spatial planning policy [82] or the use of Web GIS ''geographic information systems'' for improvements in transportation [83]. Another area highly related to the previous one is ''environmental planning'', which has now become one of the most frequently used [10]. In this context, various projects have directed their efforts to creating participatory environmental policies and regulations [42], [84]- [88]. Other cases involve initiatives promoted by local governments [57], [80], [89]- [91] and of immigration policies [92].
''Budgeting'' is an area widely used by various government agencies. Local authorities have carried out this type of process in order to carry out works in neighborhoods based on the proposals and votes of the residents [54], [57], [58], [80], [93]. A particular example is that of Spain, which has a budgeting initiative promoted by the Madrid city council (https://decide.madrid.es). In addition, it is currently being implemented in another 50 municipalities, such as Oviedo http://www.consultaoviedo.es, Valencia (https://decidimvlc.valencia.es), etc.
In some cases, the authors define as e-Participation areas activities that take place at one of the e-Participation levels, such as ''information provision'', ''deliberation'' and ''consultation'', covered within of the informative, consultative and collaborative levels, respectively, and supported by various methods and techniques. Something similar happens  with the so-called ''polling'' [27], [28], [35] and ''referenda'' [28]; in this case, we agree with [45] who defined them as techniques or e-Participation methods.
In relation to the areas, the variety is much wider. 100% of the authors cited in Table 7 identify the speech and campaigning areas. 75% of works identify the areas of community building, voting, electioneering, mediation and spatial planning. The same percentage of studies erroneously defines the deliberation and consultation levels as areas; also, the same percentage defines the polling technique as an area. In addition, 50% have represented information and collaborative environments as an area, when these correspond to the levels. Table 8 shows a classification of the different areas according to the ePfw levels at which they can be developed. We believe that all the areas in the first execution phase are tasks at the information level and can later be developed as tasks at the consultative or collaborative levels or a combination of these.

D. e-PARTICIPATION METHOD
Within a participation process, various levels and a wide variety of methods can be used to interact with the participants. These range from the traditional and commonly used to the current and ICT-oriented. Table 9 shows a classification of the public participation methods found in the literature [4], [32], [42], [48], [45]. In our analysis (ePfw Interpretation) we select only the methods that would normally be used through ICTs. In an e-Participation initiative, it is usual to find more than one method used [64]. In general, information methods are combined with consultative and collaborative methods, depending on the area in which the process is executed. It should be emphasized that one or more new methods (not represented) can be added to the classification in a subsequent investigation.
The most frequently used information methods in current initiatives are the Internet, e-mails and telephones, due to their penetration index in most countries. Similarly, at the consultative or collaborative levels, polls, surveys, internet forums and round tables are widely used. [10]. Arbter et al. [48] classify ''citizen jury'' and ''mediation'' as methods, but in our analysis these should be e-Participation areas.
In our review (ePfw), for e-Participation we discard the methods involving community fairs, print materials, feature stories, open houses, city walk, symposiums, field offices, coffee parties, town meetings, public meetings, open space conference, advisory groups, study circles, negotiated rule making, mediation, tasks forces, and Samoan circle for their null or complicated incorporation in a software, as they are activities that require interpersonal relationships. A critical task for the implementation of methods of e-Participation systems is to try to computerize activities that necessarily require person-to-person interrelations. Our proposal includes using computers for the process after the face-to-face dialogues.
As shown in Table 9, the variety of methods used in e-participation is very broad. The percentage column shows the number of jobs cited that have defined a method. Within this variety, the 100% obtained by the surveys and consensus conference is noteworthy. Similarly, 75% have obtained the methods public meetings, focus groups and workshops. A point that is of interest are the percentages obtained by the erroneous classifications, for example, citizen jury and mediation are areas, although 50% of the works have categorized them as methods.

E. e-PARTICIPATION ACTORS
As mentioned previously, the existing classifications of the diverse actors and stakeholders that interact in an e-Participation process are very varied. Table 10 classifies the different authors's proposals; the last column shows the classification according to our interpretation (ePfw).
The first actors are the ''citizens'', treated individually [9], [14], as a group [26], [27] or as organized citizen groups: Civil Society Organizations ''CSO'' and Non-Governmental Organization ''NGO'' [6]. The citizen is undoubtedly the most important actor in e-Participation processes because most of the initiatives want to know their opinion. In some cases, an ''elected representative'', who is not necessarily a politician, is also part of this process [6], [26], [27], [35]. Government institutions, including their staff and politicians, are actors represented in common by [6], [14], [26], [27]. Due to the fact that a large number of initiatives around the world are proposed by governments or public authorities, we consider that this type of actor is almost indispensable in an e-Participation project.
The ''expert'' actors are individuals or groups that are part of organizations or research institutions [6], [14], [27], [35]. In the same way, ''industry'' is included in this classification [6], [26], [27]. The role of these two types of actor is seen as having greater influence in processes that are unrelated to aspects of governments. Furthermore, while [6], [9] define the ''decision makers'' and ''facilitator'' as actors, we claim that these roles can be played by any type of actor.
In the same way, as in the previous sections, the percentage column identifies the number of times that the cited authors have defined a certain type of actor. A greater percentage (66.66%) of investigations define the actors: citizen, elected representative, government, politicians and experts or related  to research. However, 25% define decision makers and facilitator as actors, although the correct thing is to categorize them as roles.

F. e-PARTICIPATION ROLE
A variety of roles are included in an e-Participation process, primarily in terms of software utilization. Table 11 shows a classification of these roles as proposed by other authors.
A new e-Participation initiative is always proposed and initiated by a certain type of actor (e.g. Government, citizen, etc.) in order to provide a solution or support to a decision-making process. This role represents the ''owner'' of the process.
In [27], [28], and [34] we find similarities in the definition of three roles (input provider, moderator and decision maker), and we cannot agree; because of their semantic nature, these roles fulfill different functions. Also, we consider that for the sake of clarity the ''input provider or consumer'' role should be called ''participant'' because he/she is an active participant in the initiative. Another role is that of the e-Participation ''expert'' [26], [29], [30], [42], [63], responsible for the management of the process life cycle. In the proposed ePfw model, the term ''participation service provider'' explains more clearly the function of this role. The ''decision-maker'' is responsible for deciding whether the results of an initiative should be made law or included in the participatory proposal. In some cases, the owner or initiator is responsible for making decisions. Moreover, as it is an ICT domain in public participation, the role of the ''ICT expert'' is indispensable. In our analysis, this role covers all the specific sub-units of ICT's involved in software engineering or IS processes. The ''evaluator'' and ''activist'' roles are identified in the classification proposed by the cited authors. The former can be covered by the ''participation service provider'' and the latter is a type of actor.
In our analysis, the actors and roles presented in the column (ePfw) synthesize the common aspects found in the various revised proposals; this in order to provide a clear and simple to use the catalogue.
The definition of the various roles of e-participation has mostly percentages among the 3 authors shown in Table 11. The authors coincide 100% in the identification of the roles input provider and moderator or participation service provider. Likewise, the owner role appears in 66% of the proposals.
Various stakeholders can play different roles in an e-Participation process. Table 12 shows the correspondence between the various actors and roles identified for our analysis (ePfw). For example, the role of ''participation service provider'' can be carried out by a government official, a city councilor, an ordinary citizen with a knowledge of this type of process, or an expert investigator in the domain.

G. PARTICIPATION TOOL
Several works [27], [29], [32], [50], [51], [70], [73] propose a classification of the technological tools that can be used to fully or partially carry out an e-Participation process. Table 13 shows the proposed works and our ePfw interpretation. A classification based on the functionality of the tools is also proposed, identifying the web tools, complete or complex systems and other types of applications. Because of the heterogeneous nature of e-Participation initiatives, governments and practitioners decide what type of tool or technology they should use according to the level and area of application [10], [95].
The first block contains simple and generic tools that can generally be added to web portals through any development program. Web portals, blogs, online chats, podcasting, chats and visualization tools are often used for informative level initiatives and other consultative cases.
We also refer to more sophisticated and complete tools, such as the use of Content Management Systems (CMS) for the development of ad-hoc e-Participation applications [29], [31], [50], [61], [67]. In other cases, various organizations use online survey tools to support consultative levels (e.g. Google forms, Surveymonkey, Limesurvey, etc.) or e-Petition systems. In collaborative processes the tools are voting or collaborative system applications. In complex processes with high participation it is often necessary to use content and data analysis tools. Finally, any ''tools'' proposed by the authors that are not e-Participation tools in our analysis are catalogued as ''O'' type tools.
In relation to the percentage column, there are high proportions for the various types of tools categorized for e-participation. The most frequently cited are tools for pools or surveys, with 100%, followed by the 80% obtained by online chats. 60% of the authors consider the categorization of the tools weblogs, e-petition systems, consultations platforms, combined collaborative systems and content analysis tools. The rest of the tools are categorized in 40% or 20% of the studies. These results show the wide diversity of the technologies used.

IV. e-PARTICIPATION FRAMEWORKS ANALYSIS
In this section, we first describe the main existing e-Participation frameworks and other frameworks developed exclusively to evaluate e-Participation processes. The elements present in the reviewed works are then analyzed according to the components previously identified by the ePfw.

A. EXISTING e-PARTICIPATION FRAMEWORK
In the last ten years researchers have proposed ways of enhancing citizen participation in policy-making processes via ICTs. Table 14 presents a chronological summary of the e-Participation frameworks' main proposals, each one with its primary components and whether or not it has been implemented by ICT. Each work has been assigned a code (F1, . . . , Fn) for the subsequent analysis (see section IV).
Macintosh's characterization framework for e-Participation, based on an earlier study of the OECD [53] was the first relevant framework presented. The framework is structured around the level of participation, the technology used, the actors and the stage of the policy-making process VOLUME 7, 2019 in which the participation takes place [6]. This was the first reference study to characterize e-Participation. However, the model does not specify the flow of information between the various actors, process components and levels. This proposal foresees future application based on existing cases of e-Participation.
Kalampokis et al. [27] proposed an e-Participation domain model in the form of a UML class diagram. The model provides a vision based on three main areas (called subdomains): stakeholders (involved in the e-Participation process), participation process (aspects that are relevant to the traditional public participation processes) and ICT tools (which can support public participation). Although the model is a semi-formal approximation to the domain of e-Participation, the method used for its construction is not clearly specified.
The ''Sustainable eParticipation implementation model'' [47] proposes a framework that can be suitable under certain socio-economic settings and applicable to any country, in contrast to the work of [35]. This model describes seven consecutive phases for the development of e-Participation projects: policy and capacity building, planning and goal setting, programs and content development, process & tools, promotion, participation, and post-implementation analysis. The model presents sequential phases without the possibility of going back or feedback. The model lacks adequate navigability and the flow of information is not specifically represented. In addition, despite being an implementation model, they do not mention ICTs or stakeholders. These last two proposals do not include any applications to case studies or real initiatives, as they are merely theoretical proposals for future implementation.
Around the same time, Phang and Kankanhalli [51] proposed a three-step procedure for the implementation of e-Participation initiatives: identify objectives; select techniques and select ICT tools [51]. They aimed at developing a framework for the evaluation of participation initiatives (information exchange, educational and support building, decision-making supplements, input probing). The proposed framework does not consider the main components of e-Participation, such as areas, levels, acts, roles. It is considered to be a technological approach without a developed tool of its own.
The ''Reference Framework for eParticipation Projects'' [43] appears as a multidimensional model that builds the context of an e-Participation project: a domain meta model, a procedural reference model, and a library with requirements, reference models and building blocks for e-Participation. This work aimed at supporting different target groups to communicate with other project actors (with different technical and political background) on an e-Participation project. The framework, although very complete, does not include important aspects such as technology channels, stakeholders, and activities, among others. Neither has it been put into practice in a real e-Participation initiative that supports the theories proposed.
An ''Integrated model for e-Participation'' was recently introduced by Porwol et al. [68]. The model is based on the idea that interaction between citizens and decision-makers together with other related entities constitute a social system. The model depicts citizens expressing their opinions through e-Participation platforms provided by governments. Opinions, ideas, and citizen contributions make up a database of social media and in this case, are administered and managed by governments. E-Participation platforms are related to social media databases through a series of rules, capabilities, and resources. The weakness of this model is its lack of representation of specific e-Participation aspects (like phases, levels, and methods).
The so-called ''Novel Framework of eParticipation'' incorporates the Actor Network Theory (ANT) to model eParticipation processes in terms of political, economic, social, cultural, educational and technological factors. In this framework, there are three main groups of actors consisting of governmental institutions, technology, and people, with technology acting as a mediator between people and government [44]. This work has a very broad spectrum but does not give a validation or show the methodological basis used to construct the framework.
Despite the diversity of models described, none was conceived to be actually implemented, but in most cases are theoretical constructions unsupported by tools, as one would expect in the e-Participation domain. We consider that a practical solution should define an e-Participation meta-process supported by tools that allow modeling and enacting all types of e-Participation processes.

B. e-PARTICIPATION EVALUATION FRAMEWORK
Several works have also addressed the evaluation of e-Participation methods and processes. Rowe and Frewer [45] presented a framework for the evaluation of public participation methods that defined a number of theoretical evaluation criteria essential for effective public participation by acceptance criteria and process criteria [45]. This was the first reference study on the use of methods in a public participation process, but did not include an application or evaluation of a real case of e-Participation. Table 15 shows chronologically the various contributions on e-Participation evaluation frameworks. The components or scope and the e-Participation projects on which the evaluation has been carried out are given. Each work is assigned a code (E1, . . . ,. En) to be used in the subsequent analysis (see section IV).  Tambouris et al. proposed a framework for assessing e-Participation projects and tools based on distinguishing the participation areas and their ICT support [50]. The framework specifies three main layers: participation areas, categories of tools, and technologies. The work defines a series of tools to perform the evaluation and shows the results related to the number of projects per participation area, the number of tools per category, and technologies employed. As it is mainly an analysis, it does not make an exhaustive review of e-Participation tools and technologies and only evaluates European projects.
For [69], an evaluation framework helps to obtain a better understanding of the problem of e-Participation, allowing a constant relationship between learning and process improvement. Their research identifies the problem of the heterogeneity of applications and defines e-Participation as a ''hybrid of various technologies and social and political measures''. The authors define their proposal as a starting point and  maintain a critical criterion when identifying needs for improvement. The work was later complemented by another jointly with Smith et al that included a three-layered framework for evaluating e-Participation based on three levels: operational outputs, outcomes, and impacts [70]. This contribution is analytical and presents theoretical visions, but lacks implementations in real cases.
The ''Evaluation Framework for eParticipation'' defined in [25] was used for the evaluation of a number of existing Voting Advice Applications (VAAs). The framework consists of two stages: 1) ICT tools are identified and filtered into one of five participation levels (eParticipation: eInforming, eConsulting, eDiscussion, eParticipation, and eEmpowerment), and 2) the tools are evaluated by a quantitative method. The proposal is evaluated through the application of its framework in 21 VAAs. The main limitation of this contribution is its exclusive focus on VAAs and not being applied to other areas in the field.
C. e-PARTICIPATION PROCEDURE AND FRAMEWORKS ANALYSI Table 16 gives an analysis of the procedure models, frameworks and evaluation frameworks described in the previous sections. For this we identified a series of characteristics according to the criteria of the ePfw. The second section gives the evaluation methods used in each of the proposals. In Tables IV, XIV and XV, codes are assigned to the various papers.
80% of the works generally refer to the e-Participation process and 53.3% analyze the resulting outcomes. 40% of the studies identify the different phases of an e-Participation process and 33.3% the activities that originate from these, although only 13.3% model these phases or activities. Despite the high percentage that study the process, not many study its specificity, which may be the reason for the existence of disconnected information in this domain.
Another important characteristic found is the high percentage (73.3%) related to the technologies or tools used in e-Participation, emphasizing the importance of ICTs in the participation domain. 60% of the works identify the different levels, actors and participation methods and 40% identify the areas, while 33.3% identify roles. These are fundamental characteristics in the definition of analytical and theoretical frameworks. 33% of the studies also propose an evaluation method.
In this analysis, we found very low percentages for the valuable features in this domain. Only 13.3% of the researchers developed a metamodel and requirements catalogue to support the development and execution of new e-Participation tools. The same low percentage applied the theories proposed to a tool or e-Participation system that supports its foundation. The same effect was found in the review of trust or transparency aspects in this type of process.
Finally, the lower section of Table 16 shows that 60% of the proposals used a case study as an evaluation method, while 26.6% did not use any method.

V. CONCLUSIONS
The aim of this review was to obtain a complete bibliographical review of the e-Participation field based on a systematic mapping method. The studies reviewed were published between 2000 and 2019 on the use of ICTs in public participation processes. The analysis of the classic works in the area along with recent proposals, offers a global vision of the current state of e-Participation. The study was designed to be used by researchers, government agencies, practitioners or citizens as a guide to the development and implementation of e-Participation processes. e-Participation is mostly related to e-Government environments aimed at responding to solutions of collaborative democracy or policy making. We observed that in many cases e-Participation is not differentiated from e-democracy, assuming that this type of process is exclusive to governmental or policy-making initiatives. However, these processes can be applied to any type of organization (industrial, business, educational, research, government, etc.) that intends to incorporate collaborative decision processes.
The results obtained allowed us to answer the research questions previously raised. In relation to RQ1, the findings show a variety of components that integrate the eparticipation framework proposed by various authors. The components identified that constitute the ePfw framework are: the e-participation process and tools defined in 100% of the proposals; the actors are in 57.14%, followed by the methods and levels, with 42.85%. Finally, the areas are proposed in 28% of the papers.
In relation to RQ2, the findings are described in detail in Sections III and IV of this paper. Additionally, a synthesis of the research carried out is shown in Table 16. Fifteen papers were identified that present relevant theories, models and frameworks to conceptualize e-participation. 20% of the works propose the definition of models, while the remaining 80% study and define e-participation frameworks. In addition, 33% of these are specifically responsible for proposing frameworks for the evaluation of e-participation. This paper groups the different theories and conceptualizations in this area under the context of a new conceptual framework (ePfw) for e-Participation. The ePfwtheoretical framework provides a comprehensive view of the e-Participation components (e-Participation process, levels, areas, methods, actors, roles and tools). The review carried out chronologically describes the most relevant research based on each of the components specified in ePfw, allowing a comparative view of the existing proposals. The contents of each proposal are also analyzed and interpreted to give a new approach that seeks to provide a common terminology.
Also included is a review of the research that contributed to the creation of e-Participation frameworks and evaluation frameworks of this type of initiative, identifying their strengths and weaknesses. Most of the analyzed works propose theoretical frameworks that are seldom applied to real initiatives. Despite the high degree of interest in the incorporation of technologies, only 13.3% of these studies developed an ICT solution based on their proposals. On the other hand, we agree with 80% of the proposals that specify the ''process'' as one of its components. We consider that the adequate management of the process throughout its life cycle is a critical success factor, from proper planning and implementation to the evaluation and use of the results obtained in the decision-making process. In the proposed theories we also found that few included current critical aspects, such as trust, transparency and their impact on decision making.
The results of this research can be used as the basis of the definition of a new e-Participation architecture framework and will also contribute to the creation of metamodels and new ontologies in this field. In future research, we propose to use ePfw for the development of a new multilevel and multitasking e-Participation software with characteristics such as process management and dynamic tasks, which will take into account other aspects that have been ignored up to now, such as trust, security and the transparency of the information.