Prioritization Based Taxonomy of DevOps Security Challenges Using PROMETHEE

DevOps is a combination of collaborative and multidisciplinary efforts of an organization to control continuous delivery and updates of new software while guaranteeing their reliability and correctness. In the software industry, the implementation of DevOps (development and operations units) faces many challenges that are specifically associated with the security. The objective of this study is to identify and develop a prioritization based taxonomy of DevOps security challenges. The total of eighteen DevOps security challenges were extracted using systematic literature review approach and were further evaluated with experts using questionnaire survey study. Finally, the multi criteria decision making PROMETHEE-II approach was used to prioritize and develop the taxonomy of identified factors and their categories. The implications of PROMETHEE-II approach are novel in this research domain as it has been used successfully in various other domains e.g. medical, banking, internet techniques and management etc. The contribution of this study is not limited to develop the taxonomy based structure of DevOps security challenges, but also the proper prioritization of these challenges by introducing PROMETHEE-II approach in the research field of DevOps. The study results will assist the practitioners to remove the uncertainty and vagueness in the opinion of DevOps experts to secure DevOps implementation for better and continuous software development process.


I. INTRODUCTION
DevOps is a new paradigm that focus on the collaborative and multidisciplinary nature of organization to control automated delivery and updates of software while guaranteeing their effectiveness. In software industry, the DevOps is a trending technology that focus on collaboration between and within teams involved in software development. It refers to improving the performance of software enterprises (continuous deployment) by coordinating the development and operation teams in one process [1]. Software enterprises need to adapt their protocols and practices to the various modifications brought about by new technology concepts such as ''DevOps'' [2]. According to Puppet Lab 2015 report, the enterprise with DevOps The associate editor coordinating the review of this manuscript and approving it for publication was Porfirio Tramontana . environment experienced 30 times more deployment rate then the enterprises who have not adopted DevOps in their software development cycle. Therefore, to compete with software industry pillar, the enterprises are moving towards DevOps [3]. CA Technology [4] put forward their research regarding DevOps adoption, coming out with outcomes that 88% of 1425 surveyed software enterprises will move their trend towards DevOps in next five years.
Despite the popularity of DevOps, security is among the major concerns that hinder the adoption of DevOps in software enterprises [5]. This triggered the mapping of security practices with the DevOps process by promoting the security team to collaborate with development and operation teams. Rahman and Williams [6] also emphasizes on the importance of security, by breaking silos of security, sharing that knowledge with various teams of software development process in order to build the relationship between them.
Security must be treated forefront in an enterprise in-order to adopt DevOps successfully. The deployment rate of software enterprises increasing day by day e.g. Facebook deployment rate is 500 times per day [7]. Therefore, to maintain such rapid rate if DevOps team will work without coordination with the security team, they might not get the desired output in terms of secure software. The new production unit has more chances of vulnerabilities, which can only be controlled by rapid action required by collaboration of DevOps with security practices [6]. By working as, a one-unit team DevOps process will help software enterprise in achieving better quality of software [8]. These concerns motivated us to identify security challenges in software enterprises with respect to DevOps, aiming to assist the software practitioners to move in the direction of secure DevOps adoption.
Due to the increasing popularity of security within DevOps, in this paper, empirical study has been conducted to address the concerns of software practitioners in terms of secure DevOps environment. Therefore, the significance of DevOps motivated us to explore and analyze security challenges in DevOps faced by practitioners by conducting a detailed empirical study. To achieve this study objective, firstly, we will perform a systematic literature review to identify the security challenges from literature and check the validity of identified challenges in real industry considering questionnaire survey approach. Secondly, we will apply Preference Ranking Organization Method for Enrichment Evaluation (PROMETHEE) technique to prioritize the identified challenges considering their significance to DevOps.
There are many approaches of multi criteria decisionmaking methods used for prioritization in different fields of engineering. For example, (ELECTRE, AHP, WSM, TOPSIS, and PROMETHEE etc.). However, ElECTRE and PROMETHEE are outranking models, which compares the alternative pairwise for each criterion, finding the strength of preferring one to the other [84]. The PROMETHEE II method has been used by scholars in other fields of information technology, especially computer science, management and internet techniques where multi criteria decision making is required [9], [10], [13]. This method has an appropriate structural system and the results are more consistent, easy to understand and require less information as compared with AHP, ELECTRE [10], [83]. For example, Veza et al. [11] prioritized the industrial enterprise based on the enterprise's competences. They designed a special set of competence against each enterprise for evaluation purpose and rank them by applying the PROMETHEE II method. Theodorou et al. [12] applied PROMETHEE II approach to analyze the decision of Cyprus energy resources. Siahaan and Mesran [9] applied PROMETHEE II ranking technique to determine the best student at college using various attributes like skills, performance, grading etc. Liu and Guan used PROMETHEE II method to evaluate the quality of railway passenger service. The linguistic variables were transformed to fuzzy triangular scale and based on evaluation process the quality service was prioritized [10]. Zhao et al. [13] modified PROMETHEE II to provide timely evaluation of incident management plans e.g. earthquake, flood etc.
We believe that in-depth study of security challenges in DevOps will help software industry practitioners to revise their practices and develop a new progression cycle for success and development of secure DevOps practices. To meet this study objective, the following questions were designed.
[RQ1]: What are the security challenges of DevOps implementation reported in literature?
[RQ2]: Are the identified DevOps security challenges related to industry practitioners?
[RQ3]: How can we prioritize the identified challenges?
[RQ4]: How can we develop a prioritization-based taxonomy of the investigated challenges?

II. BACKGROUND OF DEVOPS SECURITY
Recent studies have focused on the importance of DevOps and recognized that, to streamline the software development cycle in terms of better performance and scalability, developers and operation teams must tune-up. This trend of coordination (development and operation teams) at real time enables the software production system to monitor and react whenever anomalies are detected [15]. Lack of correlation activities between development and operation teams cause challenging problems which include: (1) poor information and communication flow, (2) security related concerns, (3) immature systems, (4) unsatisfactory test environment etc. [14], [16].
To meet the on-demand infrastructures and continuous delivery product, DevOps software organizations are searching for active development approaches. There are many challenges of DevOps identified in literature such as ''lack of DevOps knowledge'' [69], ''conceptual gap between development and operation team'' [70], ''lack of efficient tools'' [20], ''continuous testing'' [3] etc. However, prior work is done in domain of DevOps security. Combining the expertise of development, operations and security within DevOps environment can resolve several security issues. Researchers from academia and industry agree to integrate security practices in the DevOps environment. For example, Cash et al. [18] imply to incorporate the security practices with DevOps as SecDevOps. Rahman et al. [17] stated that ''DevSecOps, SecDevOps, SecOps, and RuggedOps are aliases of Security in DevOps. These terms refer to the integration of security principles in DevOps by promoting the collaboration between the security teams, the development teams and operations teams.'' They also believe that automated testing and monitoring contributes positively while dealing with software security. Vries [19] emphasized on how the traditional security practices, with a focus on manual processing tools and documentation, are unfit in an environment of continuous deployment. The security practices must be replaced by more upgrade approaches in order to meet the requirements of continuous deployment process.
Prior studies have discussed security aspects with respect to DevOps and agile practices in software organizations.
Smeds et al. [20] highlighted the key concerns amongst organizations for adopting DevOps, but did not consider the security challenges while working in DevOps organization. Rahman and Williams [3] also discussed security in DevOps by adding additional security activities such as, security requirement analysis and security configurations with DevOps to remove vulnerabilities. Mohan and Othmane [5] investigated the main security aspects related to DevOps i.e. 'definition, best practices, configuration methods, tools of SecDevOps, team coordination and data secrecy activities' and give a suggestion of merging security with DevOps, as security is one of the key challenges which limits the adoption of DevOps. Therefore, while dealing with DevOps progression, the security must be considered as an essential part of development. In our study, we have listed the security challenges in DevOps that can contribute by helping software practitioners to upgrade their security practices related to DevOps. Furthermore, by introducing the PROMETHEE II [21] approach in the research field of DevOps we prioritized the challenges which will assist organizations to remove vagueness in the opinion of experts to secure DevOps. Our findings will also provide a taxonomy based on CAMS (DevOps principles) [22], [23], [25], which could help experts to improve their management strategies to secure DevOps continuous activities.

III. RESEARCH DESIGN
To achieve the study objectives following three research methodologies were applied ( Figure 1):  The questionnaire survey to empirically validate the identified challenges of DevOps security from industrial perspective. PHASE 3: PROMETHEE II approach to prioritize the identified challenges concerning their importance for DevOps security.

A. SYSTEMATIC LITERATURE REVIEW (SLR)
We have adopted systematic literature review approach to explore the challenges of DevOps security in literature. This step by step approach of literature collection, examination and extraction of data gives more valid outcomes as compared with other informal methods of literature review. The guidelines provided by Kitchenham and Charters [64] was adopted to explore literature which include i) planning the review, ii) conducting the review and iii) reporting the review. The phases are discussed briefly in subsection below: The search string plays an important role in the collection of data from selected studies. We have also developed the search strings using key terms and their alternatives collected from other studies by using the Zhang et al. [66] guidelines. The key words and their alternatives are given below: (''barriers'' OR ''obstacles'' OR ''hurdles'' OR ''difficulties'' OR ''impediments'' OR ''hindrance'' OR ''concerns'' OR ''challenge'') AND (''SecDevOps'' OR ''DevSec-Ops'' OR ''SecOps'' OR ''security'' OR ''privacy'') AND (''DevOps'' OR ''Development and Operation'', OR ''Continuous development and operation'').

d: INCLUSION AND EXCLUSION CRITERI
Protocols were designed to perform inclusion and exclusion criteria on literature collected in response of search strings. The same approach has been adopted in other software engineering researches as Niazi et al. [42] and Akbar et al. [44]. The considered protocols are presented below: Inclusion Criteria: • The paper should be published in the well reputed journal, white paper, book or conference.
• The article must consist of factors that have a negative influence to secure DevOps.
• The paper should have a clear concept about DevOps implementation.
• The selected article must be in English language. Exclusion Criteria: • If two studies are from similar project only the most complete one will be considered.
• The paper that does not provide detailed information about DevOps progression.
• The studies not related to DevOps security will not be considered.
• The literature studies are not considered.

e: STUDY QUALITY ASSESSMENT (QA)
In this step we perform QA to check the effectiveness of study with respect to the research objective. We have followed the Kitchenham and Charters [64] guidelines for QA. The five scale Likert scale was used for this assessment. The detail analysis of QA with developed research questions and scale is given in Appendix C (Table 15).

2) CONDUCTING THE REVIEW a: FINAL STUDY SELECTION
For further refinement of selected literature, we have adopted tollgate approach developed by Afzal et al. [67]. In the initial stage after performing inclusion and exclusion criteria we have collected 110 studies. All the steps of tollgate approach were performed carefully and total of 40 studies were collected for data extraction ( Figure 2). We have performed the QA process, to check the effectiveness of studies briefly discussed in Appendix C (Table 15).

b: DATA EXTRACTION AND SYNTHESIS
The data extraction is performed clearly by involving first three authors of this study. In initial step the themes, concepts and challenges of DevOps security were extracted from the selected studies. After synthesizing the data, we have total eighteen security challenges of DevOps. There may be biases between the study findings, to remove this concern we have conducted inter-rater reliability test [68]. We invited three external experts to participate in the data validation process. They performed all steps of data extraction by randomly selecting ten studies. After comparing the outcomes from external experts and study authors, we have calculated the Kendalls coefficient of concordance (W) [68]. The value 'W = 0' represents complete disagreement and 'W = 1' represents complete agreement. The results of W = 0.84, (p = 0.003) show the significance agreement between the external experts and study authors. The code used to calculate (W) is given in link https://rdrr.io/cran/DescTools/man/KendallW.html.

3) REPORTING THE REVIE a: QUALITY OF SELECTED STUDIES
The quality assessment is performed to measure the significance of selected literature to address the research question of this study. The 65 % of studies score more than 60% (Appendix C) which shows that the selected studies are significant enough to answer the research question of this study. We choose 50% as a threshold value while performing QA.

b: PUBLICATION YEARS OF SELECTED STUDIES
During the data extraction phase we have also extracted the publication year with the aim to check the frequency of publications literature about DevOps. The frequency analysis shows that the selected studies are from year 2011 to 2020 showing the increasing trend of research in the field of DevOps. This shows that the DevOps is an active topic in the research field of software engineering.

B. EMPIRICAL STUDY
To validate the findings of SLR and to identify more challenges of security in DevOps we have conducted a questionnaire survey approach Figure 3. This approach will assist to collect responses from industrial practitioners working to secure DevOps activities. The questionnaire approach provides the best opportunity to collect practitioner's opinion from large population [31]- [35].

1) QUESTIONNAIRE DEVELOPMENT
The design of the questionnaire was developed by using the platform of Google form (i.e. docs.google.com/forms). The questionnaire consists of three sections, that includes i) bibliographic information of survey participants, ii) the second section is about closed ended questions that consists of DevOps security challenges identified from SLR study, iii) the third section consists of open-ended questions to ask participants about additional security challenges of DevOps. VOLUME 8, 2020

2) PILOT ASSESSMENT OF QUESTIONNAIRE SURVE
The pilot assessment of questionnaire was performed with aim to check the understandability and suitability of designed questionnaire [36].
The designed questionnaire was sent to three external experts of academia and industry for evaluation. The requested experts include two industrial experts from (Virtual force Pakistan and I-Tech Malaysia) and one from academia (King Fahad University of Petroleum and Minerals, Saudi Arabia). Some modifications were suggested by the participants related to the structure of the questionnaire and to add question regarding DevOps experience in an organization. They further suggested to use tabular format for the questions response. We carefully reconstruct our questionnaire survey, according to the recommendations suggested by participants, and final format of questionnaire is given in Appendix A.

3) DATA SOURCE
The goal of this study is to identify the security challenges in DevOps. The data source plays an important role to get an opinion from targeted population. Though, to validate these identified challenges from the target population we have used Research-Gate, Emails and LinkedIn profiles. To spread the questionnaire survey, we have used the snowball technique [37], [38], [40] which is cost effective and easy way to target large scale population. The duration of data collection is from November 2019 to January 2020. We have collected total 88 responses. All responses were checked manually and 10 of them were incomplete. We didn't consider the incomplete responses for data analysis as a requirement of the questionnaire is not accomplished fully in those 10 incomplete responses. Total 78 responses were evaluated for further processing. The detail of respondents' response is given in section 4.2.

4) SURVEY DATA ANALYSI
The frequency analysis method is used to analyze the qualitative and quantitative data. This approach is effective to measure the ordinal and nominal data between the variables and across the group of variables [44]. This method has been used to statistically compare the identified factors and their significance in the software industry. Several existing studies of other software engineering domain have used the same data analysis approach [41]- [43].

C. PROMETHEE II APPROACH
The adopted approach PROMETHEE (Preference Ranking Organization Method of Enrichment Evaluation) was presented by Brans et al. for the first time in 1985 [21] and now several versions of it are available i.e. PROMETHEE I, II, III, IV, V, cluster etc. The application of the approach depends upon the nature of the problem. Considering the advantages this approach is easy to use and have a low level of complexity. Many successful applications have been treated by PROMETHEE approach in various fields such as; banking, chemistry, health care, management, information technology etc. [9], [11], [12], [39]. The PROMETHEE II was selected for this study because it is interactive and is able to classify and order alternatives which are complex and difficult to compare. This approach in addition has some other characteristics like stability, clarity and simplicity [21]. The principles of PROMETHEE II are based on pairwise comparison of alternatives for each criterion ( Figure 4). According to Yaghoobi [45] ''the alternatives are evaluated according to different criteria which have to be maximized or minimized. Each criterion should be able to distinguish the alternatives, regardless of how the alternatives behave under other criteria''. This approach provides complete ranking of alternatives, but as in many multi criteria decision making approaches, decision makers have to identify alternatives by assigning weights, scoring criteria and should have knowledge about out-ranking relationships among different alternatives and the fuzzy variable terms of scale [46]. This method consists of seven steps which are described below: STEP 1: In this step decision matrix is normalized using beneficial and non-beneficial criteria.
(beneficial criteria) (i = 1, 2, . . . , n and j = 1, 2, . . . , m) (1) where D ij is the decision maker's evaluation of the ith alternative with respect to jth criterion. STEP 2: The difference of ith alternative as compared with other ones are evaluated in this step. This means that the differences in criteria values among various alternatives should be assessed pairwise.
STEP 3: In this step we choose and calculate the preference function, P j (i, i ). Mareschal [47] discussed six different types of general preference functions ranges from 0 to 1 (Figure 4). According to Mareschal ''the PROMETHEE method induces a preference function to describe the decision maker's preference difference between pairs of alternatives on each criterion''. It is possible to choose a different function for each criterion. The usual function (or Type 1) was chosen for this study ( Figure 5). Using this function, no parameters and indifference threshold required. This usual criterion function is defined below: STEP 4: The aggregate preference function is calculated by incorporating the weights: where w j is the weight of relative importance given to jth criterion. STEP 5: For positive and negative outranking flow each alternative is compared with (n-1) alternatives [48], [49] Figure 6. Therefore, it is essential to calculate the entering and leaving outranking flows of each alternative which is given by: The leaving flow: where, n represents the number of alternatives. The entering flow measures the weakness of alternatives and leaving flow measures the strength of alternatives. STEP 6: The PROMETHEE II provides a complete preorder determined by the net outranking flow of decision alternatives: The net flow: STEP 7: Calculate the ranking of alternatives considering the net outranking flow ϕ (i). The highest the ϕ (i) the best the alternative. Another advantage of using PROMETHEE II is that it avoids incomparability between alternatives.

IV. STUDY FINDINGS
The section contains the outcomes and analysis of this study.

A. IDENTIFIED CHALLENGES VIA SYSTEMATIC LITERATURE REVIE
Various studies have discussed the factors that could negatively impact the DevOps activities in terms of security. We reviewed those studies briefly and extracted all the challenging factors including lack of secure coding standards [2], [23], [80], [81], using unsuitable performance metrics for security evaluation [2], [69], [78], [79] and lack of integrated testing tools to secure DevOps [14], [52], [53], [71]. Düllmann et al. [24] and some other researchers [69], [78], [81] considered neglecting change control in security most challenging factor of DevOps. They highlighted that proper security measures among the security and DevOps teams must be shared to make it easy to coordinate and control the security tasks in the distributed software development processing. Rahman and Williams [3] analyzed the selected sets of internet artifacts and surveyed them in nine organizations to explore experts' experiences in DevOps security practices. They explored that security manual testing and performance measures, threat modeling and scalability, compliance requirements and lack of automated testing tools are the factors having significant impact while securing DevOps activities. The same challenges have been explored in some other researches as well [2], [5], [6], [14], [16], [19], [21], [23], [25], [80]. The software enterprise can only manage the continuous deployment process if they have automated testing tools and performance measures security skills to control such activities. They must have deep knowledge about model of DevOps (CAMS) [25] in order to secure DevOps. They must hire DevOps consultants if they don't have much expertise to manage DevOps security. The consultants will provide step by step guidelines to secure and adopt DevOps activities.
The distributed nature of software development process makes it challenging to secure DevOps activities from various perspectives i.e. processing, communication, deployment, delivery and testing etc [27]. The challenging factor lack of coordination between security team and DevOps team is causing more security threats like untrusted inputs [54], [55], [74]. The proper implementation of 3C's (communication, coordination, control) must be considered to develop trust and understanding between team members of DevOps. Developer resistance not to coordinate with DevOps security team [57], [76], [77] slow-downs the whole deployment process, as no proper measures were taken to control security threats during development phase. There should not be an unrestricted collaboration between DevOps and security teams [20], [23], [58], [79] to avoid leakage of information at both ends. The leadership should support their teams to make decisions about the inadequate channel to monitor team coordination [52], [53], [71], [79] to control DevOps activities. Proper trainings and meeting sessions should be conducted to improve the expertise of teams. The capabilities to support and encourage team members by leadership to control the abundance of information causing problems to secure data [60], [62], [80] will process DevOps smoothly.
The other frequently occurred challenge in an organization to secure DevOps activities is immature automated deployment tools [61], [81], [82], due to lack of testing knowledge. The DevOps team must corporate with security team to measure such challenging factors. The enterprise generally lacks with consistence security polices design and performance measures [6], [19], [52], [72], [73] due to the rigid policy structure. They should reconsider their policies according to new framework standards for better performance. We have also identified lack of automated testing tool performance measures for security [2], [5], [20], [23], [24] as a critical challenge that have a negative impact on DevOps security. Mohan and Othmane [5] surveyed the literature and academia and pointed out that the automated testing tools plays a significance role to integrate security in DevOps. They also investigated that the security and software communities should help industry to develop secure software while applying DevOps activities. The total of 18 challenges were identified and presented in Table 1.
The reported challenges were further divided into four main categories of CAMS model, presented by Edwards and Willis [25]. They classified the DevOps activities into four phases which includes; (Culture, Automation, Measurement and Sharing). This model consists of a set of variables which is useful for the successful implementation of DevOps activities in an organization. We grouped a team consist of three authors for mapping purpose (first three authors). All the team members of the group were using key steps of coding scheme, i.e. code, sub-categories, categories and framework/ theory, of Grounded theory approach [28], [85] as followed by other researchers in their studies [30], [86], [87]. By applying this approach, all the challenges were mapped as shown in Table 2. We labeled identified challenges of security in DevOps as (CH). The aim of this categorization is to perform the PROMETHEE II approach for prioritization of challenges based on values of CAMS principles given in Figure 7. We also verified our finding by applying inter-rater reliability test [68] to remove further biases before applying PROMETHEE II. The three external were invited to verify the mapping scheme results. They counter checked all the steps performed for mapping scheme. After comparing both outcomes we have calculated the Kendalls coefficient of concordance (W = 0.635, p = 0.0005) shows the significance agreement between both results. The link of code is given in the empirical study section.

Culture (C):
The culture is defined as ''the interaction of people and groups and is driven by behavior. Substantial communication improvement can result when there is a mutual understanding of others and their goals and responsibilities.'' In traditional information technology business development and operation teams works as distinct groups. They spoke different languages as operation staff deals with 'maintaining stable environments and infrastructure' and development team deal with 'innovating, creating and moving fast and breaking things.' These goals without coordination permit both teams to perform their tasks smoothly. By changing this business concept and sharing responsibilities and tasks on the same page is a main goal of DevOps.
Automation (A): Automation deals with many needs and solves various issues of an organization. Automation can save time, cost and effort and is just similar to culture in sharing responsibilities. According to Guthrie [25] ''The impact of implementing infrastructure as code as well as using continuous integration and continuous delivery pipelines can be magnified after understanding an organization's culture and goals. It helps to think of automation as an accelerator that enhance the benefits of DevOps as a whole.'' Measurement (M): The continuous improvement process is essential while dealing with DevOps environment. Using the measurement will help the practitioners to follow the right direction. DevOps practices will encourage the practitioners to look at the entire system and assess the whole system not just focusing on small sub-sections. The main significance of using measurements has kept a balance on ''income, costs, revenue, mean time to recovery, mean time between failures, and employee satisfaction.'' Sharing (S): The last phase of CAMS is sharing. Sharing includes three components visibility, transparency, and transfer of knowledge. By sharing the knowledge with teams will help to integrate the organization loop strongly. This collective information will encourage the team to develop a strong unit that can resolve all basic ambiguities which DevOps faces in an enterprise [63].

B. EMPIRICAL STUDY RESULTS
To verify the findings of systematic literature review, a questionnaire survey was conducted with the industrial and academic experts and the analyzed responses of participants are discussed in sub-section below. The questionnaire was designed for survey analysis (Appendix -A). The total of 78 complete responses was collected from participants (DevOps experts) Table 3.
Finstad [40] stated that ''the bibliographic data of survey participates give the insight of survey respondents which shows the maturity level of collected data set.'' Shameem et al. [41] underlined that the demographic data is useful to collect information about survey respondents and to verify that the targeted population is related to the particular study. The demographic detail of DevOps experts has been discussed in sub-sections.

1) RESPONDENTS DESIGNATION
Niazi et al. [42] underlined that the position of participants in particular organization has influencing impact on the factors. The suggestions of ranking by participants are based on the level of experience the participants have with that factor. Finstad et al. [40] defines that the priority of influencing factors varies according to the designation of the targeted population. The designation-based analysis of the respondents  is presented in Figure 8. The results show that the most of the participant's designations are: researchers, software operators, project managers and software developers.

2) RESPONDENTS EXPERIENCE
The survey participants experience was also analyzed. The medium and mean were calculated and the outcomes shows 5 and 7.5 indicating the young participants respectively. In addition, the significant difference in respondents' experiences were observed briefly. The Figure 9 shows the detail of survey participants graphically.

3) ORGANIZATION SIZE
We collected data related to organization size from the respondent's bibliographic information. By considering the Australian burro of statistics [43] which stated ''small (0 to 19 employees), medium (20 to 200 employees) and large (>= 200 employees)''. we classified the size of an organization as small, medium and large. Akbar et al. [44] highlighted size of an organization as a key entity to assess the maturity level of the survey respondents. The results are graphically explained in Figure 10 showing 40% (small), 30 % (medium), and 30% (large) scale organizations respectively.

4) RESPONSES AGAINST SECURITY CHALLENGES IN DEVOPS
We have collected the responses from industry experts to validate the finding of SLR. The security challenges of DevOps reported in literature were briefly discussed in section 4.1. The responses of survey participants were classified using a The summarized results Table 3 shows that the majority of the survey respondents have a positive opinion about security challenges in DevOps showing the negative effect of factors on DevOps implementation. The frequency analysis shows that percentage of most of the challenges and their categories is > 60%. We have also designed open ended questions in a survey for participants to report additional challenges that were not identified in the available literature study.
However, no additional challenge was marked by the participants. We further noted that CH18 (neglecting change control in security 96%) is the highest reported challenging factor by survey respondents and ''Culture'' i.e. (C1 = 91%) is the highest category of investigated factors. The same approach has been adopted by other researchers in various fields of software engineering [41], [44].
Hence, based on our findings, we have further prioritize the identified factors by using the PROMETHEE II approach. Each category was compared with all identified challenges in the second survey conducted with DevOps experts. The survey was conducted with the participants who have participated before in the first survey. A total of 25 participants agreed to give a response on second survey Table 4. The questionnaire designed for second survey is provided in Appendix B. The approach discussed in section 3.2 has been used for the assessment of the questionnaire. The sample size is small which may limit our research to some extent. But small sample size has been considered previously in other research studies also [45]- [48]. For example, Wong and Li [49] conducted a survey of the intelligent building systems and evaluate their results based on nine survey responses. Lam and Zhao [50] conducted a survey based on eight participants to evaluate the teaching quality. Shahmeem et al. [41] considered five expert's responses to prioritize the factors in agile based organizations. Therefore, based on the above evaluations we can justify 25 as an acceptable sample size for survey analysis. We collected data from 25 practitioners to measure the pairwise comparison of identified challenges using PROMETHEE II approach.
To weight the categories of CAMS model, all authors of this research participated. The scale discovered by Bozbura et al. [51] was used to convert the linguistic values of weights to numbers. The average weight for each category was calculated which was further used in the PROMETHEE II approach for prioritization.
The weights of categories were further investigated by sending them to three external DevOps experts (one expert from Chongqing University, China and two DevOps experts from Soft-Tech organization Pakistan). After their approval, we have used the weighted categories for further calculation. The Table 4 shows the weighted categories and pairwise percentage ratio of each alternative calculated using a Likert scale explained previously in this section.

C. APPLICATION OF PROMETHEE II APPROACH
To prioritize the identified security challenges of DevOps with categories we used PROMETHEE II approach. The steps followed to perform the application of PROMETHEE II are discussed below: STEP 1: Normalized the decision matrix by using eq 1and eq 2 is constructed (Table 5) we have selected ''automation'' as non-beneficial criteria depending upon the nature of automation attributes (i.e. cost, time and effort etc. section 4.1) which should be reduced while developing DevOps environment. All the other criteria are beneficial criteria. VOLUME 8, 2020  STEP 2: To evaluate the difference in criteria values between different alternatives pairwise difference is calculated. For example, pairwise difference of 'lack of automated testing tools' CH1 with respect to other alternatives and criteria is shown in Table 6. The same method has been applied on other alternatives in link https://tinyurl.com/uo29yq2. STEP 3: To choose and determine the preference function we have used usual function because it does not require any parameter such as preference and indifference thresholds. We replaced the calculated values of pairwise difference considering Eq.3 and Eq.4. For example, the preference function values of ''CH1'' is shown in Table 7. The same approach has been adopted to calculate the preference function of other challenges in link https://tinyurl.com/uo29yq2.  According to the ranking of alternatives Figure 10 by PROMETHEE II approach CH1 ''lack of automated testing tools'' is marked as the most critical challenge to secure DevOps implementations. Therefore, there should be proper automation testing tools to monitor the security risks of DevOps. As we are dealing with heterogeneous nature in an organization all sites must coordinate to resolve any ambiguity on time [52], [61]. CH16 ''lack of secure coding standards'' is considered as the challenging factor in terms of DevOps security. Proper security implementations and counter measures must be done to control such risks. The standards of security must be improved with that of other software processing paradigms [4]. Another challenge  is CH18 ''neglecting change control in security'' which occurs when DevOps team and security team works with different principles and have obligations to work together. The DevOps teams must be upgraded by providing them platform of training sessions and discussion bench [53]. The DevOps team focus more on coordination of development and operational team, integrating security with DevOps will resolve several issues. The researchers and academic experts agreed to integrate security with DevOps implementations [4], [6]- [8], [59]. The above marked challenges are critical to secure DevOps proper scheduling must be performed to manage such challenges. These steps will help the organization to adopt DevOps without any consideration related to security.
Further, we have calculated the ranking of identified factors,a according to the CAMS model. For validation we send our mapping scheme to three DevOps experts (one expert from Chongqing University China and two experts from Soft-Tech organization Pakistan) section 4.2.4. Based on their recommendations, we rearrange the position of some factors, the final mapping is presented in Table 3. We have followed a formal mapping approach to categories the challenges, applied in other research studies as well [31]- [33], [44]. To avoid uncertainty and vagueness during prioritization we have been introducing a PROMETHEE II approach. The given taxonomy of categories and factors will contribute to enhance the areas which required more security practices for improvement. The overall rank is represented by 'GR' and local rank by 'LR' shown in Figure 12 respectively. Further, we analyzed that ''Security manual testing and performance configuration'' CH 2 ranked 4th in the overall ranking (GR) but ranks 1st (LR) in ''Measurement'' criteria of CAMS model. This ranking of identified challenges of DevOps security will assist the practitioners by considering the significance of factor within the process area and as a whole.

V. DISCUSSION AND SUMMARY
The key objective of this study is to identify and rank the critical challenges that affect the security of DevOps. To address this study objective, we have conducted an SLR study to explore DevOps security challenges available in the literature. The identified challenges were classified further into core categories of CAMS model. For verification of identified VOLUME 8, 2020 factors, we have conducted a questionnaire survey study. Finally, we applied PROMETHEE II approach to priorities the investigated factors with respect to their importance in the area of DevOps security.

A. INVESTIGATION OF DEVOPS SECURITY CHALLENGES (RQ1)
To investigate the challenges of DevOps security we have conducted an SLR. The identified factors (Total = 18) present the key areas where team members of DevOps must focus to secure DevOps. The challenges were further classified into four categories according to CAMS model. The section 4.1 shows the identified factors. The factors and their categorization were empirically investigated by questionnaire survey approach.

B. VALIDATION OF CHALLENGES THROUGH QUESTIONNAIRE SURVEY (RQ2)
The questionnaire survey was conducted to identify and validate the factors from industrial practitioners having knowledge about DevOps. The total of 78 responses was collected and the results of the survey show that the identified factors have a negative impact on DevOps security section 4.2.

C. PRIORITIZATION OF DEVOPS SECURITY CHALLENGES (RQ3)
We have used the step by step process of PROMETHEE II approach to prioritize the factors and their categories. The pairwise comparison of each factor with respect to categories were performed. By considering the net outranking flow of each factor we priorities them. The greater the net outranking flow is the highest is the priority of factor with respect to other alternatives. This technique provides better understanding of multi criteria decision making problems by considering the DevOps security challenges and their mapped categories. This technique provides the complete ranking of all alternatives which will assist the DevOps team to focus on areas having security related issues. The results are presented in Table 11. The result show that the CH1 ''lack of automated testing tools'' is the most critical challenge need to be resolved while dealing with DevOps security in software enterprise. Furthermore, CH16 ''lack of secure coding standards'' and CH18 ''neglecting change control in security'' are the second highest ranked challenges while securing DevOps for successful implementation. Figure 11 shows the graphical distribution of challenges based priority measures. The taxonomy of DevOps security challenges was developed based on CAMS model [25]. The core categories of CAMS model are Culture, Automation, Measurement and Sharing briefly discussed in section 4.1. The survey respondents strongly emphasized that all the categories are important for DevOps implementation. They weighted (w) the Culture 'w = 0.35' as the highest ranked category as DevOps is all about coordination and commitment of development and operational teams to share their ideas for successful implementation of DevOps. Automation 'w = 0.25' and Measurement 'w = 0.25' ranked as the second and third most important categories of investigated DevOps security challenges. The overall taxonomy of factors with respect to their categories and prioritization is explained in Figure 12. We have investigated that the ''security manual testing and performance configuration'' CH 2 is ranked, i.e. GR = 4 but in Measurement category it is ranked, i.e. LR = 1 which shows the significance of this challenge with respect to category. Similarly, ''Abundance of information is a serious threat to secure data'' CH 13 ranked GR = 9 in overall ranking but ranked, LR = 1 in Sharing category. The priority of each factor is presented in Figure 12 which will help the practitioners to resolve security concerns of DevOps by applying new practices considering the specific process area.

VI. SUMMARY OF FINDINGS
The main aim of this study is to investigate security related challenges hindering DevOps implementation in software enterprise. We have applied systematic literature study and questionnaire survey approach to identify and validate the factors having a negative impact on secure DevOps adoption. The PROMETHEE II approach has been applied to prioritize the challenging factors concerning to their significance to DevOps security. The brief summary of research question findings is explained in Table 12.

VII. THREATS TO VALIDITY
There are several threats to be addressed towards this study design. For example, there might be a threat to cover all challenges mentioned in literature. This threat may cause incomplete data collection sources, and some relevant studies might be omitted. To reduce this threat, we have developed the search strings to collect data from targeted digital libraries and performed QA steps to verify the quality of research material. The same approach has been adopted by other existing literature review studies [31], [33], [42].
An internal threat to validity is that there might be researcher's biases while finding literature. To resolve such threat, we performed inter-rater reliability test to check the researcher's biases, and the results show that the outcomes are consistent and unbiased. VOLUME 8, 2020  Another threat to validity is that the sample size n = 78 for questionnaire survey might not be strong enough to justify the validity of challenges. However, based on the other researches in the field of software engineering [30]- [33], [44] this sample size of targeted population is justifiable and for the assessment of factors and their mapping criteria. The response from survey participants were verified to evaluate the significance of identified factors in industrial and academic state.
In this study most of the survey respondents are from Asian countries and we will unable to generalize the outcomes from the other regions perspective. However, the data collected from survey consist of response from different regions and we believe that this sample data is sufficient to validate the factors.
The mapping scheme also has biases to some extent, to validate this concern we have used coding scheme to develop a mapping concept and validate it by empirical study (questionnaire survey) and considering inter-rater reliability test. This shows the significance of categories with respect to mapping alternatives. The same method is adopted in some other studies [30], [86], [87].
There is another threat of ranking the identified and reported challenges, to avoid this threat we prioritize factors through PROMETHEE II approach. The net outranking flow values are used to rank the factors after pairwise comparison of factors with categories, the greatest the outranking value the highest is the rank.

VIII. STUDY IMPLICATIONS
The main focus of this study is to bring in consideration DevOps security challenges due to which practitioners were afraid to implement DevOps activities. This study with detailed systematic literature review and empirical investigation of DevOps security challenges in software organization will provide a body of knowledge to researchers and academic experts, to make strategies and plans for epic secure DevOps adoption.
The identified challenges were mapped into four categories of CAMS model (Culture, Automation, Measurement and Sharing) for further ranking of factors, based on their significance for the successful implementation of secure DevOps environment. These ranks will provide knowledge to the practitioners to consider the most significant factor based on their priorities. The taxonomy of factors will help the practitioners to make improvements in process area which needs further development. By integrating the security practices in DevOps will secure the DevOps environment and provide new thought to validate security measures of DevOps while implementing it.

IX. CONCLUSION AND FUTURE DIRECTIONS
The secure environment of software process development is the first priority of every organization. The software organizations are finding ways to secure their production unit for effective development of products. The DevOps is the most significant approach, which provides more satisfactory results of continuous deployment. The significance of DevOps motivated us to secure DevOps process by exploring the security related challenges faced by practitioners while successful implementation of DevOps.
The SLR was conducted to explore all security related DevOps challenges available in the literature. The identified challenges were further mapped into key categories and verified from industrial and academic experts using questionnaire survey approach. We get 78 total responses which show that the identified factors are related to industrial concerns about DevOps security. These challenges must be resolved for better implementation of DevOps process. Moreover, the PROMETHEE II approach is used for the prioritization of factors and their categories with respect to their importance for the implementation of secure DevOps activities. The ranks were determined using this approach which will assist the practitioners to focus on the key areas which require further improvements to secure DevOps. ''Lack of automated testing tools'' CH1 and ''lack off secure coding standards'' CH16 are the most significant factors and must be addressed first based on their ranking. The taxonomy designed using CAMS model will show the significance of factors and categories for further perfection of DevOps practices. For further analysis, we will investigate the factors that have a positive impact on DevOps security by conducting a literature review and empirical study approach for the successful implementation of DevOps activities.

APPENDIX A
See Table 13.

APPENDIX-B
See Table 14.

APPENDIX C
See Table 15.
WU YU received the bachelor's degree in engineering automation and the Ph.D. degree in automation theory and application from Chongqing University, China, in 1992 and 1997, respectively. Since 1998, she has been working as a Teacher with the Chongqing University of Posts and Telecommunications (CQUPT), China. She is currently a professor.
MUHAMMAD AZEEM AKBAR received the M.Sc. and M.S. degrees in computer science from the University of Agriculture Faisalabad (UAF), Faisalabad, Pakistan, and the Ph.D. degree in software engineering from Chongqing University, China. He is currently working as a Postdoctoral Researcher with the Nanjing University of Aeronautics and Astronautics, Nanjing, China. He has published more than 30 research articles in well-reputed journals and conferences. His research interests include global software development, requirements engineering, empirical studies, global software requirements change management, DevOps implementation, software defect prediction, the Internet of Things, code recommender systems, and software risk management. He has an Outstanding Academic Carrier.
AHMED ALSANAD received the Ph.D. degree in computer science from De Montfort University, U.K., in 2013. He is currently an Associate Professor with the Information System Department and the Chair Member of pervasive and mobile computing with CCIS, King Saud University, Riyadh, Saudi Arabia. He has authored or coauthored more than 12 publications, including refereed IEEE/ACM/Springer journals, conference papers, and book chapters. His research interests include cloud computing, health informatics, ERP and CRM.
ABDU GUMAEI received the B.S. degree in computer science from the Computer Science Department, Al-Mustansiriya University, Baghdad, Iraq, the master's degree in computer science from the Computer Science Department, King Saud University, Riyadh, Saudi Arabia, and the Ph.D. degree in computer science from King Saud University. He is currently an Assistant Professor of computer science. He has worked as a Lecturer and taught many courses, such as programming languages at the Department of Information Systems, King Saud University. He has several researches in the field of image processing. His research interests include software engineering, image processing, computer vision, and machine learning. He has received a patent from the United States Patent and Trademark Office (USPTO), in 2013.