Toward Successful DevOps: A Decision-Making Framework

DevOps (development and operations) is a set of collaborative practices that automate delivery of new software updates with the aim to reduce the development life cycle and produce quality software products. Software organizations face several barriers while adopting DevOps practices as the integration of development and operation teams requires merger of different processes, tools, and skill sets. This study aims to develop a prioritization-based framework of the DevOps best practices based on evidence collected from industry experts. To attain the study aims, firstly, a systematic literature review was conducted to identify DevOps best practices reported in the literature. Next, a questionnaire survey study was conducted to receive insight from industry practitioners for the identified best practice. Finally, the fuzzy-AHP technique was applied to prioritize the best practices concerning to the significance for DevOps process. We believe that the identified best practices, their categorization and fuzzy-AHP based framework will help industry experts to revise and improve their strategies to make the DevOps process sustainable.


I. INTRODUCTION
Software industry is always looking for effective and flexible ways to develop quality software within limited time and cost. Recently, DevOps paradigm has gained popularity in software development process [1], [2]. DevOps provides platform for both development and operation teams to work collaboratively to develop software products. DevOps facilities cross functional shared responsibilities and trust between both types of development and operation teams [3]. DevOps substantially extends the continuous development goals of the agile movement by supporting automation of continuous integration and release processes [4], [5]. Leite et al. [6] defined DevOps as: a culture effort that automate organization infrastructure and the processing cycle of software development, guaranteeing the reliability of The associate editor coordinating the review of this manuscript and approving it for publication was Mahmoud Elish . software product. DevOps offer several benefits to software organizations such as more focus on implementation and frequent release. Moreover, DevOps also automate the build, testing and deployment processes [7]. Forsgren [7] stated that automated development process assists to reduce the human effort and enable the automated deployment according to the schedule.
Likewise, it is emphasized that the automatic development environment significantly contributed towards the development and quality of software applications [8]. The sustainable DevOps execution allow software organizations to deliver frequent small releases which helps improve visualization of modules to the end-user [9]. The small and frequent deployment offers the development teams to receive appropriate suggestions from client which assists to modify overall quality of a product [10]. In-spite of several benefits associated with sustainable DevOps, software practitioners face numbers of challenges for the sustainability VOLUME 10, 2022 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ of DevOps process such as ''fear of change'', ''conceptual deficit'', ''blame game'', and ''complex and dynamic environments'' [11]. Similarly, Jabbari et al. [12] stated that communication gap and heterogeneous environments are critical challenges for sustainable DevOps implementation in software industry. Despite challenges associated with DevOps sustainability in software industry, several well-established organizations such as Etsy, IBM, Netflix, and Flickr have successfully adopted it [13]. For example, in Flickr, effective communication and collaboration in both development and operations practitioners have helped the organization to decrease the release time. The implementations of DevOps practices in different organizations revealed that sustainable DevOps implementation enhances the systems quality and delivery process [13], [14]. Erich et al. [13] pointed out that practices for sustainable DevOps are rapidly being adopted by software organizations with an aim to gain benefits with them.
CAMS (Culture, Automation, Measurement, Sharing) presents the core areas of DevOps [15], [16]. Rafi et al. [17], Akbar et al. [18] and Plant et al. [19] indicated that to make the DevOps process sustainable the organizations need to focus on CAMS areas. Thus, the importance of sustainable DevOps process in real-world practices motivated us to conduct comprehensive systematic research to investigate and analyses the guidelines reported in the state-of-art and practices. The objectives of this study are: (1) to conduct a systematic literature review and questionnaire survey approach to explore and verify best practices of sustainable DevOps process; (2) to prioritize the investigated best practices using fuzzy-AHP approach; and (3) to develop a decision-making framework based on the rankings of best practices. As, there is no study conducted to prioritize the best practices of DevOps process. We have filled the research gap by applying the fuzzy-AHP method in DevOps process areas. We believe that the deep understanding of the DevOps best practices will assist the practitioners to manage the DevOps activities effectively and efficiently. The prioritization of best practices provides the rank order, which helps the practitioners to consider the most significant practices which are critical for the successful execution of DevOps process. To reach-out the study objectives, the developed research questions are as follow: [RQ1] What guidelines for sustainable DevOps implementation in software development organizations are reported in the literature and industry practices? [RQ2] How the explored guidelines were prioritized using fuzzy-AHP? [RQ3] What would be the prioritization-based framework for sustainable DevOps guidelines? The paper is organized as: study background is reported in section 2. The used research methodologies are discussed in section 3. The results and analysis are presented in section 4. Summary of the study findings is shown in section 5. Section 6 presents the threats to validity of study findings. The conclusions and future direction of the study are summarized in section 7.

II. BACKGROUND AND MOTIVATION
Software organizations have shown interest in adoption of software development approaches with reduced development and delivery cycle. The basic intention behind the adoption of new development approaches is the rapid change in the customers' requirements and the consideration of requested change in positive manner. Agile development approaches have been adopted by software industry to address the rapid change concern in software development life cycle [20]. The idea and the success of continues delivery comes-up with a new software development strategy known as DevOps. DevOps is a new software development methodology which focuses on collaboration between Developer and Operation teams to work in an environment where they can share goals, processes, and tools [4], [9], [21]- [23]. In software industry, experts treat DevOps as the cultural movement that assists the development environment concerning with effective communication, control, and responsibilities [20], [24]. Various studies have reported that the collaboration, automation, and services are the key aspects of DevOps [9], [25].
Dyck et al. [26] mention that the revolution caused by the DevOps significantly contributed to enhance the level of trust among practitioners that assist to transform and change the development environment in software organizations. Furthermore, Smeds et al. [23] ''highlighted that the DevOps is not only a culture change it also helps to improve the development process. Literature also reported the limitation and importance of DevOps paradigm [27]- [29]. According to Banica et al. [30], the main advantages of DevOps are product quality services and continues bonding. Similarly, Gupta et al. [31] mention that the DevOps supports in trust building between Dev and Ops practitioners. Moreover, they explored and ranked DevOps attributes that are critical to evaluate the readiness considering the adoption of DevOps in an organization. Furthermore Gill et al. [32] expressed that the DevOps contributed to develop the bridge between Dev and Ops teams that overcome the communication and coordination gape between practitioners. Wiedemann et al. [33] argued that, the DevOps provides the roadmap for project management team to support better performance, understandability, integration, relationships among teams. However, there is a need of strong collaboration, trainings, skills and effective automation to adopt DevOps practices in a practical way.'' The organization adopting DevOps also faced several critical challenges [32]. Gill et al. [32] highlighted that the process and procedure-related challenges, cultural conflicts, and the problems in operational models.
The existing literature portrays evidence-based research in the context to explore the guidelines for DevOps sustainability in software organizations. Furthermore, no research has been done to analyze the sustainable DevOps guidelines using the fuzzy-AHP approach. This detailed empirical investigations and analysis, will help the teams to understand and develop the methodologies for sustainable implementation of DevOps in software development industry.

III. RESEARCH DESIGN
The research design is outlined in Figure 1. First, a SLR was conducted to identify the best practices associated with DevOps projects. Next, a questionnaire survey study was conducted to get feedback from industry practitioners on the identified DevOps best practices. Finally, the fuzzy-AHP was used to prioritize the identified best practices.

A. SYSTEMATIC LITERATURE REVIEW (SLR)
In this study, we have used the guidelines developed by Kitchenham and Charters [34] to conduct the SLR. The SLR consists of three phases namely, ''planning the review'', ''conducting the review'' and ''reporting the review''.

1) PLANNING THE REVIEW
Planning refers to developing the protocols adopted to collect and analyses the data. The following review protocol steps were carried out to extract and analyses the literature to answer the proposed research question.

a: DATA COLLECTION SOURCE
Selection of appropriate data sources is essential to identify literature related to the research objective of the study. In this study, we followed guidelines of Chen et al. [35] and Zheng et al. [36]; and following digital repositories were selected to search for related primary studies. We followed the guidelines presented in [36], [37] to develop search string for the study. First, key terms were identified from relevant studies [1], [13], [30], [32], [38]. Next, we used the ''AND'' and ''OR'' operators to formulate the search string by using key terms and their synonyms as follows: The quality assessment process was performed to decide suitability of the selected primary studies concerning to the study objective. The QA process is performed based on Kitchemhm and Charctros [34] guidelines. The QA process was performed based on consists of five questions as shown in Table 1. Detailed results of the QA process are presented in Appendix-A.

2) CONDUCTING THE REVIEW a: FINAL STUDY SELECTION
Primarily, 860 studies were extracted in the response of the search string executed on the selected databases. The collected literature was further refined by applying the tollgate approach developed by Afzal et al. [39]. The tollgate approach consists of five phases, and each step is performed carefully, aiming to select the studies for data extraction finally. A total of 71 studies were selected for the final data extraction process as shown in Figure 2. The list of selected studies and their QA score is given in Appendix-A.

b: DATA EXTRACTION AND SYNTHESIS
The selected 71 studies ( Figure 2) were carefully reviewed to extract relevant information to answer the research questions of the study. Author one and three of the study were involved in the data extraction phase, while authors two, four and five validated the extracted data. First, main theme concepts and practices were identified from the selected studies. Next, we synthesized the collected data into 48 best practices for implementing DevOps projects.
In order to avoid potential bias in the study, we performed the ''inter-rater reliability'' test [39]. Three external experts randomly selected 12 studies and performed the data extraction process. Next, we compared findings of research team with external experts by applying the nonparametric ''Kendall's coefficient of concordance'' (W) [40]. ''The value of W=1 indicates the complete agreement, and W=0 indicates complete disagreement''. The results of W=0.84 p=0.003 shows a significant agreement between findings of research team and external experts. The used code is given in this link: https://tinyurl.com/y5fct4ql.

3) REPORTING THE REVIEW a: QUALITY OF SELECTED STUDIES
The quality of the selected studies assess using the criteria given in Table 1. According to the results given in appendix-A, 70% of selected studies scored more than 75%; and this shows that the selected studies are potential sample of literature to address the study objective.

b: EXTRACTED DATA
By carefully reviewing the each selected study, 48 best practices were identified. The list of the explored best practices is given in Table 4. The identified best practices gives the guidelines to the practitioners for the successful execution of DevOps process in real-world environment.

B. EMPIRICAL STUDY 1) QUESTIONNAIRE SURVEY
A questionnaire survey was developed to seek feedback from industry practitioners. The survey participants were asked to rank the DevOps best practices identified from the SLR study. First, the questionnaire survey was tested through a pilot study involving one developer from academia (Chongqing University, Chain) and two from industry (Virtual force-Pakistan and QSoft-Vietnam). Next, based on the feedback received from the pilot testing phase, final version of the questionnaire survey was prepared. The survey is divided into three sections. First section collects demographic data, second section seeks feedback on the identified best practices, and the third section included an open-ended question that allowed the participants to include additional best practices or comments. Final version of the survey is presented in Appendix-B.

2) DATA SOURCES
The target population for the survey was software practitioners with experience in DevOps projects. The participants of the study were recruited by using the snow balling techniques [41]- [43]. The data collection process was executed from December-2020 to February-2021. The completed surveys were manually reviewed for completeness and five incomplete responses were rejected. Finally, 93 responses were used for further data analysis process. The bibliographic information is presented in section-4.2.

3) SURVEY DATA ANALYSIS
In this study, we used the frequency data analysis approach to analyze the collected responses, as it is considered the effective way to compares the responders opinions in between the variables and across the group of variables [44]. The same approach has been adopted in the existing studies [45]- [47].

C. PHASE 3: FUZZY SET THEORY AND AHP
The fuzzy-AHP technique has been adopted in various other research domain for solving the complex decision-making problems in production houses, managerial policies and numerous other areas. For example, for selection of intelligent building systems by Wong and Li [48], prioritizing the key success factors of software projects by Yaghoobi [49], clinical engineering health technology projects assessment by Sloane et al. [50], selection and evaluation of the project in mechanical engineering by Palcic and Lalic [51], risk analysis and management of engineering projects by Wen-Ying [52], prioritizing the challenging factors of agile development in distributed software development the context by Shameem et al. [53], prioritize the coordination barrios of humanitarian supply chain management by Kabra et al. [54], improve the human decision-making problems by Albayrak and Erensal [55]. Thus, to prioritize the identified best practices of DevOps, we applied fuzzy AHP as it is successfully adopted to address the multi-criteria decision-making problem in various engineering domain. The implementation process of fuzzy AHP steps is discussed in this section.

1) FUZZY SET THEORY
The Fuzzy set theory is an extended version of classical set theory that's initially proposed by Zadeh et al. [56]. That was oriented to fix the vagueness of uncertainties of ear world practices using multicriteria decision making problems.
The basic input of Fuzzy set theory is to epitomize the vague data. In the fuzzy set, a membership function µF(x) is characterized, which maps an object between 0 and 1. The protocols of fuzzy set theory along with definition are presented in sub-sequent sections: Definition: ''A triangular fuzzy number (TFN) F is denoted by a set (vl, vm, vu)'', as presented in Figure 3. The given equation Defines the membership function µF(x) of F.
where v l , v m and v u are the crisp numbers denoting the lowest, most promising, and highest possible values respectively. The ''algebraic operational laws using two TFNs, namely (V 1 , V 2 ) are given in Table 2.''

2) FUZZY ANALYTICAL HIERARCHY PROCESS (FAHP)
FAHP is the most effective and powerful approach used to solve the multicriteria decision making problems. The key benefit of FAHP is the relative ease with which it manages the multiple criteria, easier to understand, and it can efficiently manage both qualitative and qualitative data. The following primary step of FAHP approach: ''Step1-Decompose the complex decision problem into the hierarchical structure'' (   However, conventional AHP has numerous advantages [57]- [59], but it also faced some core limitations as it is based on the ''Crisp environment'', ''Judgmental scale is unbalanced'', and the ''absence of uncertainty'', and because of these limitations the selection of judgment is subjective. The FAHP was developed to address these limitations of AHP to get results more effectively and accurately [51]. The FAHP deals with uncertainties, imprecise judgment of different experts by handling the linguistic variables. FAHP approach has been considered in different context [49], [50], [52], [54]. To address the uncertainties and vagueness we have used the FAHP suggested by Chang [60] that provides more appropriate and consistent results compared with other FAHP approaches. In a prioritization problems, let X = {v 1 , v 2 . . . , v n } signify the ''elements of main categories as an object set and U = {t 1 , t 2 . . . , t n } shows the elements of each category as a goal set. Considering the Chang [60] approach, every object is measured, and extent analysis for each goal (gi) is executed, respectively. Thus, for each object, there are (m) extent analysis values that can be obtained with the following Equation'' (2) and (3): T 1 gi T 2 gi T m gi , i = 1, 2, . . . , n (3) where, all F j gi, (j = 1, 2, . . . , m) are fuzzy triangular numbers (TFNs).
The following are the critical steps of Chang's extent analysis method [60]: Step 1-The element of fuzzy synthetic extent (S i ) for the i th object using Eq. (4): To achieve the expression m j=1 V j gi , execute the fuzzy addition operation of m extent analysis using Eq. (5): and to make the expression , the fuzzy addition operation is performed on V j gi (j = 1, 2, . . . , m) value, as follow using Eq. (6): Finally, calculate the inverse of the vector with the help of Eq. (7): Step 2-As F a and F b are two fuzzy triangular numbers, then these fuzzy numbers need to be compared that is knows as Degree of possibility i.e.
Here, d indicate the highest intersection point between D, µV a, and µ Vb ( Figure 5). The values of T 1 (V a ≥ V b ) and T 2 (V a ≥ V b ) are compulsory for determining the value of P 1 and P 2 .
With the help of Eq. 12, calculate the weight vector.
Step 4-The normalized weight vectors are calculated using Equation 13, and the result will be a non-fuzzy number (known as defuzzification) which represents priority weight for the criteria:'' where W is a non-fuzzy number.
Step 5-Checking consistency ratio: The pairwise matrices should always be consistent in fuzzy AHP [53]. Therefore, it is necessary to check the consistency ratio of each pairwise comparison matrices [61], [62]. To do so, the graded mean integration approach is utilized for de-fuzzifying the matrix. A triangular fuzzy number, denoted as P = (l, m, u), can be de-fuzzified to a crisp number as follows: After the defuzzification of each value in the matrix, consistency ratio (CR) of the matrix can easily be calculated and checked whether CR is smaller than 0.10 or not. For this, two primary parameters, i.e., consistency index (CI) and consistency ratio (CR) are used, which are defined using Equations 14 and 15, respectively.
where, I max : the largest eigenvalue of the comparison matrix,'' n: the number of items being compared in the matrix RI: the random index and its value can opt from Table 3.
To be a consistent matrix, the computed value of CR should less than 0.10. If the value of CR is found to be higher than 0.10, the decision-maker has to make the pairwise judgments again.

IV. RESULTS AND ANALYSIS
A total of 48 best practices were identified from the literature. We also map the best practices into different categories of CAMS framework (culture, automation, measurement and sharing) [17], [63]. The coding based scheme [64] was used to map the identified DevOps best practices in the core categories of CAMS. The mapping team consist of three authors of this study (Author no.1, 3 and 4). The mapping results are given in Table 4.

A. RESULTS OF EMPIRICAL INVESTIGATIONS 1) RESPONDENTS' BIBLIOGRAPHIC INFORMATION
In the questionnaire survey, we had participants are from 20 different countries, as shown in Figure 6. Moreover, we had 26 (22%) participants from small organizations, 49 (42%) are belongs to medium organizations, and 41 (35%) are from large scale organizations, as shown in Figure 7.

a: RESPONDENTS WORKING EXPERIENCE
The results presented in Figure 8 shows the experience of survey respondents' range between 2 to 20 years. The mean and medium were calculated, and the results (6 and 5.5 respectively) indicate the young pool of the respondents. Thus, there is a good combination of survey participants having different experience levels related to software development activities.

b: RESPONDENT's DESIGNATIONS
Cois et al. [38] mention that the responses are varied with respect to the designation of participants. Gupta et al. [31] reported that a respondent could only be measured appropriately if the participants deal with it frequently. The analyzed results show that most of the survey participants either project manager or software developers. The detailed results are shown in Figure 9.

2) RESPONDENTS FEEDBACK
Questionnaire survey study aimed to get the feedbacks of experts about the identified best practices and their categories. During the data collection process, a total of 116 complete response were considered for further analysis. The collected responses were summarized into three core categories, i.e., Positive (strongly agree, agree), neutral, Negative (strongly disagree, disagree) Table 5. The results presented in the positive category shows the opinions of those participants who agree with the identified best practices identified via SLR and their categories. The responses presented in the negative category shows the opinions of those respondents who do not agree with identified best practices their categories. The results presented in the neutral category shows the responses of those participants who do not have any idea with the impact of identified factors.
The results of the empirical study presented in Table 5 shows that most of the survey participants agree as   the reported best practices could positively influence the adoption of DevOps in software organizations. It is observed that BP41 (Enterprises should focus on building a collaborative culture with shared goals, 91%) is reported as the most important best practices from the survey participants. We further noted that BP9 (Emphasize Quality Assurance Early, 88%) and BP40 (Keep All Teams on the Same Page, 88%) are declared as the second highly considered best practices by the survey respondents.
Moreover, it is noted that C4 (Culture, 93%) is the most important category of the investigated best practices as considered by the survey participants. C3 (Sharing, 88%) and C1 (Measurement, 84%) is declared as the second and third highest regarded as categories of the best practices considered by the survey participants.

B. APPLICATION OF FUZZY-AHP
The fuzzy-AHP approach was used to prioritize the investigated best practices with respect to their significance for the   success and progression of DevOps paradigm. To perform the fuzzy-AHP analysis, we used the ''MATLAB R2016b programming environment developed by math works is an American privately held corporation'', which was executed on a computer with an Intel Corei3, 3.5-GHz processor and 8-GB memory. The adopted phases of Fuzzy-AHP approach are presented in the subsequent sections.

1) STEP-1 PROPOSED HIERARCHY STRUCTURE OF REPORTED BEST PRACTICES AND THEIR CATEGORIES
To apply the fuzzy-AHP, the critical session making problem is arranged in a hierarchy structure (as presented in Figure 4). The proposed hierarchy structure (Figure 10) was developed by considering the investigated best practices and their core categories. The main objective of the study is found on the first levels (i.e., prioritization of DevOps best practices), the categories and their corresponding best practices are given on level-2 and level-3, respectively. The proposed hierarchy structure is presented in Figure 10.

2) STEP-2 CONDUCTING THE PAIRWISE COMPARISON
The purpose of this study is to prioritize the identified best practices and their categories concerning their significance for the successful implementation of DevOps paradigm. To perform the pairwise comparison (for fuzzy-AHP analysis), we have developed a questionnaire and contacted respondents of the first survey. A total of 29 responses were received from the survey participants. All the responses were manually reviewed to check for incomplete data. We found that all the 29 responses were complete. A sample of the pairwise questionnaire survey (second survey) is given in Appendix-C. Small sample size can be one potential issue with the application of fuzzy-AHP analysis. However,   several existing studies have used a similar size of the dataset to perform the AHP analysis [48], [65]- [67]. For example, Shameem et al. [53] conducted an AHP analysis to prioritize the influencing factors of distributed agile software development based on the responses collected from five experts.
Similarly, Cheng and Li [66] prioritize the success factors of construction partnering by considering the data collected from nine experts. Lam and Zhao [67] conducted a survey study with eight experts to investigate the influencing factors of teaching quality. Moreover, Cheng and Li [66] conducted an AHP analysis for the selection of intelligent buildings system by considering the responses collected from nine experts. Therefore, we have performed a fuzzy AHP analysis by considering the data collected from 29 experts which is an acceptable sample size for generalizing the results of this study.
The data collected via the fuzzy AHP survey were transformed in geometric mean to evaluate the pairwise comparison of the DevOps best practices and their respective categories. The geometric mean is useful to transform the expert's judgments into TFN numbers; the formula used to apply the geometric mean is given below: (17) a=Weight of each response n=Number of responses. Linguistic variable against their triangular fuzzy Likert scales is given in Table 6. To develop the pairwise comparison matrixes of the reported best practices and their respective categories; the triangular fuzzy conversion scale (Table 6), proposed by Bozbura et al. [68] was adopted.

3) STEP-3 CALCULATING THE LOCAL PRIORITY WEIGHT OF EACH BEST PRACTICE AND THEIR RESPECTIVE CATEGORIES: A NUMERICAL EXAMPLE
The priority vector is calculated using the pairwise comparison matrix. The pairwise comparisons of the best practices' categories are presented in Table 7 and the priority vector of the categories of best practices presented in Table 10. Local Priority Weight (LPW) of all the main categories of the best practices were calculated using Equation 3. First, the synthetic extent values of four categories, i.e. measurement, automation, sharing and culture, were determined, and the priority weight of each category was calculated using The degree of possibility using Equation 6 is determined. The minimum degree of possibility (priority weight) for each pairwise comparison was calculated using Equation 8. Therefore, the weight vector was determined as W' = (1, 0.030019, 0.69836, 0.36405) ( Table 8). When these values were normalized, the importance of attributes was calculated as W = (0.4789, 0.01435, 0.3337). The given results reveal that ''culture'' is the most significant category as it has highest priority weight as compared to the other categories of the best practices.

4) STEP-4 TEST THE CONSISTENCY OF THE PAIRWISE MATRIX
In ''this section, we presented a step-by-step calculation of the procedure followed to check whether a given pairwise matrix is consistent or not. For this, we have considered the Table of Categories (Table 9). A triangular fuzzy number of the pairwise comparison matrix of the main categories is defuzzified to crisp number using Equation 14 and obtained the corresponding Fuzzy Crisp Matrix (FCM) as shown in Table 9: The largest Eigenvector (λ max ) value of the FCM matrix is calculated by calculating the column sum of each column of FCM matrix (Table 9) and then divide each element of FCM matrix by column sum. Moreover, the priority weight is calculated by taking the average of each row'', as shown in Table 10.
where, Cj= sum of the columns of Matrix [C] (Table 7), W= weight vector (Table 10), therefore λ max = 2.7 * 0.11591+ 7.0 * 0.29500+ 3.7 * 0.17028+ 5.2 * 0.41882= 4.1067 Based on the calculation, the largest Eigenvalue (λ max ) of the matrix FCM is 4.1067. The dimension of FCM is 4. Therefore n=4 and the Random Consistency Index (RI) is 0.9 for n=4 (Table 3). Therefore, equation 15 and 16 are used to calculate the consistency index and consistency ration as follows: The calculated value of CR is 0.039503<0.10; therefore, the pairwise comparison matrix developed for the categories of best practices is consistent and acceptable. Similarly, the consistency ratio for all the categories of the best practices is checked, and the results along with a pairwise comparison of VOLUME 10, 2022   measurement, automation, sharing and culture categories are presented in Table 11, 12,13 and 14.

5) STEP 5-CALCULATING THE GLOBAL WEIGHTS
The local weigh (LW) is used to determine the ranking of a particular best practices within their specific category, and the global weight (GW) presents the impact of the best practices on overall study objective (i.e., prioritization of DevOps success factors). The GW is used to determine the final ranking of the best practices compared with all the investigated 48 best practices beyond their categories. The GW is calculated by    multiplying the LW of a best practice with their category weight. For example, the LW of BP1 (Organizations start DevOps practices with small projects, 0.099531) and the category weight is C1 (Measurement, 0.11591); so, the GW of BP1= (0.099531) × (0.11591) = 0.011537. By comparing the local rank of BP1 within their category, it is ranked as the second-highest priority best practices.
While comparing its GW with all other 48 best practices, it stands out 39 th most important best practice for the successful implementation of DevOps paradigm. The results presented in Table 15 shows that the GW of BP41  (Enterprises should focus on building a collaborative culture with shared goals, GW=0.041591) is the highest priority best practice for DevOps adoption and its progression in software organizations. Moreover, BP44 (Assess your organization's readiness to utilize a microservices architecture, GW=0.039183) and BP 38 (Educate executives at your company about the benefits of DevOps, to gain resource and budget support, GW=0.038798) are declared as the second and third most significant best practices for DevOps paradigm. The final ranking of all the other best practices is presented in Table 15.

V. SUMMARY AND DISCUSSION
Summary of the findings for each question is presented as follows: RQ1 (What guidelines for sustainable DevOps implementation in software development organizations are reported in the literature and industry practices?) As a result of the SLR, a total of 71 studies were identified. The primary studies were carefully reviewed, and a total of 48 DevOps best practise were identified. The best practices were further categorized in the core categories of CAMS model (i.e., Culture, automation, measurement and sharing). The mapping of the best practices into CAMS is used to develop the hierarchy structure required for the fuzzy-AHP.
RQ2 (How the explored guidelines were prioritized using fuzzy-AHP?) A questionnaire survey was also conducted to seek feedback from practitioners on the identified best practices and their respective categorization. The survey results indicate that industry practitioners agree with the identified best practices and their respective categorization.
RQ3 (What would be the prioritization-based framework for sustainable DevOps guidelines?) The step-by-step protocols of fuzzy-AHP was applied to prioritize investigate the DevOps best practice. To perform the fuzzy-AHP analysis, the pairwise matrixes of the best practices of each category were developed based on the expert's opinions. All the steps of fuzzy-AHP were carefully applied and the priority weights of each best practice were determined. By applying the fuzzy-AHP analysis, the prioritization weight (global weight) of each best practice was determined. The results show that BP41 (Enterprises should focus on building a collaborative culture with shared goals, GW=0.041591) is the highest priority best practice for DevOps adoption and its progression in software organizations. Leite et al. [6] highlighted that DevOps required a cultural change in the software development organization, as it offers continues and a collaborative work environment between the developers and operators. Gupta et al. [31] and Marijan et al. [69] also highlighted the importance of collaborative culture for the successful adoption of DevOps paradigm. Moreover, BP44 (Assess your organization's readiness to utilize a microservices architecture, GW= 0.039183) and BP38 (Educate executives at your company about the benefits of DevOps, in order to gain resource and budget support, GW=0.038798) are ranked as three most priority best practise of DevOps paradigm.
The framework of the investigated best practices was developed by using both global and local ranks (Table 15). The objective of framework development is to show the impact of each best practice in their own category and for overall DevOps paradigm. For example, BP1 (Organizations start DevOps practices with small projects) is locally ranked as the 2nd most important best practice for the successful implementation execution of DevOps paradigm. An interesting observation is that BP1 is ranked as the 39th.  Similarly, BP2 (Include modelling for legacy infrastructure and applications in your DevOps plans) is declared as the 4th most important best practices in 'Management category' and ranked as the 42ndwith respect to global rankings.
The local and global ranks of each best practice are presented in Figure 11, which renders the impact of a particular best practices within their respective category and for overall project compared with all the identified 48 best practices. Moreover, C1 (Measurement, CW=0.41882) is ranked as the most significant category of the motivators. Furthermore, it is observed that C2 (Automation=0.295), C3 (Sharing, CW= 0.17028) is ranked as the second and third most significant categories of the best practices. Hence, the critical focus on these areas could assist the organization in the successful execution of the DevOps paradigm.

VI. THREATS TO VALIDITY
One of the limitations of the study is potential researcher's bias in the investigated best practices using a literature review study. To address this comment, the ''inter-rater reliability test'' was performed, and the results shows no significant biasness in the literature study findings. Another potential threat to validity is the potential to generalize the study results. The best practices are identified by applying a well-established SLR method. Moreover, the identified best practices were further validated by seeking input from 116 industry practitioners. the generalization of the questionnaire survey is the small size of the data set. Moreover, the fuzzy-AHP was performed to rank the investigated best practices and their respective categories considering the experts opinions. The consistency ratio of pairwise comparison matrixes was determined and the results presents the acceptable internal validity of fuzzy AHP analysis results.

VII. CONCLUSION AND FUTURE DIRECTIONS
DevOps is an approach which combines development and operations to enable agility during software development process. The implementation of DevOps practices is complex, and this motivates us to explore the best practices that are important for success of DevOps paradigm in software organizations. As a result of the systematic literature study, a total of 48 best practices were identified. The identified best practices were further mapped in the core categories of CAMS model. Moreover, the questionnaire survey study was conducted to get the insight of experts on the identified best practices. The results of the questionnaire survey study indicated that the identified best practices are in line with real-world practices. Finally, the investigated best practices were further prioritized with respect to their significance for DevOps practices using fuzzy-AHP. The prioritization results show that 'enterprises should focus on building a collaborative culture with shared goals', 'assess your organization's readiness to utilize a microservices architecture' and 'educate executives at your company about the benefits of DevOps' are important best practices. The categorization of investigated best practices and their rankings provides a framework that could assist the academic researchers and industry practitioners in revising and developing the new effective strategies for the sustainable DevOps process in software organizations.
As part of future work, we plan to conduct multivocal literature review and case studies to explore the additional best practices associated with DevOps paradigm. In addition, we plan to identify the critical challenges and success factors that need to be addressed for the successful execution of DevOps practices in software organizations. Ultimately, we plan to develop a readiness model which will assist the practitioners in assessing and improving their DevOps implementation strategies.