Dynamic Performance Evaluation of Blockchain Technologies

In recent years, the rapid development of blockchain technologies have attracted worldwide attention. Its application has been extended to many fields, such as digital finance, supply chain management, and digital asset transactions. For some enterprises and users, how to choose the most effective platform from many blockchains to control costs and share data is an important issue. To comprehensively evaluate the blockchain technologies, we first construct three-level evaluation indicators in terms of technical, market, and popularity indicators. Then, we propose an improved global DEA-Malmquist index without explicit inputs to assess the dynamic performance of blockchain technologies. Finally, we carry out an empirical analysis to evaluate 31 public blockchains’ performance from May 2018 to April 2020. The results indicate that the overall performance of blockchain technologies is basically on the rise. Some blockchain technologies that have not yet received widespread attention have shown good dynamic performance.

With the mature use of big data and cloud computing, Swan summarized the advantages of blockchain technologies through theoretical research and pointed out that these technologies can promote the development of supply chain finance [3]. Pilkington introduced that public blockchain is an important part of the blockchain [4]. Dinh et al. pointed out that the blockchain is maintained by a group of nodes that do not fully trust each other, which is a shared digital ledger [5]. Smetanin et al. provided a systematic review of current blockchain evaluation approaches [6]. Previous studies on blockchain evaluation include both analytical and simulationbased approaches. For instance, many researchers have studied the application of queuing theory in the field of blockchain evaluation [7]- [14]. By introducing the Markov process and Markov decision process, researchers evaluated the blockchain system from the perspectives of stability and anti-attack ability [15]- [21]. By adopting the random walk model, researchers have studied double-spending attack issue in blockchains [22], [23]. Lu showed us the application of blockchain technology in IoT, security, data management and other fields [24].
It is important to note that previous studies mostly focus on a specific blockchain technology (such as Bitcoin, Ethereum, IOTA), or target a specific scenario (such as selfish mining, double-spending attacks). However, there are still few studies on comprehensive evaluation of blockchain technologies. China Center for Information Industry Development (CCID) released the first global public blockchain technology assessment index in May 2018 [25]. It is worth noting that although Bitcoin's technical ranking is low, it is still one of the most popular blockchains. Furthermore, Tang et al. designed 3 first-level indicators and 11 second-level indicators to evaluate the public blockchain and ranked the public blockchain through the TOPSIS method [26]. However, in Tang's study, a single period evaluation method was used instead of a global evaluation method, and the dynamic performance changes of blockchain technologies were not considered. For this end, we will introduce a non-parametric method to evaluate the dynamic performance of blockchain technologies.
Data Envelopment Analysis (DEA) is a non-parametric method to identify best practices of peer decision making units (DMUs), in the presence of multiple inputs and outputs.
Since it was first introduced by Charnes et al. [27], there have been numerous studies in many areas. DEA provides not only efficiency scores for inefficient DMUs, but also provides for efficient projections for those units onto an efficient frontier. Considering that the R&D process of blockchain is different from the traditional production process, it is difficult for us to obtain and quantify the actual investment in the blockchain technologies' research and development process. We will use the Index DEA model to convert the inputs of different decision-making units to 1, and then directly compare the output results [28]. In terms of output indicators, we will conduct a comprehensive analysis of blockchain technologies from three levels: Technical indicators (CCID: Basic-tech, Applicability, and Creativity), Market indicators (The average rate of return and The standard deviation of the rate of return) and Popularity indicators (Google search heat). Besides, the dynamic evaluation of blockchain technologies over time is an urgent problem to be solved. Then, we will further analyze the two elements in the Malmquist index, which are technology frontier shift and technical efficiency change [29], [30]. Through the technology frontier shift, the development or decline of all DMUs can be measured. While technical efficiency change is used to measure changes in technical efficiency. Although the standard DEA-Malmquist productivity index has been widely adopted, it has encountered an infeasible problem in calculating directional distance function (DDF) across periods [31]- [33]. To avoid the infeasible problem, we introduce the global Malmquist productivity (GM) index developed by Pastor and Lovell [34]. It should be emphasized that the global benchmark technology covers all benchmark technologies over the same period. In this way, the GM index gets rid of the problem of infeasibility [35]. Besides, some studies have also dealt with the problem of calculating DDF with DEA-Malmquist index [36]- [38].
Different from previous studies, this paper has the following contributions. First, we integrate DEA and global Malmquist index to evaluate the dynamic performance of blockchain technologies. Second, we adopt an index DEA model to calculate the global Malmquist index, according to the fact that the blockchain without explicit inputs in practice. Third, we further investigate the efficiency change (EC) and technical change (TC) to show the improvement of blockchain technologies over time. Finally, evaluate the performance of 31 public blockchains over 24 months. By adjusting the length of the observation period, we found that the development of different blockchain technologies shows different trends throughout the period and each subperiod. And some blockchain technologies that have not yet received widespread attention show good dynamic performance.
The rest of the paper is organized as follows. In section II, we introduce the proposed evaluation model. Section III presents data, empirical study, the results, and discussion. Section IV summarizes the study. Acronyms and abbreviations can be seen in the appendix, Table 6.

A. EVALUATION INDICATOR
With the development of blockchain technologies, more and more public chain projects have been launched. How to choose the best performing one among many public chain projects has become a new problem. To evaluate the blockchain project, we need to consider the performance of three aspects: technical performance of the blockchain, market performance of cryptocurrency, and popularity of the project. Therefore, we have constructed an indicator system from three levels. First, technical indicators, In Tang's research, the basic technology and applicability obtained from China Center for Information Industry Development (CCID) are used as two sub-indicators under the technical indicators [26]. Similarly, we obtain three indicators of basic technology, applicability, and creativity from the CCID index. They comprehensively described the technical level of blockchain. Second, market indicators, the market performance of cryptocurrency (issued through the blockchain project) can well reflect investors' expectations of blockchain projects, here we mainly consider two indicators, mean and standard deviation of the rate of return. Third, popularity, through Google Trends we can get the popularity of each blockchain project. Fig. 1 shows the indicator system we constructed.
The R&D process of blockchain technology is different from the traditional production process. Through the architectural design, protocol design, expansion design, application design and programming realization of the project team, a blockchain technology was born. Users join the decentralized blockchain network and conduct transactions with other network participants. The platform issues cryptocurrencies to incentivize users. At the same time, cryptocurrencies can also be traded between users. Based on blockchain technology, users can encrypt data before sending, and add identity verification in the transmission authorization process to prevent the influence of personal factors. Any operation involving personal data can be used for identity authentication for decryption and confirmation of rights. The characteristics of tampering and operation records and other information are recorded on the chain and synchronized to the block network. Blockchain technology can also be applied to many fields such as copyright protection, logistics chain traceability, and supply chain finance. However, it is difficult for us to obtain and quantify the specific data of the input in the R&D process (e.g. funds invested, the personnel involved, and the length of work). Plus, in the R&D process of blockchain technology, the most important thing is the design of consensus mechanism, account model, and smart contract. Different consensus mechanisms and different account models are not comparable to each other. Therefore, when designing the evaluation system, we shifted our focus to the output of blockchain technologies. Specifically, we comprehensively evaluate the performance of blockchain technology from the three perspectives: experts (ratings of experts), investor attitudes (the situation of blockchain corresponding to currencies) and social popularity (Google popularity).
The indicator system we proposed comprehensively considers a blockchain technology from technical status, market performance and popularity. The first level of indicators evaluates the current state of blockchain technologies and shows the performance of blockchain technologies from the perspective of experts. The second level of indicators evaluates the market performance of blockchain technologies. Through the fluctuation of cryptocurrency prices, we can see the confidence of investors in the development potential of blockchain technologies from the perspective of investors. The third level of indicators analyzes the popularity of blockchain technologies among the public, and the popularity directly determines the participants and investors that blockchain technologies can obtain. The index system formed by the three levels of indicators covers the evaluation of blockchain technologies from the perspectives of experts, investors, and followers, and comprehensively considers the current performance and development potential of blockchain technologies.
In this study, due to the non-parametric method we adopted, there is no need to compare the importance of indicators at different levels. In the following work, we will treat all six indicators as the desirable (undesirable) output of blockchain technologies in the evaluation model.

1) TECHNICAL INDICATORS
The technical evaluation of the blockchain project mainly considers the basic technical level, application level and innovation ability of the public chain, which correspond to CCID's evaluation indicators Basic-tech, Applicability and Creativity, respectively. These three indicators are quantified by the expert scoring method. Since CCID has established a technology assessment index for public blockchains [25], its scoring results will be viewed as the following three indicators. In the basic technology evaluation, the CCID index mainly examines the realization function, basic performance, security, and centralization degree of the public chain.

b: APPLICABILITY (y 2 )
In the application-level evaluation, the first-phase evaluation model mainly inspects and evaluates the application scenarios supported by the public chain, the number and ease of wallets, and the development support on the chain. In the evaluation of innovation, it mainly inspected the scale of public chain technologies' innovation team, code update status, code influence and other aspects.

2) MARKET INDICATORS
Issuing cryptocurrency is a common way for blockchain technologies to reward participants. Based on different consensus algorithms such as PoS, PoW, BOINC, the blockchain platform will issue cryptocurrencies to users. Users can also make profits through transactions after obtaining cryptocurrencies. The fluctuation of cryptocurrency prices reflects the attitude of investors towards the future development trend of corresponding blockchain technology.
With the development and popularization of blockchain technologies, cryptocurrency has gradually become a financial asset that has attracted much attention. The price fluctuations of cryptocurrencies are relatively large in all financial asset investments. The market performance of cryptocurrencies reflects investors' confidence in the blockchain technology behind the currency. The prices of cryptocurrencies are closely linked to the development of corresponding blockchain projects. In our evaluation system, we use the monthly mean of daily returns (y 4 ) and the monthly standard deviation of daily returns (b 1 ) to examine the market performance of cryptocurrencies. Since the cryptocurrency adopts a continuous trading system, we take the daily price at 0:00 as the opening price, and the 24:00 price as the closing price. After calculating the rate of return on a daily basis, we obtain the average of the monthly rate of return as y 4 . Similarly, we can obtain the monthly standard deviation of daily returns as b 1 . The former represents the profitability of virtual currencies, while the latter represents the stability of profitability. It should be noted here that the monthly standard deviation of daily returns (b 1 ) is an undesirable indicator. The smaller the value of this indicator, the better the cryptocurrency performance.

3) POPULARITY INDICATORS
In addition to the above indicators, the public's sentiment towards blockchain projects directly affects the development of blockchain projects. More attention means more participants and investors, which are very important for the development of blockchain projects. In the search market, Google handles around 90% of searches worldwide [39]. The popularity of search terms over time and across various regions of the world can be compared in Google Trends [40]. We use the average Google search heat (y 5 ) of a public blockchain to reflect the popularity of this blockchain.

B. EVALUATION PROCESS
In this special case, we cannot get explicit inputs of each blockchains in each period, which makes us need to modify traditional GM index. Next, we will introduce a new Malmquist index to deal with this problem.
Fist, we define the contemporaneous benchmark technology for period t as and define the global benchmark technology as the convex envelop of all contemporaneous technologies, i.e., Then the contemporaneous Malmquist index is defined as , whereas its global counterpart is defined on S G as Next, to deal with desirable outputs and undesirable outputs simultaneously, we introduce the Directional Distance Function (DDF) into this model. The directional distance function was first introduced by Chung, et al. [41]. DDF makes it possible for decision makers to decide the direction  of projection decision by adjusting the input/output direction vector. When considering only desirable and undesirable outputs, we define DDF on global technology T G and take the VOLUME 8, 2020 specific form as Since the global DDF utilizes all blockchains over all periods in constructing the production possibility set, it is computed for Blockchain o at period t under VRS assumption as Although the two adjacent periods refer to the global frontier when calculating the Malmquist index, the respective frontiers are still used in the calculation of efficiency changes: The closeness of the frontier of period t + 1 and the global frontier can be represented by The larger the ratio, the closer the frontier of period t + 1 is to the global frontier; The closeness of the frontier of period t and the global frontier can be represented by The larger the ratio, the closer the frontier of period t is to the global frontier; The change of the frontier from period t to period t + 1 can be expressed by the ratio of the above two ratios: The Malmquist index can be decomposed into efficiency changes and technical changes: Because all DMUs evaluated refer to the same global reference set, this Malmquist index does not have the infeasibility VOLUME 8, 2020 problem; because each period references the common global frontier, this Malmquist index is transitive, i.e.

III. EMPIRICAL ANALYSIS
In this section, we apply the modified GM index to assessing 31 public blockchains' performance from May 2018 to April 2020. which are Basic-tech (y 1 ), Applicability (y 2 ), Creativity (y 3 ), the monthly mean of daily returns (y 4 ) and Google search heat (y 5 ). And an undesirable indicator, the monthly standard deviation of daily returns (b 1 ). Table 1 shows the descriptive statistical results of each indicator, including the mean, median, maximum, minimum, standard deviation, kurtosis, and skewness. The standard deviation of y5 is the largest, while the standard deviation of other variables is relatively small. The skewness values of the variables are all not 0, and the kurtosis values are not 3, indicating that the data does not strictly obey the normal distribution. Therefore, by introducing the global Malmquist index, which is a nonparametric method, the evaluation of blockchain technologies will be more reasonable. Table 7 (see Appendix) presents more detailed descriptive statistics (geometric mean, standard deviation, maximum, minimum, and range) of thirty-one public blockchains in the data. As shown, under the Basic-tech indicator, the average level of EOS is the highest, while Bitcoin cash is the lowest. From the applicability of all aspects of the public chain, 31 blockchains are similar. In addition, EOS has the largest Creativity of 61.1, while the smallest of Bytecoin is only 1.1. It is worth mentioning that Bytecoin's Google search heat is also the smallest on average. In terms of the average rate of return, among the 31 public blockchains, only NEO, Bitcoin, and Tezos have positive values. The standard deviation of the rate of return reflects similar instability of the overall performance of the blockchains.

B. ANALYSES FROM THE MODIFIED GM INDEX
We calculated the modified Global Malmquist index of each year and its components EC (efficiency change) and TC (technical change) from May 2018 to April 2020. The whole period GM performance index is the geometric mean of all 24 months' GM indexes, namely, the 23rd root of the product of the growth rates from the 23 pairs of months (from May 2018 -June 2018 to March 2020 to April 2020). The results of the monthly GM index and its components EC and TC are presented in table 8, 9, and 10 (see Appendix) for the whole sample period.
To observe the overall situation of the dynamic performance of each public blockchain project, we calculated the geometric mean of the monthly indicators of each blockchain, as shown in Table 5. For a more detailed analysis, we divided the 24 periods into four stages (May 2018 to November 2018, November 2018 to May 2019, May 2019 to November 2019, November 2019 to April 2020), as shown in table 6.
From Table 2, we can see that most of the blockchain projects have improved throughout the observation period, and the top five with the largest average improvement are IOTA, Zcash, Hcash, XLM, and NULS. In contrast, the geometric averages of GMs of NEO, Decred, Ontology, Bytecoin, and Monero are all less than 1, meaning that from the overall level, these several blockchains have experienced different degrees of regression. Among them, the average GM of EOS and Ethereum are both 1, and there is no obvious improvement or regression from period 1 to period 24. Bitcoin, which has attracted a lot of people's attention, performed well in this evaluation. Its mean GM is 1.018. From the decomposition of GM, we can see that Bitcoin's progress mainly comes from technological progress. The mean EC of bitcoin is equal to 1, which means that from period 1 to period 24, bitcoin has not made significant progress compared to other blockchains. According to CCID's previous evaluation results, this is likely due to its slow transaction speed and flaws in the consensus mechanism. Table 3 gives more detailed information. In the stages of May 2018 to November 2018 and May 2019 to November 2019, many DMUs have a GM index of less than 1. In the second half of 2018, the China Banking and Insurance Regulatory Commission and other five ministries and commissions jointly issued the ''Reminder on the Risk of Illegal Fund Raising in the Name of ''Virtual Currency'' and ''Blockchain''''. At the same time, many countries have adopted a series of regulatory measures against the ICO projects, and these policies have caused a short-term negative impact on the development of blockchain. In the second half of 2019, the headquarter of the People's Bank of China issued the document ''Strengthening the Prevention and Control of Supervision and Combating the Trading of Virtual Currency'' to further strengthen the supervision of ICO projects. VOLUME 8, 2020  In the past, blockchain projects used cryptocurrency as the main reward mechanism. With the increasingly strict supervision of virtual currency transactions in countries around the world, blockchain projects have been directly and negatively affected in terms of market performance and popularity. In terms of technical indicators, due to the impact of negative news during these two periods, the basic technology, applicability, and creativity also declined. In terms of technical indicators, due to the impact of negative news during these two periods, the basic technology, applicability, and creativity also declined.
The other two stages only have two blockchains each with a GM index less than 1. Combined with the overall situation, Two periods of regression (May 2018 to November 2018, May 2019 to November 2019) may be caused by the transformation of blockchain technologies from ICO-oriented to application-oriented.
Bitcoin and Ethereum are the two blockchain projects that people are most concerned about. We list their GM index by period in Table 4. Bitcoin has 14 GM indices that are greater than 1 in 23 pairs of periods, while Ethereum has 15 pairs that satisfy GM indices greater than 1.The efficiency change of the two blockchain projects is 1, and by looking at the global efficiency of the two projects in each period, we find that they have their own advantages. Bitcoin is always the hottest, and the technical indicators of Ethereum are advantages. Therefore, they have the highest overall efficiency in each period, which is 1. This makes sense. Since GM is equal to EC times TC, in this case where EC is equal to 1, the GM changes of both projects come from TC.
In Table 5, we compare the rankings from three different blockchain technology evaluation methods. The second and third columns show the global Malmquist index and ranking of blockchain technologies (under the entire observation period). The fourth and fifth columns show the global Malmquist index and ranking of blockchain technologies (from July to August 2018). The last column is the ranking of blockchain technologies obtained by Tang using the Topsis method in August 2018.
Obviously, the blockchain technologies that have shown great progress on the overall level (GM index greater than 1) are not well ranked in Tang's method. For example, the top three, IOTA, Zcash, and Hcash, are respectively 23rd, 12th and 11th in Tang's single month ranking results. At the same time, in the single-period evaluation, the ranking results obtained by the global Malmquist index method are also very different from the Topsis method. Top three blockchain technologies, Litecoin, Stratis, and Hcash, are ranked 16th, 20th, and 11th respectively under Tang's method.
This difference comes from two aspects. The first is that Tang's method mainly focuses on the cross-sectional comparative evaluation of the current blockchain technology performance. However, the global Malmquist index method pays more attention to the dynamic performance of blockchain technology, in other words, the change between the previous performance and the current performance. This is very important in the dynamic performance evaluation of blockchain technologies. There is no doubt that the blockchain technology that can maintain continuous progress will have stronger competitiveness in the foreseeable future. The second is that Tang's method can only consider the performance of a single period at a time. However, due to the transitivity of the global Malmquist index, we can evaluate the performance of blockchain technology in any time span as needed. This advantage makes this method can more comprehensively consider the dynamic performance of blockchain technology. Based on the above factors, the dynamic performance evaluation of blockchain technology obtained under this method can provide some more noteworthy information.

IV. CONCLUSION
This paper provides a dynamic evaluation method for the user's selection process in many blockchain projects.
First, we propose a new indicator system for the dynamic performance evaluation of blockchain technologies. Through multi-perspective indicators, we can conduct a more comprehensive evaluation of the performance of blockchain technologies. Second, we apply the modified GM index method for the evaluation of blockchain technologies' dynamic performance. Unlike Tang's static evaluation method, our method can evaluate the dynamic performance of blockchain technology (the progress/regression of blockchain technology). Third, different from previous studies that focused on a single object and a single scenario, this study has achieved a multi-perspective comprehensive evaluation of blockchain technologies performance through a benchmark method.
Under the modified GM index, the application fields of the Malmquist index have been further expanded. In this study, we evaluated the performance of 31 public blockchains over 24 months. Compared with other scholars' research, our research results reflect more details. Accurately show the dynamic performance of each public blockchain among different periods; through GM index decomposition, the reasons for the improvement of blockchain performance are clarified; by adjusting the length of the observation period, we found that the development of different blockchain technologies shows different trends throughout the period and each subperiod. This allows us to evaluate the dynamic performance of blockchain technology in more detail.
This research still has certain problems. In the follow-up research, the index system should be further enriched. Compared with the second-hand data, the first-hand data can better show the performance of the blockchain. In future research, the impact of policy supervision on blockchain performance should be discussed in more depth.

See table 6-10.
ZHONGBAO ZHOU was born in 1977. He is currently a Professor with the School of Business Administration, Hunan University, China. His research interests include reliability modeling, system optimization, and decision making.
RUIYANG LI was born in 1995. He received the B.S. degree in mathematics and applied mathematics from Hunan University, China, in 2017, where he is currently pursuing the Ph.D. degree in management science and engineering with the School of Business Administration. His research interests include system optimization and decision making. HELU XIAO was born in 1986. He is currently an Assistant Professor with the Business School, Hunan Normal University, China. His research interests include system optimization and decision making.