Dynamic Metric Accelerated Method for Fuzzy Clustering1

Features in data samples usually need a unified dimension by a standardization process before clustering. However, there still exists a nonstandardized metric in which the distance between samples is greater than 1 after features are standardized. It is difficult to find the optimal search path if the data sample metrics are not standardized. To address this problem, we develop a dynamic-metric accelerated method for fuzzy clustering by introducing a metric matrix, whose diagonal elements consist of infinite norms of the metric matrix into the Fuzzy C-Means (FCM) clustering algorithm and its derived algorithms. More specifically, we focus on constructing a dynamic metric matrix that is used to unify the metric between data samples and updating cluster centers to optimize the search path of the cluster center. In addition, we propose a new evaluation index named the Coefficient of Variation Metric (CVM) to evaluate metric effectiveness. The dynamic metric accelerated method, whose complexity remains unchanged, can effectively accelerate the iteration speed of fuzzy clustering. The comparisons between the algorithm using the dynamic metric accelerated method and the corresponding algorithm on UCI, business district and COVID-19 CT image datasets show the superiority of the dynamic metric accelerated method in accelerating effect and clustering performance.


i. intrODUctiOn
Generally, clustering depends on a metric that describes the similarity between data samples [1] [2][3] [4]. Metrics can express rich information, and metric-based clustering approaches in which the metrics of data samples are the main component of objective functions have been widely used in industrial applications [5]. For the above reasons, research on clustering algorithms focuses on algorithm design and metric research [6].
What's more, Metric plays an more important part in fuzzy clustering. There are many metrics in fuzzy clustering, such as Euclidean distance [7], Minkowski distance [8], Manhattan distance [9], Chebyshev distance [10], Mahalanobis distance [11] [12] [13], angular cosine [14] [15] [16], correlation coefficient [17], entropy [18] [19] [20], and Hamming distance [21]. In addition, in recent years, many studies have been done on metric learning of the kernel method [22] [23] [24] [25] [26]. The learning process of the fuzzy clustering algorithm depends on the metric that describes associations between different objects. Fuzzy c-means (FCM) uses Euclidean distance to measure relationship between samples and distance centers [27]; SFCM measures relationship between data samples, distance centers and guidance information by using Euclidean distance [28]; eSFCM measures data samples and cluster centers and entropy to measure guidance information by using Euclidean distance [29]; SMUC measures relationship between samples and distance centers and entropy to measure guidance information by using Mahalanobis distance [30]; In addition, there are some kernel methods [31] [32]. However, the learning process of the relationship between data samples will be affected due to the limitations of metrics. The clustering algorithms based on Euclidean distance have the advantages of fast iteration convergence and stable results [33]. However, in the face of complex data in practical applications, these clustering algorithms are sensitive to data sample dimensions and cannot apply to nonlinear data [34]. The introduction of the Mahalanobis distance can solve these problems, but it will also lead to the exponential growth of algorithm computation requirements [30].
In the past, research mainly focuses on three kinds of metrics in fuzzy clustering: Entropy [35] [36][37] [38], Euclidean distance [39] [40][41] [42] , and Mahalanobis distance [43][44] [45]. Relative entropy measures the difference of data from the perspective of probability distribution and uncertainty [51][52] [53]; it has the advantage of being insensitive to common noises [54] [55].Euclidean distance is the most popular metric in objective functions of fuzzy clustering because it can reflect the real distance in the sample space [46]. However, there exists a case that Euclidean distance ignores the differences among different features in the sample, which makes it difficult to reflect the associations between data samples [47]. The Mahalanobis distance can better reflect the correlation between data samples, so the process of fuzzy clustering is not affected by the feature dimensions, but the calculation of the inverse matrix of the covariance matrix in Mahalanobis distance greatly increases the complexity of the calculation [48] [49] [50]. In addition, none of the three metrics mentioned above will change with the iteration. We call this kind of metrics the static metrics. In our research, the standardization of static metrics can achieve normalization at the feature level, but can not guarantee the normalization of distance. This paper analyzes this phenomenon and proposes an algorithm acceleration method based on dynamic measurement .
Our work is summarized as follows: 1) We introduce distance standardization and how it affects the clustering process using static metric algorithms 2) We propose a dynamic metric accelerated method for fuzzy clustering that has better adaptability than the traditional static metric. In addition, we analyze the time complexity and optimality of metric 3) For the dynamic metric, we define a Coefficient of Variation Metric (CVM) to evaluate the effectiveness of the metric.

4)
We use the dynamic metric accelerated method to improve the metric effect in clustering and set experiments on the classic UCI dataset, classic image dataset, business circle dataset [56] and COVID-19-CT dataset [57] to verify the effect of the dynamic metric accelerated method.
ii. rELatED WOrK a. fUzzy c-MEans cLUstEring (fcM) FCM is a widely used unsupervised fuzzy clustering algorithm. The Euclidean distance which is used in FCM is widely used in clustering algorithms.
Let us first assume that the sample set to be clustered is Euclidean space, and c is the number of clusters. The objective function of FCM can be expressed as [27]: For the convenience of introduction, the objective function is expanded as follows: where I is the identity matrix,A is a metric matrix ， only when A = I , eq.2 is equivalent to eq.1 .and the corresponding metric is Euclidean metric. VOLUME XX, 2017 1 mis any real number ( m>1) which denotes the degree of fuzziness, ij u is the membership degree of the j-th sample j x belonging to the i-th cluster whose centroid and ij u satisfies the following constraint condition: By minimizing (1) and using the Lagrange optimization, we obtain the following alternative update equations for the cluster center i v and the membership degree ij u : KErnEL-basED fUzzy c-MEans cLUstEring Of MEtric accELEratED (KfcM) KFCM is an unsupervised fuzzy clustering algorithm that projects the original data into the kernel space and takes the inner product of the kernel space as the metric of the algorithm.
Let V= {v 1 , v 2 ,...v c } be the V cluster centers in the kernel space. c is the number of clusters. This minimizes the following objective function subject to conditions as considered in FCM: is the Gaussian radial basis function, and its form is as follows: (8) In the iterative process of KFCM, the memberships and the cluster centers are updated as follows: c. sEMi-sUpErVisED fUzzy c-MEans cLUstEring (sfcM) SFCM is a semi-supervised fuzzy clustering algorithm that utilizes Euclidean distance to express the relationship of the membership matrix and the prior membership matrix. Its objective function is as follows: ,..., ( 1 cj j j u u u  , and ij u satisfies the following constraint condition: (12) m is the weighted index, which is an empirical value, and the value is usually 2. Then, we obtain the iterative formula of the membership degree and clustering center: is an a priori membership matrix, which is transformed from label information;  is a predetermined suppression coefficient.

D. EntrOpy
sEMi-sUpErVisED fUzzy c-MEans cLUstEring (EsfcM) eSFCM is a semi-supervised fuzzy clustering algorithm that utilizes information entropy to express the relationship of the membership matrix and the prior membership matrix. Its objective function is as follows: Among them,  is an empirical parameter.

E. sEMi-sUpErVisED MEtric-basED fUzzy cLUstEring (sMUc)
SMUC is a semi-supervised fuzzy clustering algorithm using Mahalanobis distance and entropy. By introducing the Mahalanobis distance on the basis of eSFCM [29],semi-supervised metric-based fuzzy clustering (SMUC) algorithm [30] was proposed, and its objective function is as follows: where is subject to conditions (2) and (6) and the following conditions: x u x u M (19) There is the following optimal solution [30]: iii. accELEratED itEratiVE MEthOD fOr fUzzy cLUstEring a. MOtiVatiOn: At present, clustering algorithms usually need to standardize sample features to eliminate the influence of different feature dimensions before conducting iterative calculations. After data normalization, the value of each sample feature is limited to [0,1] for iterative calculations, but the value of the sample distance will exceed [0,1] after calculations, as shown in Figure  It can be seen from Figure 1 that the standardized feature does not mean that distance (i.e., metric) between samples is also standardized.
To verify the necessity of standardizing metrics, we built artificial datasets. The distance between samples in dataset (a) was less than 1, and some of the distances in the dataset (b) were more than 1 (as shown in Figures 2 and 3): Synthetic dataset 2 in which the distance between samples is less than 1 Figures 2 and 3 represent the simulation dataset of dataset 1 and dataset 2, respectively, whose feature number is 2 (two-dimensional space) and sample number is 100. Let us assume that the number of cluster centers is 2, the initial cluster center is [0.3, 0.5; 7, 0.5], the maximum number of iterations of clustering in dataset (a) is 50, and the solution path-length of the two clustering centers is 0.0029. The maximum number of iterations of clustering in dataset (b) is 52, and the solution path length is 0.0097.
If the sample exhibits the phenomenon in Figure 3, then after the usual iterative process, the search path may become longer and indirect, as shown in Figure 4 : We are committed to making the solving path closer to Figure 4 (a) instead of Figure 4 (b).

b. MEtric cOMpLEXity
The Mahalanobis distance, which has a good effect on eliminating the dimensional effect, is commonly used to metric the continuous type of feature. At present, the Mahalanobis distance has a good influence on data of different dimensions, so the Mahalanobis distance is widely used for metric data of different dimensions. However, it is difficult for the Mahalanobis distance to apply to image recognition and other fields because it needs to calculate the covariance matrix and its inverse [30], resulting in a large amount of calculations (as shown in Figure 5).

c. DynaMic stanDarDizatiOn Of EUcLiDEan MEtric
In the iterative process of the algorithm, there exists a case in which the distance between the sample and the cluster center is greater than 1, so it is necessary to compress the distances so that all the distances after the transformation are in [0,1] (as shown in Figure 6). When the distance between the sample and cluster center is small, the rounding error produced in the calculation process will have a great impact on the clustering process. Therefore, it is necessary to expand the distance by transformation. The effect of stretching is shown in Figure 7: In Figures 6 and 7, the left side is a nonstandardized distance, and the right side is a standardized distance. To ensure that the mathematical relationship between distances is unchanged and that the algorithm is fast iterative, we satisfy the distance with the following relationship between the standardized distance and the nonstandardized distance:

D. DynaMic MEtric accELEratiOn
The Euclidean distance between the sample and the cluster center expression in clustering expands the corresponding metric is Euclidean distance. (19), the corresponding metric is the Mahalanobis distance.

2)When
3)When A is a diagonal matrix whose elements consist of the positive infinite norm of the distance matrix, the corresponding metric is a dynamic metric whose matrix changes with each iteration. We call the method using dynamic metrics for fuzzy clustering as the dynamic metric accelerated method for fuzzy clustering.
Calculation steps of the metric matrix in the dynamic metric accelerated method: 1) Compute the Euclidean distance between the sample and the cluster center to get the Euclidean distance matrix ; 2) Calculate the infinite norm according to the distance matrix; 3) Calculate the dynamic measure matrix A based on the infinite norm .
The Euclidean distance between j Then, the metric matrix A composed of the infinite norm of the metric matrix is as follows: where  || || D is the infinite norm of the metric matrix: Therefore, the distance calculation expression of the dynamic metric is:

iV. anaLysis Of WEighting EXpOnEnt M in fcM
The dynamic metric will inevitably affect the membership matrix when the iterative process changes, and the weighting exponent M of the FCM model plays a role in determining the validity of FCM partitions. Therefore, we explore the impact of the weighting exponent M in FCM when using the dynamic metric acceleration method. In this part, the effect of the value of M is analyzed through experiments using the Wisconsin breast cancer dataset 3 . The internal evaluation index data reconstruction error rate [59], the external evaluation index purity [60] and the number of iterations are used to analyze the influence of the m value on the dynamic metric and conventional Euclidean metric in FCM.

2) intErnaL EVaLUatiOn inDEX:
V RE is the error rate of data reconstruction, which is defined as follows: (31) It analyzes the difference between reconstructed data and the original data after clustering.    Table 1. Table 2 shows that with the increase in the m value, the clustering error rate of the FCM algorithm increases. At the same time, combined with Nikhil R. pal's conclusion [58], we can conclude that M = 2 is a better choice in the dynamic acceleration matrix.

E. EVaLUating inDicatOr anD VErificatiOn
To measure the effectiveness of the standardization of metrics, we analyzed the maximum and minimum distances between data sample and cluster centers.
FCM experiments were carried out on the Wisconsin breast cancer dataset 4 with 2 clusters. The maximum and minimum distance curves are shown in Figure 8.In order to better describe this process, we define some metric related concepts  Definition 1: Iterative max/min distance The maximum / minimum distance from the sample's t-th iteration to i v is:   (33) where T is the maximum number of iterations, However, due to Max_d . To avoid this case and express the intensity of the transformation between distances better, we normalize it to obtain the improved range. To express the normalization of the distance range conveniently, we define the following function: (35) The improved extreme deviations of the distance are: The improved distance extreme deviations diagram is drawn, as shown in Figure 9: The difference between the maximum and the minimum of the distance between the sample and each cluster center is increasing. The change process of the distance is shown in Figure 10.
We call the above distance as the Coefficient of Variation Metric (CVM). The CVM is used to evaluate this case. Before defining CVM , we need to explain the Variation Metric ) (VM .

 Definition 3: Variation Metric
As Figure 11 shows, we define the D-value as and its mathematical expression is: where t is the number of iterations. The smaller ) (t VM is, the smoother the iteration process and the better the normalization effect. (38) The coefficient of variation metric satisfies the following properties:
If the mean value of VM is 0. 5 In addition, the value of CVM is independent of the initial value One example of dynamic metric acceleration is the iteration process, and the variation of metric is shown in Figure 11.
A larger CVM represents a more stable iteration for the metric.
The CVM is calculated and shown in Figure 9 as: After acceleration by the dynamic metric in Figure   11, From the comparison of Figure 9 and Figure 11, we can see that the measured acceleration method effectively improves the optimization of the iteration. In order to explore the optimization effect of using dynamic indexes, we use FCM algorithm to experiment and try to verify its optimality. The dataset used in the experiment is shown on the left side of the table. The experiment reflects the optimal dynamic metric by using the change of internal and external evaluation indices before and after dynamic metric acceleration in multiple datasets.In parentheses, the left side is the algorithm without dynamic metric, and the right side is the algorithm with dynamic metric. We let 5   , 5 10   e , the maximum number of iterations is 100, λ=10, and randomly initialize cluster centers in experiments. Five labeled samples were used as supervisory information in all experiments.
We compare the traditional fuzzy clustering algorithm with the dynamic metric accelerated fuzzy clustering algorithm, FCM-M, for five datasets with the same m value. The results are shown in Table 4.The results are the mean of 10 repeated experiments. The experimental results show that the dynamic metric not only maintains the FCM clustering effect, but also achieves a better iterative speed for most data sets and ensures the standardization of the metric. Combined with the above results, we can conclude that the dynamic acceleration measurement method can achieve iterative acceleration without affecting the optimal solution, and can stably achieve the purpose of normalized distance.
Vi. DiscUssiOn On tiME cOMpLEXity In the previous section, we present a dynamic acceleration method based on infinite norms. This section discusses the time complexity of various metric calculations.
Here, n is the number of samples, d is the number of features, and c is the number of cluster centers.
The time complexity of the three kinds of metrics are shown in Table 1.

Vii. iinfLUEncE anaLysis Of EDgE pOints
The Euclidean distance of the sample points at the edge of the cluster is farther from other points, so change of the edge points will affect the infinite norm in the metric matrix, and our metric accelerated method depends on the norm in the dynamic metric. Therefore, this section discusses the impact of edge points in clusters on clustering speed and accuracy. In this part, the sample pairs far away from the main points are analyzed by using iris experiment. Readers can obtain the preprocessed image datasets here 5 .   Table 5 shows the influence of edge points on the acceleration algorithm. In terms of the maximum distance, the change of edge points changes the maximum distance, while in terms of clustering purity and iteration speed, the 5 https://github.com/ChoiNgai/paper_DynamicMetricClustering/tree/m ain/data/edge points/ change of edge points do not affect the accuracy and speed of the algorithm.

Viii. EXpEriMEnt
This section investigates the acceleration effect and performance of the dynamic-metric acceleration method applied to different algorithms in real datasets to verify its effectiveness.

1) EXpEriMEntaL sEtUp
The experiment was conducted on a series of real datasets. These datasets include data with clusters that are not linearly separable, with noise, with numerous features and with partial redundancy [58]. In addition, image data classification [59] is also included. The statistical data of all datasets are summarized in Table 2. The image datasets have been preprocessed in the experiment, and readers can obtain the preprocessed image datasets here 6 .
To evaluate the effectiveness of the methods presented in this paper, the dynamic metric method is used to improve the fuzzy clustering algorithm based on Euclidean distance, Mahalanobis distance and entropy. In the experiment, we analyze the effect of the dynamic metric method by comparing the performance before and after the improvement.

2) aLgOrithM iMprOVEMEnt
We choose the fuzzy clustering algorithm based on the Euclidean distance, Mahalanobis distance and entropy to verify the acceleration effect of the dynamic metric method, and we choose FCM as the unsupervised fuzzy clustering method based on the Euclidean distance, SFCM as the semisupervised fuzzy clustering method based on the Euclidean distance, SMUC as the unsupervised fuzzy clustering method based on the Mahalanobis distance, and ESFCM as the semi-supervised clustering algorithm based on the Mahalanobis distance and entropy. We introduce the dynamic metric acceleration method into the FCM, KFCM, SFCM, eSFCM and SMUC algorithms to replace the static metric of the original algorithm. The algorithms that  The membership matrix and the iterative formula of the clustering center of FCM are as follows (three iterative formulas):   (2); The membership matrix and the iterative formula of the clustering center of KFCM are as follows (three iterative formulas): ) ,..., ) ,..., ) ,...,  (18);

3) EVaLUatiOn inDEX
Experiments compare the clustering accuracy and running time before and after the improvement of the algorithm. The internal evaluation index data reconstruction error rate, the external evaluation index purity, and the number of iterations are used to evaluate the optimization effect of the iterative process.
In the following experimental Figures 11, 12 and 13, the dynamic metric acceleration method is represented by the L-∞ metric speedup. Moreover, we make 5   , 5 10   e , the maximum number of iterations is 100, λ=10, and randomly initialize cluster centers in experiments.

a. EXtErnaL EVaLUatiOn inDEX:
Purity reflects the similarity between clustering results and reality. We propose an internal evaluation index iter C Purity / ) , ( to evaluate the improvement level of the clustering accuracy of the algorithm: b. intErnaL EVaLUatiOn inDEX: V RE analyzes the difference between reconstructed data and the original data after clustering. Equation (55) reconstructs data by using the membership matrix and clustering center obtained by clustering. The reconstructed data should be as similar to the original data as possible. A small VRE value means that the algorithm created a well-clustered effect.
Fewer iterations mean a faster search speed. We found that the difference between the results of the two indices was small and stable over the corresponding intervals in the experiment. Therefore, we propose an internal evaluation index to VRE * iter to evaluate the improvement level of the iterative performance of the algorithm: (55) where t is the number of iterations of the algorithm. From the experimental results in Figure 13, except for the KFCM algorithm, the algorithms have better effects after using the metric accelerated iteration method, which shows that this method is not suitable for the fuzzy clustering algorithm of the kernel function. From the experimental results in Figure 14, the metric acceleration method still has a poor effect on the KFCM algorithm. Some datasets will not be improved on eSFCM and SMUC, and the effect is the same as that of the unimproved method, indicating that this method still has room for improvement in the nonlinear metrics. From the experimental results in Figure 15, the metric acceleration method performs well in algorithms other than KFCM.From the comparison of SMUC algorithm with SFCM and eSFCM, it can be seen that our acceleration method is weaker than the algorithm based on Mahalanobis distance in the optimization effect of the algorithm based on Euclidean distance. The experiments of FCM and SFCM show that the optimization effect of unsupervised algorithm is lower than that of semi supervised algorithm.

EXpEriMEntaL rEsULts anaLysis
From the above experimental results, it can be seen that the metric acceleration method in this paper achieves highquality results in the Euclidean distance and Mahalanobis distance metrics. However, for some nonlinear metric methods, the effect is not improved (such as KFCM and metrics with the kernel method), which indicates that this method is not suitable for nonlinear metrics, such as the kernel function. In the experiment, while combining the external evaluation index and iteration speed, the acceleration effect of dynamic metric is not substantial when dealing with linearly inseparable data. The optimization effect of our acceleration method based on Euclidean distance is weaker than that based on Markov distance, and the optimization effect of unsupervised algorithm is lower than that of semi supervised algorithm.

iX. cOncLUsiOns
While focusing on the problem of unstandardized distance in clustering algorithms, we develop a fuzzy clustering acceleration method based on dynamic metrics. This method standardizes the distance between instances to improve the search efficiency of the iteration process. We prove its stability in metric normalization, and verify that its optimization performance is not affected by the change of edge points in clusters. The method is applied to five common fuzzy clustering algorithms, and the experimental results on eight real datasets show that the method has an effective acceleration, while maintaining the stability of clustering accuracy. Furthermore, the algorithm uses the Chebyshev distance to standardize between features, which keeps the time complexity of the algorithm constant during metric iterations. In addition, the experimental results show that the dynamic metric acceleration method is less effective in the algorithm with the Gauss kernel function.