Regression-Based Hyperparameter Learning for Support Vector Machines | IEEE Journals & Magazine | IEEE Xplore

Regression-Based Hyperparameter Learning for Support Vector Machines


Abstract:

Unification of classification and regression is a major challenge in machine learning and has attracted increasing attentions from researchers. In this article, we presen...Show More

Abstract:

Unification of classification and regression is a major challenge in machine learning and has attracted increasing attentions from researchers. In this article, we present a new idea for this challenge, where we convert the classification problem into a regression problem, and then use the methods in regression to solve the problem in classification. To this end, we leverage the widely used maximum margin classification algorithm and its typical representative, support vector machine (SVM). More specifically, we convert SVM into a piecewise linear regression task and propose a regression-based SVM (RBSVM) hyperparameter learning algorithm, where regression methods are used to solve several key problems in classification, such as learning of hyperparameters, calculation of prediction probabilities, and measurement of model uncertainty. To analyze the uncertainty of the model, we propose a new concept of model entropy, where the leave-one-out prediction probability of each sample is converted into entropy, and then used to quantify the uncertainty of the model. The model entropy is different from the classification margin, in the sense that it considers the distribution of all samples, not just the support vectors. Therefore, it can assess the uncertainty of the model more accurately than the classification margin. In the case of the same classification margin, the farther the sample distribution is from the classification hyperplane, the lower the model entropy. Experiments show that our algorithm (RBSVM) provides higher prediction accuracy and lower model uncertainty, when compared with state-of-the-art algorithms, such as Bayesian hyperparameter search and gradient-based hyperparameter learning algorithms.
Published in: IEEE Transactions on Neural Networks and Learning Systems ( Volume: 35, Issue: 12, December 2024)
Page(s): 18799 - 18813
Date of Publication: 17 October 2023

ISSN Information:

PubMed ID: 37847631

Funding Agency:


Contact IEEE to Subscribe

References

References is not available for this document.