Deep Neural Network Pretrained by a Support Vector Machine

Recently, deep neural networks (DNNs) have shown strong performance in many applications. Some models achieve state of the art in a wide range of fields, such as natural language processing and image processing. Support vector machines (SVMs) have also been a popular approach thanks to their performance. Their criterion selects effective variables in a dataset, and kernel methods help the models extract useful features for prediction. In this paper, we propose a unique method to apply parameters of a pretrained SVM to a DNN. Our proposed method initializes the first layer of a DNN to behave as the pretrained SVM or to extract powerful features from input variables. As a result, the DNN is successfully tuned during the training, because it inherits the advantages of SVMs. To show performance of our proposed method in the experiments, we apply it to classification of a toy dataset and to a sentiment analysis of movie reviews. The results show that the pretrained parameters have a significant effect on the optimization of DNNs.