Our framework learns to dynamically balance local losses to optimize a global loss via meta-learning. A local loss block attached to a hidden layer captures local error s...
Abstract:
The standard training for deep neural networks relies on a global and fixed loss function. For more effective training, dynamic loss functions have been recently proposed...Show MoreMetadata
Abstract:
The standard training for deep neural networks relies on a global and fixed loss function. For more effective training, dynamic loss functions have been recently proposed. However, the dynamic global loss function is not flexible to differentially train layers in complex deep neural networks. In this paper, we propose a general framework that learns to adaptively train each layer of deep neural networks via meta-learning. Our framework leverages the local error signals from layers and identifies which layer needs to be trained more at every iteration. Also, the proposed method improves the local loss function with our minibatch-wise dropout and cross-validation loop to alleviate meta-overfitting. The experiments show that our method achieved competitive performance compared to state-of-the-art methods on popular benchmark datasets for image classification: CIFAR-10 and CIFAR-100. Surprisingly, our method enables training deep neural networks without skip-connections using dynamically weighted local loss functions.
Our framework learns to dynamically balance local losses to optimize a global loss via meta-learning. A local loss block attached to a hidden layer captures local error s...
Published in: IEEE Access ( Volume: 9)
Funding Agency:
Keywords assist with retrieval of results and provide a means to discovering other relevant content. Learn more.
- IEEE Keywords
- Index Terms
- Local Loss ,
- Loss Function ,
- Neural Network ,
- Deep Neural Network ,
- Localization Signal ,
- Local Function ,
- Neural Network Layers ,
- Cross-validation Loop ,
- Dynamic Weight ,
- Weight Loss ,
- Training Dataset ,
- Convolutional Layers ,
- Classification Task ,
- Hidden Layer ,
- Cross-entropy Loss ,
- Training Strategy ,
- Intermediate Layer ,
- Class Position ,
- Focal Loss ,
- Prediction Loss ,
- Exponential Smoothing ,
- Loop Algorithm ,
- Softmax Loss ,
- Similarity Matching ,
- Main Paper ,
- Average Gap ,
- Robust Representation ,
- Bilevel Optimization ,
- CIFAR-100 Dataset ,
- Max Iteration
- Author Keywords
Keywords assist with retrieval of results and provide a means to discovering other relevant content. Learn more.
- IEEE Keywords
- Index Terms
- Local Loss ,
- Loss Function ,
- Neural Network ,
- Deep Neural Network ,
- Localization Signal ,
- Local Function ,
- Neural Network Layers ,
- Cross-validation Loop ,
- Dynamic Weight ,
- Weight Loss ,
- Training Dataset ,
- Convolutional Layers ,
- Classification Task ,
- Hidden Layer ,
- Cross-entropy Loss ,
- Training Strategy ,
- Intermediate Layer ,
- Class Position ,
- Focal Loss ,
- Prediction Loss ,
- Exponential Smoothing ,
- Loop Algorithm ,
- Softmax Loss ,
- Similarity Matching ,
- Main Paper ,
- Average Gap ,
- Robust Representation ,
- Bilevel Optimization ,
- CIFAR-100 Dataset ,
- Max Iteration
- Author Keywords