The Improvement of the Algorithm EG for the Relative Entropy Loss

A long standing idea in online learning is Exponential Gradient (EG) update protocol. Its online weight updating rules is derived from incorporating components of the relative entropy distance and relevant entropy loss as the objective function. In this manuscript, we study the behaviors of the EG algorithm with relative entropy loss in the scene of conjugate of conjugate. More specifically, we propose that a principled solution to circumventing approximation step for the traditional EG algorithm can be found by conjugate of conjugate strategy. Moreover, an upper bound for the error loss is presented to demonstrate the effectiveness of our proposed update strategies. Meanwhile, we give three different protocols for the choice of the parameter and compare the results on noisy data. In addition, we validate the effectiveness of our devised exponential gradient update rules in the classification and regression scenarios on real-world data sets.