Adaptation: Blessing or Curse for Higher Way Meta-Learning

The prevailing literature typically assesses the effectiveness of meta-learning (ML) approaches on tasks that involve no more than 20 classes. However, we challenge this convention by conducting a more complex and natural task setup to test the fundamental initialization, metric, and optimization approaches. In particular, we increase the number of classes in the Omniglot and tieredImagenet datasets to 200 and 90, respectively. Interestingly, we observe that as the number of classes increases, ML approaches perform in reverse order of their degree of adaptation, with prototypical network (ProtoNet) outperforming almost no inner loop (ANIL) and model-agnostic meta-learning (MAML). ProtoNet, which does not require adaptation, is marginally affected by the increase in task complexity, while ANIL and MAML are highly affected. Despite performing full feature backbone and classifier adaptation, Meta Long Short-term Memory (MetaLSTM++) exhibits an intriguing behavior of performing well. To this end, we analyze the backbone learned by different algorithms and the influence of adaptation from different perspectives. We uncover that ProtoNet learns better, and MetaLSTM++ learns the worst backbone, but the generalizability to unseen data for MetaLSTM++ is compensated by powerful adaptation to the meta-test support sets, due to its learned data-driven optimizer.