On: Microsoft’s New MT-DNN Outperforms Google BERT

[riku_iki]

Yeah, but in case of ELMo it was fine-tuning (training of pretrained language model and task model together), not just transfer learning.

[phowon → riku_iki]

With ELMo, the pretrained weights are frozen. Only the scalars for ELMo layers are tuned (as well as the additional top-level model, of course).