Scrambled Sentences Work for Pretraining Large Language Models

"[...] we pre-train MLMs on sentences with randomly shuffled word order, and show that these models still achieve high accuracy after fine-tuning on many downstream tasks -- including on tasks specifically designed to be challenging for models that ignore word order." https://t.co/tNw7Mghsa7

- David McClure (@clured) April 15, 2021