Culture Magazine

Scrambled Sentences Work for Pretraining Large Language Models

By Bbenzon @bbenzon

"[...] we pre-train MLMs on sentences with randomly shuffled word order, and show that these models still achieve high accuracy after fine-tuning on many downstream tasks -- including on tasks specifically designed to be challenging for models that ignore word order."

- David McClure (@clured) April 15, 2021

Back to Featured Articles on Logo Paperblog