Masked language model
A model trained to fill in hidden words in a sentence using the words on both sides, as BERT was.
Also: MLM, masked language modelling
In masked language modelling, some tokens of a sentence, 15 per cent in BERT's case, are hidden or swapped, and the model must guess the originals from the surrounding text. Because it can look both left and right, the model learns rich two-way representations, which is ideal for understanding tasks such as search and classification. The trade-off is that masked models are not natural text generators. The approach powered BERT, RoBERTa and many later encoders and embedding models.