DEEP-LEARNING FOR INDUSTRIAL DATA ANALYTICS

Abstract

Deep-learning based models are prevalent in all areas of Natural Lan- guage Processing (NLP). Advent of attention based deep bi-directional architectures (Transformers) has given rise to powerful language mod- els (e.g. BERT, RoBERTa and XLNet). These pre-trained models utilize large unsupervised corpora along with custom language mod- elling objectives. For instance BERT (Bi-directional Encoder Repre- sentations from Transformers) is a pre-trained transformer encoder trained on Wikipedia (2.5B words) and Book corpus (800M words). RoBERTa and XLNet are other language models with different training corpus and language modelling objective than BERT. Language gener- ation objectives in NLP use transfer learning from pre-trained models. Uni-directional and bi-directional modelling objectives are used for pre-training the language models. Bi-directional language modelling objective is more popular and involves permuting or masking random text from datasets. Unsupervised corpora spanning entire languages, form the datasets for pre-training. Resulting models are generic and can be used for different language understanding objectives with mini- mal number of task specific parameters. Pre-training language models is computationally expensive.