How to Develop a Word-Level Neural Language Model and Use it to Generate Text
Last Updated on September 3, 2020
A language model can predict the probability of the next word in the sequence, based on the words already observed in the sequence.
Neural network models are a preferred method for developing statistical language models because they can use a distributed representation where different words with similar meanings have similar representation and because they can use a large context of recently observed words when making predictions.
In this tutorial, you will discover how to develop a statistical language model using deep learning in Python.
After completing this tutorial, you will know:
- How to prepare text for developing a word-based language model.
- How to design and fit a neural language model with a learned embedding and an LSTM hidden layer.
- How to use the learned language model to generate new text with similar statistical properties as the source text.
Kick-start your project with my new book Deep Learning for Natural Language Processing, including step-by-step tutorials and the Python source code files for all examples.
Let’s get started.
- Update Apr/2018: Fixed type in model description
- Update May/2020: Fixed a typo in the expectation of the model.