We shall use the popular PTB and text8 datasets for our demonstrations.
The Penn Treebank (PTB) dataset is a by-product of Penn Treebank project carried out at UPenn (https://catalog.ldc.upenn.edu/ldc99t42). The PTB project team extracted about one million words from the three years of Wall Street Journal stories and annotated them in Treebank II style. The PTB dataset comes in two flavors: Basic Examples, that are about 35 MB in size, and Advanced Examples, that are about 235 MB in size. We shall use the simple dataset that consists of 929K words for training, 73K words for validation, and 82K words for testing. You are encouraged to explore the advanced dataset. Further details on the PTB dataset are available at the following link: http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz.
