Comparing recurrent neural networks with similar models

In recent years, RNNs, similarly to any neural network model, have become widely popular due to the easier access to large amounts of structured data and increases in computational power. But researchers have been solving sequence-based problems for decades with the help of other methods, such as the Hidden Markov Model. We will briefly compare this technique to an RNNs and outline the benefits of both approaches.

The Hidden Markov Model (HMM) is a probabilistic sequence model that aims to assign a label (class) to each element in a sequence. HMM computes the probability for each possible sequence and picks the most likely one.

Both the HMM and RNN are powerful models that yield phenomenal results but, depending on the use case and resources available, RNN can be much more effective.

Hidden Markov model

The following are the pros and cons of a Hidden Markov Model when solving sequence-related tasks:

• Pros: Less complex to implement, works faster and as efficiently as RNNs on problems of medium difficulty.
• Cons: HMM becomes exponentially expensive with the desire to increase accuracy. For example, predicting the next word in a sentence may depend on a word from far behind. HMM needs to perform some costly operations to obtain this information. That is the reason why this model is not ideal for complex tasks that require large amounts of data.

These costly operations include calculating the probability for each possible element with respect to all the previous elements in the sequence.

Recurrent neural network

The following are the pros and cons of a recurrent neural network when solving sequence-related tasks:

• Pros: Performs significantly better and is less expensive when working on complex tasks with large amounts of data.
• Cons: Complex to build the right architecture suitable for a specific problem. Does not yield better results if the prepared data is relatively small.

As a result of our observations, we can state that RNNs are slowly replacing HMMs in the majority of real-life applications. One ought to be aware of both models, but with the right architecture and data, RNNs often end up being the better choice.

Nevertheless, if you are interested in learning more about hidden Markov models, I strongly recommend going through some video series (https://www.youtube.com/watch?v=TPRoLreU9lA) and papers of example applications, such as Introduction to Hidden Markov Models by Degirmenci (Harvard University) (https://scholar.harvard.edu/files/adegirmenci/files/hmm_adegirmenci_2014.pdf) or Issues and Limitations of HMM in Speech Processing: A Survey (https://pdfs.semanticscholar.org/8463/dfee2b46fa813069029149e8e80cec95659f.pdf).