Text Summarization with Seq2seq Attention and Transformer Networks

Summarizing a piece of text challenges a deep learning model's understanding of language. Summarization can be considered a uniquely human ability, where the gist of a piece of text needs to be understood and phrased. In the previous chapters, we have built components that can help in summarization. First, we used BERT to encode text and perform sentiment analysis. Then, we used a decoder architecture with GPT-2 to generate text. Putting the Encoder and Decoder together yields a summarization model. In this chapter, we will implement a seq2seq Encoder-Decoder with Bahdanau Attention. Specifically, we will cover the following topics:

• Overview of extractive and abstractive text summarization
• Building a seq2seq model with attention to summarize text
• Improving summarization with beam search
• Addressing beam search issues with length normalizations
• Measuring the performance of summarization...