Modeling and evaluation

Modeling will be broken into two distinct parts. The first will focus on word frequency and correlation and culminate in the building of a topic model. In the next portion, we will examine many different quantitative techniques by utilizing the power of the qdap package in order to compare two different speeches.

Word frequency and topic models

As we have everything set up in the document-term matrix, we can move on to exploring word frequencies by creating an object with the column sums, sorted in descending order. It is necessary to use as.matrix() in the code to sum the columns. The default order is ascending, so putting - in front of freq will change it to descending:

> freq <- colSums(as.matrix(dtm))

> ord <- order(-freq)

We will examine the head and tail of the object with the following code:

> freq[head(ord)]
new  america  people   jobs    now  years 
        193      174     168    163    157    148

> freq[tail(ord)]
wright written yearold youngest...