Coding – Bag-of-N-grams

Any consecutive N words become an N-gram. Not all N-grams are meaningful. For example, there are uncommon phrases in this list: “Let us,” “us join,” “join the,” “the New,” “New York,” “York City,” “City marathon,” “marathon in,” and “in 2022.” The novelty of Gensim is that it tries to create common phrases for N-grams. To make this use case interesting, in the following code, I have added two more sentences to make the lyrics longer. It has the 3-gram phrase “come and visit”:

doc_list_Ngrams = ["Start spreading the news",
"You're leaving today",
"I want to be a part of it, New York, New York",
"Your vagabond shoes, they are longing to stray",
"And steps around the heart of it, New York, New York",
"Come and visit us",
"Come and visit the city...