Data cleaning is one of the most important and time-consuming tasks when it comes to natural language processing (NLP):
"There's the joke that 80 percent of data science is cleaning the data and 20 percent is complaining about cleaning the data."
– Kaggle founder and CEO Anthony Goldbloom in a Verge Interview
In this chapter, we will discuss some of the most common text pre-processing ideas. This task is universal, tedious, and unavoidable. Most people working in data science or NLP understand that it's an underrated value addition. Some of these tasks don't work well in isolation but have a powerful effect when used in the right combination and order. This chapter will introduce several new words and tools, since the field has a rich history from two worlds. It borrows from both traditional NLP and machine learning. We&apos...
