Relearning text preprocessing in the era of LLMs

Text preprocessing involves the application of various techniques to raw textual data with the aim of cleaning, organizing, and transforming it into a format suitable for analysis or modeling. The primary goal is to enhance the quality of the data by addressing common challenges associated with unstructured text. This entails tasks such as cleaning irrelevant characters, handling variations, and preparing the data for downstream NLP tasks.

With the rapid advancements in LLMs, the landscape of NLP has evolved significantly. However, fundamental preprocessing techniques such as text cleaning and tokenization remain crucial, albeit with some shifts in approach and importance.

Staring with text cleaning, while LLMs have shown remarkable robustness to noise in input text, clean data still yields better results and is especially important for fine-tuning tasks. Basic cleaning techniques such as removing HTML tags, handling special characters...