11.4.2. Text Preprocessing

Text Preprocessing • Cleaning and preparing raw data using:

Text preprocessing is the crucial step that occurs after text acquisition in the NLP pipeline. It involves cleaning and organizing the raw text data to make it suitable for analysis and model training. This stage ensures that the data is devoid of unwanted elements that could affect the results of further NLP processes.

Imagine you are preparing vegetables for a salad. Before making the salad, you need to wash the vegetables, cut them into appropriate sizes, and remove any spoiled parts. Similarly, text preprocessing prepares raw text data by 'cleaning' it and making it fit for the next steps in analyzing and understanding it.

o Tokenization: Splitting sentences into words.

Tokenization is the process of dividing a sentence into smaller units called tokens, which can be words, phrases, or even characters. For example, the sentence 'I love pizza' can be tokenized into three tokens: 'I', 'love', and 'pizza'. This step is essential because it allows the algorithm to analyze words individually.

Think of tokenization like breaking down a jigsaw puzzle. Each piece of the puzzle represents a word in the sentence. Just as you cannot complete the puzzle without understanding each piece, algorithms need tokens to fully comprehend the structure of the text.

o Stopword Removal: Removing common words like 'the', 'is'.

Stopword removal involves eliminating commonly used words from the text that do not add significant meaning. Words like 'the', 'is', 'and', etc., are usually considered stopwords. Removing these helps in reducing the dimensionality of the data and focuses the analysis on the more meaningful words.

Consider reading a book filled with filler words that do not add value to the story, such as excessively repeating phrases. When summarizing the book, you would skip those phrases and focus on the key plot points. Similarly, stopword removal streamlines the analysis by focusing only on relevant content.

o Stemming: Reducing words to their root form (e.g., running → run).

Stemming is the process of reducing words to their base or root form. For example, the words 'running', 'runner', and 'ran' can all be reduced to 'run'. This helps in grouping different variations of a word, simplifying the analysis by treating them as the same term.

Imagine a family tree where you have different generations of the same family. Instead of talking about each family member individually, you refer to them collectively as the 'Smith family'. Stemming simplifies language in a similar way, clustered under one root form.

o Lemmatization: Converting words to base form (better than stemming).

Lemmatization is similar to stemming but more sophisticated. Instead of just removing prefixes and suffixes, it transforms words to their root base form based on their meaning. For instance, 'better' becomes 'good' and 'running' becomes 'run'. This creates a more accurate representation of the meaning behind the words.

Think of lemmatization as a comprehensive curriculum where students learn the fundamentals of subjects rather than just learning isolated facts. By understanding the core concepts (base forms), they can better apply knowledge to various contexts.

--