9.8. Evaluation Metrics for NLP

• Accuracy – For classification.

Accuracy is a straightforward metric that measures how often the model makes correct predictions. It is calculated as the number of correct predictions divided by the total number of predictions made. In the context of classification tasks, this means that if a model classifies items into categories, accuracy shows the proportion of items correctly categorized overall.

Imagine a teacher grading a class of 100 students on a test. If 90 students pass the test, the accuracy of the students passing is 90%. Similarly, if a model correctly identifies the sentiment of 90 out of 100 movie reviews as positive or negative, its accuracy is also 90%.

• Precision, Recall, F1-score – For imbalanced classes.

Precision, Recall, and F1-score are crucial when dealing with imbalanced datasets where one class is much more prevalent than another.

• Precision measures the accuracy of positive predictions; it shows how many of the predicted positives were actually positive.
• Recall measures how many actual positives were correctly identified by the model.
• F1-score is the harmonic mean of precision and recall, providing a balance between the two. It’s particularly useful when you want to take both false positives and false negatives into account, giving a single score for model performance.

Consider a scenario in a medical test for a disease that only affects 1% of the population. If the test identifies 100 people as having the disease, but only 10 actually do, the precision would be low. Meanwhile, if it fails to identify 5 out of 10 actual cases, the recall would also suffer. The F1-score helps in understanding the model's overall effectiveness in identifying true positive cases despite the imbalance.

• BLEU – For machine translation.

BLEU (Bilingual Evaluation Understudy) is an algorithm for evaluating the quality of text that has been translated from one language to another. It compares a machine-generated translation with one or more high-quality reference translations, measuring how many words and phrases are matched. The score is usually between 0 and 1, with higher scores indicating better translation quality.

Think of BLEU like a spelling and grammar checker for translated text. If a student translates a sentence from English to Spanish, BLEU checks how many words match the way a native speaker would translate it. If the student translates 'I love ice cream' correctly to 'Me encanta el helado', the BLEU score would be high; however, if it were translated poorly, BLEU would indicate a lower score.

• ROUGE – For summarization.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of metrics used to evaluate automatic summarization and machine translation. It measures the overlap of n-grams (a contiguous sequence of n items from a given sample of text) between the generated summary and reference summaries. Key metrics include ROUGE-N (precision and recall of n-grams), ROUGE-L (longest common subsequence), among others.

Imagine you wrote a summary of a book, and a teacher has a standard summary. ROUGE would help determine how many key phrases or ideas from the teacher's summary are present in your version. If you capture most of the essential points, your ROUGE score is high, indicating a good summary.

• Perplexity – For language modeling.

Perplexity is primarily used in the context of language models, where it measures how well a probability distribution predicts a sample. A lower perplexity indicates that the model predicts the test data better, reflecting its performance in terms of predicting text sequences. It can be interpreted as the model's uncertainty in predicting the next word in a sentence.

If you were asked to predict the next word in a story and you were confused by what you've read, you'd have high perplexity. However, if the context is clear and you can easily guess the next words, your perplexity is low. In language modeling, a model with low perplexity can predict the next word in a sentence more reliably, similar to how someone familiar with a story would know what to expect next.

--