30.4. Example with Real Data

Suppose we test an AI model on 100 emails: • 60 are spam (positive class) • 40 are not spam (negative class)

In this example, we are evaluating an AI model that classifies emails into two categories: spam and not spam. We have a total of 100 emails, out of which 60 are identified as spam (the positive class), and 40 are identified as not spam (the negative class). This establishes the context of our data and what we are trying to predict.

Imagine you have a personal email account. Out of every 100 emails you receive, you notice that 60 of them are promotional offers or junk mail (spam), while the other 40 are important messages from friends or work (not spam). Understanding this distribution helps us see how the AI model's performance will be measured.

Model prediction results: • TP = 50 • FP = 5 • FN = 10 • TN = 35

The model made several predictions that we can classify into four categories: True Positive (TP), False Positive (FP), False Negative (FN), and True Negative (TN). Here, the model correctly identified 50 spam emails as spam (TP), mistakenly classified 5 legitimate emails as spam (FP), failed to recognize 10 spam emails (FN), and correctly identified 35 legitimate emails as not spam (TN). These values will be used to construct our confusion matrix.

Think of it like a guest list for a party. You invited 60 friends (spam) and 40 acquaintances (not spam). Out of your total guests, 50 friends showed up (TP), while 5 acquaintances crashed the party as if they were friends (FP). Additionally, you missed 10 friends who tried to join but were turned away (FN), and 35 acquaintances were appropriately recognized and not allowed in (TN). This helps illustrate how well the model performs in distinguishing between two classes.

Let’s form the confusion matrix: Predicted Spam Predicted Not Spam Actual Spam 50 (TP) 10 (FN) Actual Not Spam 5 (FP) 35 (TN)

From the prediction results, we can create a confusion matrix, which visually represents how many predictions were accurate and inaccurate. The matrix is structured with the actual class on one axis and the predicted class on another. Each cell indicates the counts for each combination of actual and predicted classes: 50 true positives, 10 false negatives, 5 false positives, and 35 true negatives.

Imagine you organize the guest list on a chart. Each row represents those who actually attended your party (Actual Spam vs. Actual Not Spam), while each column reflects who you thought would show up (Predicted Spam vs. Predicted Not Spam). The way you fill in this chart helps you understand where you got it right or wrong, just like assessing the model's predictions against actual outcomes.

Now compute the metrics: • Accuracy = (50 + 35) / 100 = 85% • Precision = 50 / (50 + 5) = 90.9% • Recall = 50 / (50 + 10) = 83.3% • F1 Score = 2 × (0.909 × 0.833) / (0.909 + 0.833) ≈ 87%

We can derive important performance metrics directly from the confusion matrix. Accuracy measures the overall correctness of the model's predictions, which is 85%. Precision quantifies the quality of positive predictions, revealing that 90.9% of predicted spam emails were actual spam. Recall assesses the model's ability to identify all relevant instances, showing that 83.3% of all actual spam were correctly identified. The F1 Score, which balances precision and recall, is approximately 87%, indicating reasonable accuracy in situations with varying importance of these metrics.

Continuing with the party analogy, accuracy is like saying how many of the people who arrived were actually on the guest list. Precision is scrutinizing those who said they were friends and evaluating how many really were friends; a high precision indicates most were friends. Recall focuses on ensuring all friends were invited and not missed, while F1 Score serves as a combined metric, similar to asking if your guest list represented a well-balanced mix of all your friends and acquaintances.

--