8. Model Evaluation Metrics

Training a machine learning model is only half the job — the other half is checking how reliable it is. Just knowing whether the model is 90% accurate is not enough — it could still be very poor in real-world performance (especially when one class dominates the dataset). So, we need multiple metrics to get a full picture.

Evaluating a model's performance is crucial because accuracy alone can be misleading. For instance, a model could achieve high accuracy by predicting a predominant class all the time, while completely failing to identify instances of the minority class. Hence, multiple metrics are necessary to assess a model comprehensively.

Think of a teacher who gives grades based solely on class participation rather than understanding of the material. A student might excel in participation (high accuracy) but still fail the tests (poor performance). Only looking at one aspect doesn't give a complete view of their abilities.

A confusion matrix shows the number of:

• True Positives (TP): Correctly predicted positive cases
• True Negatives (TN): Correctly predicted negative cases
• False Positives (FP): Incorrectly predicted as positive
• False Negatives (FN): Incorrectly predicted as negative

The confusion matrix is a table that helps visualize the performance of a classification model. By showing the counts of true positive, true negative, false positive, and false negative predictions, it allows us to see not just how many predictions were correct, but also the types of errors being made.

Imagine a doctor diagnosing a disease. The confusion matrix helps the doctor understand how many patients were correctly diagnosed (true positives and negatives) versus misdiagnosed (false positives and negatives), which is crucial for improving diagnostic accuracy.

Accuracy is the ratio of correctly predicted observations to the total observations. Accuracy = (TP + TN) / (TP + TN + FP + FN)

⚠ Warning: Accuracy can be misleading in imbalanced datasets.

Accuracy gives a basic measure of performance by calculating the proportion of correct predictions. However, in situations where the classes are imbalanced (e.g., 95% pass and 5% fail), a model could achieve high accuracy simply by predicting the majority class, which doesn't reflect its ability to correctly identify the minority class.

If an employer knows that only 5 people out of 100 in job applications are suitable and hires all 95 unsuitable candidates, they may achieve a high 'accuracy' in their selections, but they will ultimately employ the wrong people—a problem for the organization in practice.

Precision is the percentage of correct positive predictions. Precision = TP / (TP + FP) It answers: 'Of all predicted positives, how many were truly positive?'

Precision helps assess the accuracy of positive predictions by focusing specifically on how many of the instances classified as positive are actually positive. This is particularly useful in fields where false positives can have significant consequences.

Consider a spam filter for emails. If it classifies 10 emails as spam and only 6 are actually spam, the precision would be 60%. This means the filter produces a lot of false positives, which could result in important emails being missed.

Recall is the percentage of actual positives that were correctly predicted. Recall = TP / (TP + FN) It answers: 'Of all actual positives, how many did we detect?'

Recall measures a model's ability to find all the positive instances. It indicates how many of the actual positive cases were correctly identified, which is crucial in applications like medical diagnosis, where failing to identify positives can have severe outcomes.

Imagine a security system designed to catch intruders. If it detects only 7 out of 10 true intrusions, its recall is 70%. This means that 30% of intrusions go unnoticed, which could lead to safety issues.

F1 Score is the harmonic mean of precision and recall. F1 = 2 * (Precision * Recall) / (Precision + Recall) Useful when you want to balance both false positives and false negatives.

The F1 Score provides a single metric that balances both precision and recall, making it especially useful when the two metrics offer contradictory information. It helps ensure that when precision is high, the recall is also reasonably high, indicating balanced performance.

In a medical test scenario, if a test has high precision but low recall, many sick patients may be overlooked. The F1 Score helps to capture the trade-off between ensuring fewer wrongly identified healthy individuals and capturing as many sick individuals as possible.

📘 ROC Curve: ROC = Receiver Operating Characteristic

• Plots True Positive Rate vs False Positive Rate
• Helps visualize the performance across all thresholds

📘 AUC = Area Under the Curve:

• Measures the entire two-dimensional area underneath the ROC curve
• The higher the AUC, the better the model

The ROC curve is a graphical representation that showcases the performance of a binary classifier by plotting true positive rates against false positive rates at various thresholds. The AUC quantifies the overall effectiveness of the model; a score close to 1 indicates a strong model, while a score near 0.5 suggests a poor model.

Imagine a quality control system in a factory where the goal is to catch defective products. The ROC curve helps visualize how well different settings for defect detection perform, enabling adjustments to maximize detection without significantly increasing false positives.

--