8.4. Performance Metrics in AI

8.4.1 Accuracy

• Measures the percentage of correct predictions.
• Formula: $\text{Accuracy} = \frac{\text{Correct Predictions}}{\text{Total Predictions}} \times 100$

Example: If out of 100 test images, 85 were classified correctly: $\text{Accuracy} = \frac{85}{100} = 85\%$

Accuracy is a basic metric used to evaluate how well an AI model makes predictions. It calculates the ratio of correct predictions made by the model to the total number of predictions it made. If every prediction were correct, the accuracy would be 100%. A model with an accuracy of 85% means it correctly predicted 85 out of 100 test cases, showing that while it performs well, there is still some room for improvement.

Think of accuracy like a student taking a quiz. If the student answers 85 out of 100 questions correctly, they receive an 85% score on that quiz. This score gives a clear indication of the student's performance, similar to how accuracy reflects the performance of an AI model.

8.4.2 Precision

• Measures how many of the predicted positives are actually correct. $\text{Precision} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}}$

Precision is a key metric that focuses specifically on the positive predictions made by the model. It answers the question: 'Of all the instances that the model predicted as positive, how many were truly positive?' For example, in spam detection, if the model predicts that 10 emails are spam but only 7 are really spam, the precision would reflect that, indicating the reliability of the model when it predicts a positive outcome.

Imagine you're a referee in a soccer game, and you have to call fouls. If you call 10 fouls but only 7 were actually fouls, your precision rate is about 70%. Just like the referee's calls need to be accurate, an AI model needs high precision to be trusted in its positive predictions.

8.4.3 Recall (Sensitivity)

• Measures how many actual positives the model correctly predicted. $\text{Recall} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}}$

Recall gives insight into the model's ability to capture all positive instances within the dataset. It answers the question: 'Of all the actual positives, how many did the model correctly identify?' A high recall means the model successfully identifies most of the relevant data points. For instance, in medical diagnoses, high recall is crucial to ensure that most patients with a condition are correctly identified.

Consider a wildlife protector searching for endangered species in a forest. If there are 100 endangered animals and the protector finds 90 of them, the recall is 90%. This high recall is important because missing even a few can significantly affect the species' survival, just as recall is vital in models aimed at identifying critical conditions.

8.4.4 F1 Score

• Harmonic mean of precision and recall. $F1 = \frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$ F1 score is useful when there is class imbalance.

The F1 Score is a measure that combines both precision and recall into a single metric. It is particularly useful when dealing with imbalanced datasets, where one class is more significant than the other. Instead of focusing only on one aspect, the F1 Score provides a more balanced perspective on how well the model performs in terms of both identifying true positives and reducing false positives. A higher F1 Score indicates a better balance between precision and recall.

Imagine a student who excels in math but struggles with writing. If the student only focuses on math (like precision) and neglects writing (like recall), their overall performance might suffer. The F1 Score acts like a report card that combines both subjects, giving a more holistic view of the student’s abilities, just as it provides a comprehensive view of an AI model's effectiveness.

--