8. Evaluation

Evaluation in AI is the process of testing the trained model to check its accuracy and performance. The goal is to measure how well the AI system performs on unseen data (called the test set). Evaluation helps in: • Validating the effectiveness of the model • Avoiding underfitting and overfitting • Selecting the best-performing model • Fine-tuning for better results

Example: Suppose you trained an AI model to recognize handwritten digits. Evaluation will tell how accurately it identifies new digits it hasn’t seen before.

Evaluation in AI involves assessing how well a trained model performs when it encounters new, unseen data. This process is crucial for ensuring that the model can make accurate predictions in real-world situations and is not just reflecting the data it was trained on. Through evaluation, we can confirm whether the model is functioning effectively, identify potential issues like underfitting (where the model is too simple) or overfitting (where the model is too complex), and determine which version of the model performs best. For instance, if we have an AI model for recognizing handwritten numbers, evaluation will measure its performance and accuracy when it sees new examples of digits.

Imagine an athlete training for a marathon. Just like the athlete needs to run test races to see how well they can perform and identify their strengths and weaknesses, an AI model needs evaluation to ensure it can 'run' well with new data. For example, if the athlete runs a test marathon and finds they can finish in a great time, the evaluation shows their training was effective. Similarly, an evaluation of the AI's digit recognition will show if it can accurately identify numbers it hasn’t seen during training.

AI models can behave differently when exposed to new data. Evaluation helps ensure: • Correctness: Does the model predict accurately? • Robustness: Can it handle real-world inputs? • Generalization: Is it good only on training data or on new data too?

Without evaluation, you risk deploying a faulty or biased model.

Evaluating AI models is vital because these models can perform differently when faced with new data. Evaluation ensures that the model is correct in its predictions, robust enough to deal with various inputs from real-world scenarios, and capable of generalizing its learning to new data rather than just repeating what it learned from the training data. If a model is not evaluated, it can lead to deployment of systems that are unreliable or biased, which can cause significant issues in their application.

Think of a restaurant chef who has perfected their recipes through practice. If they never taste the dish before serving it to customers, they risk presenting a meal that might not meet the standards. The evaluation is akin to that tasting process; it ensures the dish is perfect and ready for the public. An AI model without evaluation is like that untested meal—it might not perform well when it matters the most.

1. Training Set
   • Used to train the model.
   • The model learns patterns from this data.
2. Validation Set
   • Used during training to tune the model parameters.
   • Helps avoid overfitting.
3. Test Set
   • Used after training to evaluate the final performance.
   • Never used during training.

In machine learning, we generally use three types of datasets for evaluation: training set, validation set, and test set. The training set is the data on which the model learns and identifies patterns. The validation set is used during the training process to fine-tune the model's parameters, helping to avoid overfitting by ensuring the model does not learn noise from the training data. Finally, the test set is a completely separate dataset used to evaluate the model's performance after it has been trained and validated. This separation is crucial because it provides an unbiased evaluation of how well the model will perform on real-world, unseen data.

Imagine preparing for a big exam. You have three types of study materials: your main textbook (training set), practice tests (validation set), and a final mock exam that you haven’t seen before (test set). While you study and practice, you're learning from the textbook and refining your knowledge using practice tests, but the final mock exam will show how well you can apply that knowledge under exam conditions.

Here are key metrics used to evaluate AI models: 8.4.1 Accuracy • Measures the percentage of correct predictions. • Formula: Correct Predictions Accuracy = ×100 Total Predictions

Example: If out of 100 test images, 85 were classified correctly: 85 Accuracy = = 85% 100

8.4.2 Precision • Measures how many of the predicted positives are actually correct. True Positives Precision = True Positives + False Positives

8.4.3 Recall (Sensitivity) • Measures how many actual positives the model correctly predicted. True Positives Recall = True Positives + False Negatives

8.4.4 F1 Score • Harmonic mean of precision and recall. 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛×𝑅𝑒𝑐𝑎𝑙𝑙 𝐹1 = 2× 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛+ 𝑅𝑒𝑐𝑎𝑙𝑙 F1 score is useful when there is class imbalance.

Performance metrics are essential for evaluating AI models. The key metrics include accuracy, precision, recall, and the F1 score. Accuracy measures the overall percentage of correct predictions, allowing us to see how well the model performs generally. Precision focuses on the correctness of positive predictions – of all instances the model marked as positive, how many were actually correct. Recall quantifies how well the model identifies actual positive instances – of all real positives, how many did the model predict correctly. Lastly, the F1 score blends precision and recall into a single metric, which is particularly useful when there is an imbalance in the classes being predicted, helping to provide a more balanced view of model performance.

Imagine a doctor diagnosing a disease. Accuracy is like saying how many times the doctor gets the diagnosis right overall. Precision would be reflecting on how often a positive diagnosis made by the doctor is actually correct, while recall looks at how many real patients with the disease were diagnosed correctly. The F1 score is akin to having an evaluation metric that considers both the doctor's accuracy and their reliability in diagnosing the disease effectively, thus giving a fuller picture of their capability.

A Confusion Matrix is a 2x2 table that helps visualize the performance of a classification model.

Predicted Positive Predicted Negative Actual Positive True Positive (TP) False Negative (FN) Actual Negative False Positive (FP) True Negative (TN)

This table helps calculate accuracy, precision, recall, etc.

A Confusion Matrix is a tool that provides a visual representation of how a classification model performs. It is laid out in a 2x2 format showing actual versus predicted results. The four quadrants include True Positives (correct predictions of the positive class), False Negatives (missed positive predictions), False Positives (incorrect predictions of the positive class), and True Negatives (correct predictions of the negative class). This structure allows us to derive important metrics like accuracy, precision, and recall, making it easier to see where a model is succeeding and where it is failing.

Think of a school teacher grading multiple-choice tests. The confusion matrix can be imagined as a grading sheet where you log how many students answered correctly (True Positives), how many students who answered incorrectly should have gotten the answer right (False Negatives), how many students wrongly marked an answer as correct (False Positives), and how many students answered incorrectly as intended (True Negatives). This sheet helps you understand not only how many students passed or failed but also specific patterns in their responses.

Overfitting • The model performs well on training data but poorly on test data. • Learns noise and unnecessary details. Underfitting • The model performs poorly on both training and test data. • Fails to learn the patterns.

Goal: Build a model that generalizes well to new data.

Overfitting occurs when a model is too complex, capturing noise and fluctuations in the training data instead of the underlying patterns. This leads to high accuracy during training but poor performance on new, unseen data. Conversely, underfitting happens when a model is too simple, failing to capture the underlying trend in the data, resulting in inadequate performance on both training and testing datasets. The goal of model training is to strike a balance—to create a model that generalizes well, meaning it performs effectively on new data as well as on the training data.

Consider a musician preparing for a concert. If they only practice a single song repeatedly (overfitting), they may be technically skilled at playing just that, but struggle with other songs during the performance. On the other hand, if they just play random notes without focusing on learning the pieces (underfitting), they will not perform well either. The best approach is for the musician to practice a variety of songs to prepare adequately for a concert, ensuring they can adapt to different pieces.

Cross-validation is a method used to test the model multiple times on different subsets of the data to ensure consistent performance. • K-Fold Cross-Validation: The data is divided into K parts, and the model is trained and tested K times. • This helps reduce the variance and gives a more reliable performance estimate.

Cross-validation is a technique that enhances the evaluation of machine learning models by testing them on different subsets of data to assess their performance. The most common method is K-Fold Cross-Validation, where the dataset is divided into K smaller sets, or folds. The model is trained on K-1 folds and tested on the remaining fold; this process is repeated K times, with each fold being used as the test set once. This method helps to mitigate variance in performance measures and provides a more reliable estimate of how well the model will perform on unseen data.

Think of a cooking competition where judges need to taste dishes from various chefs. Instead of tasting just one dish from each chef on a single day (which might give a biased view), they taste one dish from each chef multiple times over several days. This way, they get a better feel for each chef's cooking style and quality. Similarly, cross-validation allows us to evaluate a model's performance more thoroughly across different scenarios.

Common tools used for evaluating models include: • Scikit-learn: Offers functions like accuracy_score, confusion_matrix, etc. • TensorFlow/Keras: Built-in methods to evaluate deep learning models. • Google Teachable Machine: For visual AI models in schools and simple projects.

There are various tools available for evaluating machine learning models. Scikit-learn is a popular library in Python that provides simple functions to calculate performance metrics such as accuracy and confusion matrices. TensorFlow and Keras are more advanced libraries designed to work with deep learning models, providing built-in methods to facilitate evaluation processes. For educational purposes and simpler projects, Google Teachable Machine offers a user-friendly interface to create and evaluate visual AI models without extensive programming knowledge.

Imagine you have a toolbox filled with different tools for specific tasks. Just like you would choose the right tool—like a hammer for nails or a screwdriver for screws—you would pick the appropriate evaluation tool for your AI project. Scikit-learn is like a versatile multi-tool for common models, while TensorFlow and Keras are like specialized advanced tools designed specifically for complex projects, making evaluation more effective.

Let’s say you trained an AI model to detect spam emails. After training: • You feed it 1,000 new emails. • 800 are non-spam (ham), and 200 are spam. • Model correctly identifies 180 spam emails (TP) but wrongly labels 20 ham emails as spam (FP). • It misses 20 spam emails (FN). From this data, you can compute: • Accuracy, Precision, Recall, F1 Score using formulas. • Evaluate if your model is reliable or needs improvement.

In this example, an AI model was developed to identify spam emails. Upon testing with 1,000 new emails, the model made specific predictions—identifying 180 out of 200 actual spam emails correctly, but also mistakenly tagging 20 non-spam emails as spam. Additionally, it missed 20 emails that were actually spam. Using this information, one can compute various performance metrics such as accuracy, precision, recall, and the F1 score to evaluate the model’s effectiveness. This evaluation highlights whether the spam detection model is functioning well or still needs improvement.

Picture a student taking a test on identifying fruits. After training themselves with different fruit pictures, they take a final test with new pictures. If they correctly identify most of the fruit images, the results (like correctly recognizing apples and oranges) will help determine if they truly learned about them or if they need more practice. Similar to how the student's exam results can show their understanding, the AI model's performance metrics give insights into how well it has learned to detect spam emails.

--