4.1. Metric Description

Mean Absolute Error (MAE) Average of absolute errors.

Mean Absolute Error (MAE) quantifies the average magnitude of errors in a set of predictions, without considering their direction. It calculates how far off each prediction is from the actual value and takes the average of those absolute differences. The lower the MAE, the better the model's predictions are, indicating higher accuracy.

Imagine you are a weather forecaster predicting daily temperatures. If your forecast is off by 3 degrees one day, 5 degrees the next, and 0 degrees the day after, the MAE would be (3 + 5 + 0) / 3 = 2.67 degrees. This means, on average, your predictions are off by about 2.67 degrees.

Mean Squared Error (MSE) Penalizes larger errors (squared).

Mean Squared Error (MSE) is similar to MAE in that it measures the average errors in a set of predictions. However, it squares each error before averaging them, meaning larger errors have a greater impact on the MSE value. This makes MSE sensitive to outliers, as a single large error can significantly increase the MSE. Achieving a lower MSE indicates a better-performing model.

Continuing with the weather forecasting example, let's say you predicted 25 degrees, but the actual temperature was 30 degrees. The squared error would be (30 - 25)² = 25. If you had another day where you predicted 15 degrees, but it was actually 30 degrees, the squared error would be (30 - 15)² = 225. The MSE would be the average of these squared errors, which highlights the bigger error more than MAE does.

Root Mean Squared Error (RMSE) Square root of MSE.

Root Mean Squared Error (RMSE) is derived from the MSE by taking its square root. This brings the error metric back to the same unit as the original data, making interpretation easier. RMSE provides a measure that maintains the penalizing nature of MSE while also placing it in understandable units. An RMSE closer to 0 indicates a better fit to the data.

If we return to forecasting, after calculating the MSE from multiple forecasts, say you find it to be 100. Taking the square root gives you an RMSE of 10. This means that, on average, your predictions vary by 10 degrees from the actual temperatures, which is an intuitive figure to grasp since it's in the same unit as the temperatures.

R² Score (R-squared) % of variance explained by the model.

The R² score, or R-squared, quantifies how much of the variability in the dependent variable can be explained by the independent variables in the model. It ranges from 0 to 1; an R² of 0 means the model explains none of the variability, while an R² of 1 means it perfectly explains the variability. A higher R² value indicates a better-fitting model.

Think of a teacher assessing how well their teaching methods explain students' performance on tests. If the R² score is 0.9, it means 90% of the variation in test scores can be explained by the teaching methods, while 10% is due to other factors. This highlights how well your model (or teaching) is performing.

Example:

from sklearn.metrics import mean_squared_error, r2_score
predictions = model.predict(X)
print('MSE:', mean_squared_error(y, predictions))
print('R² Score:', r2_score(y, predictions))

In this example, using Python's scikit-learn library, we evaluate a regression model's performance. We first generate predictions from the model. Then, we apply the mean_squared_error function to compare the actual values (y) with the predicted values (predictions), which provides us with the MSE. Similarly, using the r2_score, we obtain the R-squared value for the model, allowing us to understand how well our model explains the variance in the data.

If you are a chef trying to perfect a recipe and you use this coding example as part of your chefs’ notebook, it will help you assess how well your latest attempt matches the ideal outcome. Just like in cooking when you taste-test to evaluate your dish’s flavor, in modeling, you evaluate the MSE and R² to check how close you are to the perfect recipe for predictions.

--