Mean Squared Error (MSE)

Concept: Imagine you predict an exam score of 80, but the student actually got 75. The error is 5. If another student was predicted 90 but got 98, the error is -8. MSE takes all these individual errors, squares them, and then averages them. Squaring the errors does two important things:

1. It makes all errors positive, so positive and negative errors don't cancel each other out.
2. It penalizes larger errors much more heavily than smaller ones. An error of 10 contributes 100 to the MSE, while an error of 5 contributes only 25.

Formula:

$$MSE = \frac{1}{n} \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2$$

Where:
- n: The total number of observations (data points).
- Y_i: The actual (observed) value of the dependent variable for the i-th observation.
- \hat{Y}_i (pronounced "Y-hat sub i"): The predicted value of the dependent variable for the i-th observation.

Interpretation:
- A lower MSE indicates a better fit of the model to the data. The closer the predicted values are to the actual values, the smaller the squared differences, and thus the smaller the MSE.
- Units: The unit of MSE is the square of the unit of the dependent variable. If you're predicting prices in dollars, MSE will be in "dollars squared," which isn't very intuitive. This is why RMSE is often preferred.

Root Mean Squared Error (RMSE)

Concept: RMSE directly addresses the unintuitive units of MSE. It's simply the square root of the MSE. By taking the square root, RMSE brings the error metric back to the same scale and units as the original dependent variable. This makes it much easier to interpret the magnitude of the errors in a practical context.

Formula:

$$RMSE = \sqrt{MSE}$$

Interpretation:
- Like MSE, a lower RMSE signifies a better-performing model.
- Units: The units of RMSE are the same as the unit of the dependent variable. So, if you're predicting prices in dollars, RMSE will be in dollars. An RMSE of $5 means, on average, your predictions are off by about $5. This makes it a widely used and highly interpretable metric.
- Sensitivity to Outliers: Since it's derived from MSE (which squares errors), RMSE is still sensitive to large errors (outliers).

Mean Absolute Error (MAE)

Concept: Instead of squaring the errors, MAE takes the absolute value of the differences between predicted and actual values. This means it measures the average magnitude of the errors without considering their direction (whether the prediction was too high or too low).

Formula:

$$MAE = \frac{1}{n} \sum_{i=1}^{n} |Y_i - \hat{Y}_i|$$

Interpretation:
- A lower MAE signifies a better model.
- Units: The units of MAE are also the same as the unit of the dependent variable, similar to RMSE.
- Robust to Outliers: Unlike MSE and RMSE, MAE is less sensitive to outliers because it doesn't square the errors. A very large error contributes proportionally to the MAE, rather than disproportionately as in MSE/RMSE. If your data contains many outliers, MAE might be a more representative measure of typical prediction error.

R-squared (R²)

Concept: R-squared, also known as the coefficient of determination, is a very popular metric that tells us the proportion of the variance in the dependent variable that can be explained by our independent variables in the model. Think of it as answering the question: "How much of the variability in the target variable can our model explain, compared to simply guessing the average value?"

Formula:

$$R^2 = 1 - \frac{SS_{res}}{SS_{tot}}$$

Let's break down the components:
- SS_res (Sum of Squares of Residuals): This is the sum of the squared differences between the actual observed values (Y_i) and our model's predicted values (\hat{Y}_i). It represents the unexplained variance or the variance that our model could not account for.

$$SS_{res} = \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2$$

• SS_tot (Total Sum of Squares): This is the sum of the squared differences between each actual observed value (Y_i) and the mean of the dependent variable (\bar{Y}). It represents the total variance inherent in the dependent variable that needs to be explained.

$$SS_{tot} = \sum_{i=1}^{n} (Y_i - \bar{Y})^2$$

Interpretation:
- Range: R-squared values typically range from 0 to 1.
- R² = 0: Indicates that the model explains none of the variability of the dependent variable around its mean. Essentially, your model is no better at predicting than simply using the average of the target variable.
- R² = 1: Indicates that the model explains all of the variability of the dependent variable around its mean. This means your predictions perfectly match the actual values (a perfect fit). This is rare in real-world scenarios due to inherent noise.
- Higher is Generally Better: A higher R² generally suggests a better fit of the model to the data. For example, an R² of 0.75 means that 75% of the variance in the dependent variable can be explained by the independent variables in your model.
- Caution with Interpretation: While R² is useful, it has limitations:
- Adding more predictors (even irrelevant ones) will never decrease R². It will always stay the same or increase, even if the new predictors aren't genuinely helpful. This can lead to overfitting if you keep adding features.
- High R² doesn't necessarily mean the model is good for prediction. A model could have a high R² on the training data but perform poorly on new, unseen data if it has overfit.
- No Causality: R² measures correlation, not causation. It doesn't tell you if the independent variables cause the changes in the dependent variable.