29.10.1. Bias

• Error due to wrong assumptions in the model.

Bias in a model refers to the error that arises from overly simplistic assumptions made by the model when trying to learn from the training data. When the model makes assumptions that don't accurately represent the true underlying relationships in the data, it can lead to consistent, systematic errors in predictions. Essentially, bias reflects the model's tendency to miss relevant relations between features and target outputs.

Imagine you are trying to predict the outcome of a basketball game based solely on the players' heights. If height is your only factor, your predictions may not reflect the game's actual outcome because you are ignoring many other important aspects such as teamwork, strategy, or player condition. This simplified assumption leads to biased predictions, similar to how a biased model overlooks important data relationships.

• High bias = underfitting.

When a model has high bias, it fails to capture the underlying patterns in the data, leading to underfitting. Underfitting occurs when a model is too simple to learn from the complexity of the training data, resulting in low performance both on training data and unseen data. In other words, the model is not utilizing enough information to make accurate predictions.

Consider a student studying for a math exam. If they only memorize a few basic formulas without understanding how to apply them in different situations, they will likely struggle with exam questions that require deeper knowledge and application. This lack of depth in understanding mirrors how a high-bias model fails to learn effectively from the data.

Variance: • Error due to too much sensitivity to small variations in the training set.

Variance refers to the model's tendency to be influenced too heavily by small fluctuations or noise in the training set. A model with high variance learns the training data very well, including its noise and outliers, but performs poorly on new, unseen data. This means it has become too tailored to the training data, failing to generalize to other datasets.

Think of a chef who tries to perfect a dish using only one set of ingredients and avoids making adjustments when using different ingredients or cooking methods. If the chef only follows this specific recipe, they may struggle when presented with variations, leading to inconsistent performance. This illustrates how a model high in variance struggles with new data just like the chef struggles with variations in a recipe.

• High variance = overfitting.

Overfitting occurs when a model learns the details and noise of the training data to such an extent that it negatively affects its performance on new data. High variance is a sign of overfitting; the model fits too closely to the training data and fails to generalize. This results in great accuracy on training data but poor accuracy on unseen test data.

Imagine a person rehearsing for a play but memorizing their lines so strictly that they can't adapt when things go off-script. If the person can't think on their feet or adjust to unexpected changes during the performance, they will struggle to engage with the audience. Similarly, an overfit model cannot adapt to new data, mirroring how the actor fails to connect due to an inability to think outside their memorized lines.

--