6.4. Handling Missing and Incorrect Data

Sometimes, data is incomplete. Common reasons:

• Human error during data entry
• Data corruption

Missing values in datasets occur when some data points are not recorded or available. This can happen due to simple mistakes like typos when entering data (human error) or issues like system malfunctions leading to lost data (data corruption). Understanding why data is missing is crucial as it can affect the analysis.

Imagine you are putting together a puzzle, but some pieces are missing. You can't see the full picture because key sections are absent. Similarly, when data is missing, it can lead to incomplete analysis and inaccurate results.

Techniques to Handle Missing Data:

• Remove rows or columns with missing data
• Fill with average/mean/median
• Fill with a default or most common value

To manage missing data, several techniques can be employed:

1. Remove Rows/Columns: If too much vital data is missing from a row or column, it may be better to discard it altogether.
2. Fill with Average/Mean/Median: Another common method is to replace missing values with the average (mean) or median of the existing data, which can help maintain the overall data distribution.
3. Fill with a Default Value: Lastly, sometimes you can substitute missing values with a default or most frequent value from that column. This method might be useful in certain contexts where a specific value makes sense, like using '0' for number fields when relevant.

Think of treating a plant. If one branch is dying, you can either cut it off (remove), use fertilizer to boost growth (average), or replace it with a healthy cutting from another plant (default value). Each choice impacts the overall health of your garden (dataset) differently.

An outlier is a data point that differs significantly from other observations. Example: A student scoring 100 when most scored between 30-70.

Outliers are values that stand out from the rest of the data because they are much higher or lower than the rest of the values. For instance, if most students score between 30 to 70 on a test, a score of 100 would be considered an outlier. Identifying outliers is essential because they can skew the results of statistical analyses if not accounted for properly.

Imagine you are at a party, and everyone is dressed casually, but one person shows up in a tuxedo. This person, like an outlier, looks very different from the norm and could be seen as exceptional. In data, outliers may reveal interesting trends or errors in data collection.

Handling Outliers:

• Visualize using graphs (box plots, scatter plots)
• Decide whether to keep, transform, or remove them

To address outliers effectively, you can take the following steps:

1. Visualize: Use graphs like box plots or scatter plots to get a clear picture of where outliers fall relative to other data points. This visualization allows for easier identification of how outliers affect the dataset.
2. Decision Making: After identifying outliers, you need to decide what to do with them. You could keep them if they are valid data points that provide useful information, transform them if they seem to distort conclusions (like normalizing them), or remove them entirely if they are erroneous.

Think of a sports team where most players score between 10 to 20 points per game. However, one player regularly scores 50 points. You could analyze why this player performs differently (keep), adjust their training strategy to help others improve (transform), or decide that they are a one-off case to be excluded from the team's average performance metrics (remove).

--