5.4.1. Detecting Missing Values

Raw data collected from various sources often contains missing values, which can impact analyses and modeling outcomes.

Missing values occur when data is not recorded for certain observations or variables in a dataset. This can happen for a variety of reasons, such as errors during data collection, data entry mistakes, or intentional omissions. Understanding how to detect these missing values is crucial because they can skew results and lead to incorrect conclusions if not handled properly.

Imagine you are trying to calculate the average score of students in a class, but some students didn’t submit their tests. If you include scores of only those who submitted, or if you ignore the missing scores, the average will not accurately represent the class's performance. Recognizing and addressing these missing scores is essential to get a fair representation.

To detect missing values in a DataFrame, you can use the following code:

import pandas as pd
df = pd.read_csv("data.csv")
print(df.isnull().sum())

In Python, detecting missing values can be done using the isnull() function from the Pandas library. When this function is called on a DataFrame, it returns a new DataFrame of the same shape, where each entry is True if the original DataFrame entry is null (missing) and False otherwise. By summing this result (sum()), you get a count of how many missing values are present in each column, which helps in understanding the extent of missing data.

Think of your data as a bookshelf where some book titles are missing. By counting how many spaces are empty in each shelf, you get a clear idea of how many books are missing, which helps you decide whether to fill these gaps by purchasing new books or ignoring them in your collection.

The output from the print(df.isnull().sum()) command shows the number of missing values for each column, helping you to identify which features have missing data.

When you run the command to check for missing values, the output lists each column alongside the number of missing entries. For example, if the output shows that the 'Age' column has 5 missing values, this indicates that there are 5 rows in which the age information is absent. This information allows data analysts to prioritize which columns may need attention or further cleaning.

Imagine you're checking your shopping list and notice that some ingredients are missing for a recipe. By counting and identifying exactly which items are absent, you can decide what you need to buy before you start cooking. Similarly, knowing which data points are missing helps you to plan your next steps in data preparation.

--