5.17.4. Accuracy assessment

The increased use of remote sensing data and techniques has made analysis faster and more powerful, but the spectral and spatial complexity in the images have also created increased possibilities for errors. Thematic maps generated from remotely sensed image may contain some errors because of; (i) geometric error, (ii) atmospheric error, (iii) clusters incorrectly labeled after unsupervised classification, (iv) training sites incorrectly labeled for supervised classification, and (v) separability of classes.

This chunk introduces the concept of accuracy assessment in remote sensing. It emphasizes the growing use of remote sensing data for fast and powerful analysis while acknowledging that this complexity can lead to errors in classification. Various sources of potential errors are listed, including geometric inaccuracies, atmospheric distortions, mistakes during classification processes, and the challenge of distinguishing between different classes in an image.

Think of remote sensing data like a complex puzzle. The more pieces you have, the faster you can put the puzzle together, but the more pieces there are, the easier it is to get some pieces in the wrong place. Each piece represents a pixel in an image, and if they are not accurately situated, the entire puzzle—or analysis—can be misleading.

Image classification is considered to be incomplete without estimating the accuracy of classification. It measures the agreement between a reference (assumed as 100% accurate) and a classified image of unknown accuracy. The accuracy assessment or error analysis is the quantitative comparison to validate the classified map with the actual reference data/image.

Here, the importance of accuracy assessment is highlighted. It states that for any classification done using remote sensing images, assessing accuracy is crucial. Accuracy assessment quantifies how well the classified image agrees with a reliable reference. Essentially, it's a quality check to see if the classification aligns with what is actually occurring on the ground.

Imagine you're a teacher grading an exam. You have an answer key that you believe is perfect. When you grade the tests, you'll compare each student's answers to that perfect answer key. This is similar to how accuracy assessment works in remote sensing—as a check to ensure the classification is correct against a trusted source.

A typical approach for validation and accuracy assessment is to use a statistically sound sampling design to select a sample of ground locations (number of pixels) in the scene. The land use or land cover classification assigned at these ground locations is actually compared with the true classification to ascertain the accuracy. An error matrix or confusion matrix is thus generated, as shown in Figure 5.48.

This chunk discusses the method of validating the classifications through ground sample locations. It explains that using a systematic sampling design is essential to ensure that selected samples accurately represent the larger area being studied. By comparing these selected ground locations to the classifications made in the remote image, an error matrix can be created, which helps visualize and quantify the accuracy.

Consider the process of testing a new recipe. You wouldn't taste every single batch you make for quality; instead, you might do a taste test on a few selected servings. By comparing these selected samples to your ideal outcome, you get a good idea of how well your recipe is performing. This sampling process parallels the way ground samples help validate remote sensing classifications.

To select the ground samples, five important sampling techniques are often used. These are: 1. Simple Random Sampling: observations are randomly placed, and no rules are used. 2. Systematic Sampling: observations are placed at equal intervals according to a planned strategy. 3. Stratified Random Sampling: a minimum number of observations are randomly placed in each class. Sampling points are generated proportionate to the distribution of classes in the image. 4. Stratified Systematic Unaligned Sampling: a grid is laid out which provides even distribution of randomly placed observations. 5. Cluster Sampling: randomly placed “centroids” used as a base of several nearby observations. The nearby observations can be randomly selected, systematically selected, etc.

This portion outlines various strategies to select ground samples for accuracy assessment. Each technique has its advantages and is useful in different situations. For instance, simple random sampling ensures no bias but doesn’t account for specific class distribution. Systematic sampling provides equal representation based on a predetermined method, while stratified random sampling ensures that each class is proportionately represented. Cluster sampling uses groups to simplify the process of gathering samples.

Imagine you're surveying a large park. If you just picked random spots, you might miss out on certain areas like playgrounds or ponds. By organizing your sampling—perhaps visiting every tenth bench (systematic), ensuring you check both playgrounds and quiet areas (stratified), or sampling from different regions of the park as clusters—you can ensure your survey accurately reflects the entire park experience.

Columns of the table in Figure 5.48 are the reference (ground truth) classes, while rows are the classes of classified image whose accuracy is to be assessed. Various cells of the Table show number of pixels for all the possible correlations between the ground truth and the classified image.

This section describes the structure of the error matrix, a crucial tool for evaluating classification accuracy. The matrix displays the relationship between classified data and actual ground truth data. The columns represent the actual land cover classes, while the rows show how classified results correspond with these truths, allowing users to see where classifications meet or fail.

Think of the error matrix like a scoreboard in a sports game, where each team’s score corresponds to how many correct points they received from their plays (ground truth) against what the referees ruled (classified image). By looking at the scoreboard, you can quickly assess the performance of each team and spot where they scored right or wrong.

The accuracy of a classification can be defined as; (i) Overall accuracy, (ii) Producer’s accuracy, (iii) User’s accuracy, and (iv) Kappa coefficient.

This chunk introduces the different metrics used to quantify classification accuracy. Overall accuracy provides a general assessment of how many pixels were correctly classified compared to the total number of samples. Producer's accuracy measures how well the reference data have been classified, while User's accuracy evaluates how reliable the classified results are regarding ground truth. Kappa coefficient is a statistical measure that accounts for chance agreement.

Imagine a student receiving grades for various subjects. Overall accuracy is like their average grade in all subjects, Producer's accuracy reflects how well they did in major exams (the subjects they are focused on), User's accuracy shows their performance in class participation (how well they represent their knowledge), and the Kappa coefficient is akin to adjusting grades considering that some tests are easier than others.

The non-diagonal cells in the matrix contain classification errors, i.e., the number of pixels in reference image and the classified image don’t match. There are two types of errors: under-estimation (omission errors) and over-estimation (commission errors).

Here, the focus is on the classification errors derived from the error matrix. Omission errors occur when a true class is not represented in the classification (missed classifications), while commission errors happen when pixels belong to one class are incorrectly assigned to another (false classifications). Understanding these errors helps in pinpointing the weaknesses in the classification process.

Think of it like a job application process. If you’re applying to a job that requires certain qualifications, omission errors are like ignoring applicants that truly meet those requirements (they were omitted), while commission errors are like mistakenly hiring someone who doesn’t meet those qualifications (they were wrongly accepted). Understanding these errors helps improve the hiring process.

The overall accuracy of classification can sometimes be misleading, as it does not reveal if error was evenly distributed between the classes or if some classes were really badly classified and some really good. Therefore, it is always better to compute the values of commission and omission errors.

This concluding chunk cautions against relying solely on overall accuracy when assessing classification results. An overall high accuracy might mask significant disparity in errors between different classes of data. Thus, understanding the distribution of errors through commission and omission metrics is essential to truly evaluate the quality of the classification.

It's similar to a restaurant review where an overall rating might seem high due to a few excellent dishes. However, if most items on the menu were below par, that high rating is misleading. Assessing both the excellent and poor dishes gives a clearer picture of the restaurant's performance, just as looking at both omission and commission errors reveals the true accuracy of a classification.

--