That's a fabricated example, really. Real maps do it. The JRC looked at its own Global Forest Cover map for 2020, which was produced as part of the EU Deforestation Regulation. The overall accuracy was 91%. 18% of the area that the map said was forest wasn't, in fact, forest, and 8% of the actual forest was missed. The headline was fine. The class that actually mattered was less accurate than the headline implied.
This post is about land cover accuracy assessment for the one class that your decision is going to be based on. That's the wetland in an environmental assessment, the built-up land in an urban planning assessment, the single crop in an agricultural monitoring scenario. In our last post, we looked at the accuracy of AI-extracted buildings in terms of precision, recall, and a sampling check here. This post is the same approach applied to classified maps where every pixel has a class, and one of those classes is the one that drives your decision.
Why Overall Accuracy Hides the Class That Matters
Overall accuracy is just one number really: the number of samples the map got right divided by all the samples. That seems fair enough. The problem is, of course, that the big classes dominate it.
Class imbalance flatters the map
A case in point would be a district where the ground is predominantly cropland, built-up land, and water, with wetland covering just 4%. The classifier can get the major land cover classes spot on and fail completely on wetland, yet the overall score is still in the 90s. Wetland is only 4 samples per 100, so a total failure there is at most a loss of 4 points. The twist is that the project that commissioned the map may care about those 4 points and nothing else.
What the forest map shows
The JRC forest map is a perfect real-world example. The assessors looked at 21,752 sample units, and the errors were not distributed evenly. You had clusters of misclassifications in areas of open dry forest (e.g., eastern Brazil and central Australia) where the lack of tree cover is very easy to misinterpret as forest. The JRC makes it very clear that the map is an aid to risk assessment only and is neither mandatory nor legally binding. A national or global average will not tell you anything about how the map performs on your individual plots.
It's not just forest. The Land & Carbon Lab's comparison of global land cover maps found accuracy in the region of 50% in mixed/heterogeneous landscapes, and no class performs well across all products. Wetlands are a known liability. A comparison of two continental products over Africa found they performed very poorly on wetlands, shrublands, and snow/ice, with agreement indices below 0.4.
We made the same point with regard to drone surveys 5cm GSD does not mean 5cm accuracy. Thematic accuracy is subject to the same trap. You have a headline figure for your data, and your client is basing their decisions on that figure.
Reading a Confusion Matrix: User's and Producer's Accuracy
A confusion matrix compares what the map says with what the reference source says, sample by sample. The rows are the map classes, the columns are the reference classes, and the diagonal is where they agree. The table below is illustrative (it's made up numbers for teaching purposes) from 1,000 randomly placed reference points, not from a Spatialox project.
| Map class | Ref: Cropland | Ref: Built-up | Ref: Water | Ref: Wetland | Row total | User's accuracy |
|---|---|---|---|---|---|---|
| Cropland | 565 | 10 | 4 | 12 | 591 | 95.6% |
| Built-up | 18 | 235 | 0 | 0 | 253 | 92.9% |
| Water | 12 | 2 | 100 | 8 | 122 | 82.0% |
| Wetland | 5 | 3 | 6 | 20 | 34 | 58.8% |
| Column total | 600 | 250 | 110 | 40 | 1,000 | |
| Producer's accuracy | 94.2% | 94.0% | 90.9% | 50.0% | Overall: 92.0% |
Illustrative data. Percentages derived from the counts.
Sum the diagonal, and you get 920 correct out of 1,000 (92%). Now look at wetland. Of the 40 reference points that are actually wetland, the map has found 20. That's a producer's accuracy of 50% (half the wetland is missing). Of the 34 reference points that the map has classified as wetland, only 20 are actually wetland. So the user's accuracy is about 59%. Roughly four in ten of the points mapped as wetland are in fact something else.
What the two numbers answer
So what does each number mean? Producer's accuracy asks: of the wetland that is actually there, what proportion did the map pick up? That's an omission error. User's accuracy asks: when the map says it's wetland, how often is it correct? That's a commission error. ML people would call those recall and precision. We went through them in the AI validation post when we were looking at building footprints, so I won't go through them all again.
The main point with land cover is actually quite simple. You need the numbers for the class that is important, and you need them separately. Different decisions will penalise different types of error.
How many reference points does a rare class need?
A 50% producer accuracy from 40 points is not precisely 50%. With that small a sample, the 95% interval is going to be around +/-15 percentage points (my quick binomial approximation). It shrinks to +/-10 with 100 points and to +/-5 with 400.
So the issue isn't how many samples you've taken in total. It's how many of them are in the class you're actually interested in. If wetland is 4% of the area, then a simple random sample of 1,000 points will give you approximately 40 wetland points. That's why rare but important classes are generally sampled in a stratified manner. The good practice recommendations of Olofsson and co-workers advocate a probability sampling design that is tailored to the project's objectives and with confidence intervals reported on both accuracy and area.
Are overall accuracy figures of any use at all? And what about kappa?
Overall accuracy is by no means useless. It's just a very poor summary in isolation. Keep it, but put it alongside the per-class user's and producer's accuracy with the intervals.
Kappa is quite a different proposition. After a number of years working with it Pontius and Millones recommended that the kappa family be abandoned in favour of quantity disagreement and allocation disagreement. Quantity disagreement tells you whether the map has the right amount of each class. Allocation disagreement tells you whether that amount is in the right places. A map could have the correct total amount of wetland but in the wrong places, and a single kappa value would not reveal that.
Land-Cover Accuracy Assessment Needs Independent Validation Data
Why random splits flatter your map
Neighbouring pixels are very similar. If you just scatter your reference points at random and split them into train and test sets, then lots of the test pixels will be right next to training pixels from the same patch. The model has effectively just seen them. The score is therefore a measure of memory rather than skill.
The studies are unequivocal on the magnitude of the effect. In the case of tree species mapping from drone imagery, random hold-outs can inflate performance by as much as 28% relative to spatially blocked hold-outs. A roof detection study found that random splits and k-fold cross-validation can bias accuracy by as much as 17%, and overall accuracy can be artificially pushed above 99%. In the case of Sentinel-2 forest mapping, spatial cross-validation was found to be a much better reflection of the actual map quality than the standard non-spatial approach, which overestimates it.
What independent looks like
What you need to do is separate the two tasks. The training data will be used to train the classifier. The validation data will be used to assess the finished map. The validation points therefore need to be a probability sample drawn independently of the training polygons. They need to be labelled by someone who hasn't seen the classifier's output against a better reference than the imagery you've classified (higher resolution imagery or field checks would be such a reference). If you can only split one dataset, then split by site or spatial block, not by pixel.
And of course treat the reference labels as fallible too. The very same good practice paper lists assessment of variability and error in the reference classification as part of the job.
Why the Mapped Area Is Not the True Area
Here's another important point. Counting wetland pixels gives you an area, and classification errors will bias that area. The recommended 'best practice' is to estimate the area from the reference sample and report a confidence interval (which is exactly what the same paper recommends).
Using the example matrix above: the map labels 34 of the 1,000 points as wetland, so the pixel count gives 3.4%. The reference points indicate that 40 of the 1,000 are actually wetland, so the sample estimate is 4.0% (my calc) with a 95% interval of c. 2.8% to 5.2%. On a 100,000 ha study area, that's 3,400 ha on the map and somewhere between 2,800 and 5,200 ha on the evidence. In a stratified design, you weight each class by its mapped area, but the principle is the same. If your report gives a wetland area for a baseline, then that interval should be alongside the number.
When is a land cover map good enough?
This is where most projects get stuck. Pontius, Francis, and Millones argue that judging results as acceptable or not on the basis of arbitrary metric thresholds is unscientific. I'd put it in practical terms: a threshold only has any meaning if it's derived from what the error costs. You start with the decision and then pick the error that you can afford.
| Decision the map supports | Error that costs the most | Number to check |
|---|---|---|
| Screening sites for wetland before an environmental assessment | Missing real wetland (omission) | Producer's accuracy for wetland |
| Restricting land use or issuing notices based on mapped wetland | Flagging land that is not wetland (commission) | User's accuracy for wetland |
| Reporting wetland area for a baseline | Both errors, plus a biased area figure | Area estimate with its confidence interval |
| Tracking one crop type for acreage estimates | Usually bias in the area figure | Both accuracies plus the area estimate |
Then put it in the scope prior to anyone classifying a pixel. For instance: for the wetland class, producer's accuracy of at least [X]% and user's accuracy of at least [Y]%, each with a 95% confidence interval, estimated from an independent probability sample with at least [N] reference points in that class. The X, Y, and N are all deliberately left as placeholders. They are derived from what a wrong answer costs you (as agreed with the client), not from some paper or the last project that you've done.
If the map fails the test, then you have a few options. You can improve the weak class via more training samples in the confusable areas. You can add in other inputs such as seasonal imagery or radar (the global products that use time series are better at handling seasonal classes, as the Land & Carbon Lab comparison above demonstrates). You can merge the classes that can't be separated. Or you can just use the map for decisions where the weak class doesn't actually matter. If a client comes to you and challenges your numbers, then the answer is evidence (the same logic as in how to prove an accuracy claim was met).
This is the reporting format I'd want to see on any classified deliverable (ours included) i.e., a per-class table with intervals, the sampling design in one paragraph, and area figures with their uncertainty. We see all too often the case of maps that look very good in terms of presentation but are built on very weak foundations. Our post on flood maps and DEM quality is a case in point.
What to Take Away
Three things. Decide which class is the one that makes the decision prior to any pixel being classified. Ask for the user's and producer's accuracy for that class (with intervals) from independent validation points. Treat the mapped area as an estimate rather than a measurement.
If you're looking at a classification project or you've got a map whose accuracy no one's actually checked for your class, our remote sensing team can help you put together the validation before the results are due. The first question we always ask is "what would it cost you to get it wrong?"