The AI mapped the whole city over the course of the night. Ten thousand buildings, all drawn as clean polygons, on the map. Looks complete.
The problem is that no one knows how much of that map is wrong.
That’s the quiet risk with AI feature extraction (i.e., the AI looks at imagery and detects buildings, roads, rooftops, land cover, and draws them as GIS features). A high detection count is going to be interpreted as proof of success. It isn’t. It just tells you how many things the AI has detected, not how many of them actually exist, and it tells you absolutely nothing about the buildings that the AI has failed to detect. So to the point: before any AI output goes into an operational GIS, you need two numbers (precision and recall), and you need them for each feature class and each type of area. This post looks at the errors you will actually see, why one accuracy figure will hide all of them, and a sampling workflow you can run prior to anything being published.
A Big Detection Count Is Not a Quality Check
What AI Feature Extraction Gets Wrong
The errors aren’t random. They repeat, and once you’ve got the patterns, you can spot them.
Phantom buildings. In a USGS led study presented at the 2026 ISPRS Congress, a model fed aerial imagery plus lidar-derived height and intensity still classified tall tree crowns as buildings, and the much simpler 4-band version also picked up vehicles. A 2026 preprint on UAV footprints describes the same family of false positives: polygons that look like buildings but are in fact on vegetation, on roads, in shadows, or on bare soil.
Merged rooftops. Semantic segmentation labels pixels, not buildings. If two roofs are adjacent, then the model will often end up with a single blob. That’s exactly what the TernausNetV2 authors describe as closely packed objects of the same class collapsing into a single connected component. A practitioner test published this month demonstrates the effect on a count. In the old town of Vitré in France, there were 447 Microsoft polygons, each covering two or more buildings from the reference layer, and the largest of these covered 66 houses. (Again, this was just one town, and the author’s company sells QGIS tools, so it’s a worked example, not a benchmark in any way.)
Missed small structures. Sheds and the like are very easy to lose. Overture Maps in its Buildings Guide states that during the conflation process it will drop machine learning detections below a certain size and that this can result in some valid features being missed. The minor errors that were left in the USGS study were predominantly small buildings adjacent to tree canopy. Google’s own Open Buildings paper also notes failures on large and complex buildings and spurious detections where natural features are misinterpreted by the model.
Roads that look right but don’t connect. Here the score itself misleads. SpaceNet points out that the pixel metrics will penalise a small error in road width very heavily, whereas a short gap caused by a shadow or an overhanging tree will be penalised very lightly. That’s why they’ve got a path length metric. A road layer can score very well and still not be able to route a vehicle. We looked at the network side of that in the road connects; the network says otherwise.
The Same Model Gives Different Quality in Different Places
Here’s the interesting bit. Quality isn’t a property of the model in itself. It’s the model, the imagery, and the location all together.
A comparison across five cities looked at Microsoft’s footprints against OpenStreetMap. The average IoU (the overlap between the matched polygons) was around 0.84 in Houston and around 0.46 in Caracas. Google’s footprints in Caracas averaged around 0.27. The authors are quite upfront about one of the limitations of OSM: it’s not the absolute truth. So treat these as agreement scores rather than absolute accuracy. The point is still valid, though. A figure quoted for one location is not necessarily going to be the same in your location.
Why isn’t overall accuracy enough?
The 5% Problem
Suppose buildings cover 5% of an image. A model that detects nothing at all will have 95% of the pixels correct. Overall accuracy is going to favour the large, easy class (which in this case is background). That’s why guidance on class imbalance in RS (remote sensing) is so strongly in favour of class-level stats and why a headline figure of “96% accurate” tells you absolutely nothing about buildings.
Precision and Recall in Plain Words
One note on terminology. By “precision” here we mean classification precision, not measurement repeatability and it’s a quite different concept to the positional accuracy that you’d report on a survey.
| The plain question | Metric | Error it exposes |
|---|---|---|
| Of what the AI actually drew, how much of it is real? | Precision (user’s accuracy) | Commission: false features |
| Of what is actually there, how much did the AI draw? | Recall (producer’s accuracy) | Omission: missed features |
The Olofsson et al land change mapping guidance is precisely what is being asked for: the error matrix, the overall accuracy, the user’s accuracy (commission) and the producer’s accuracy (omission) with confidence intervals. It’s written in the context of land change maps, but the same logic applies to any AI-extracted layer. ISO 19157 has both types of error (commission and omission) under completeness (alongside positional, thematic and temporal quality) as per this review of that standard.
When the Totals Look Fine, and the Features Do Not
Let me give you an example. These figures are fabricated to illustrate the mechanics; they’re not taken from a Spatialox project.
You have 10,000 buildings in your layer. A review shows 88% are real (so 8,800 are correct and 1,200 aren’t). You then go and map a few sample areas by hand and find the AI has caught 82% of the buildings that actually exist. So the real number of buildings is around 10,700 and approximately 1,900 are missing.
Read that again. The layer says 10,000, and the reality is circa 10,700, so the total is pretty much in the ballpark. There are however some 3,100 individual features that are wrong or missing. The false detections and misses largely cancel each other out in the total, and that’s precisely how a layer will pass a quick sanity check. The Vitré test on the actual data shows the same thing: Microsoft’s total footprint area was within about 7% of the reference (44.6 vs 41.6 hectares) and its building count was less than half (1,636 vs 3,715).
So the appropriate metric depends on the job. If you’re after area, then area is the metric. If you’re after a count per parcel, then you want the count. If you’re after routing, then you want connectivity.
How do you measure what the AI got wrong?
Check Precision by Sampling the Detections
This is straightforward, really. Take a random sample of the detections and check each one against imagery or the field. If your area is a mix of dense blocks, rural land, and industrial sites, then use a stratified sample. That’s what Olofsson and co. would recommend for that very reason.
In our hypothetical example, we’re looking at 400 detections, and 352 of those are correct (88%). With a simple random sample, the 95% confidence interval is about ±3 percentage points (my arithmetic). The true precision is therefore somewhere in the region of 85% to 91% (i.e., between 900 and 1500 false features in 10,000). That range is vastly more useful than a single percentage with no error bar attached.
Check Recall by Mapping What Is Actually There
This is the step that people invariably skip, and it’s the difficult one. You can’t measure omission from the list of detections as the buildings that have been omitted aren’t on the list. You have to go to sample areas, map every building individually, and then see how many the AI has found. Take tiles across different settlement types and do it by careful digitising from better imagery or by field check.
Choose the Reference Carefully
The reference has to be more reliable than the AI (otherwise you’re just measuring agreement rather than accuracy). The Olofsson paper makes the exact same point: the reference classification should be considerably more accurate than the map under test.
The USGS Houston study is a good warning. The labels were taken from Microsoft’s footprints, and the authors point out that errors in that reference get propagated into the scores. They also report that the scores dropped on tiles the model had never seen: F1 of 0.87 on the validation data vs 0.79 on 15 held-out tiles and IoU of 0.76 vs 0.66. A drop like that is absolutely normal. That’s precisely why you test on data that you haven’t seen before.
A second warning comes from the UAV preprint. Both of its segmentation models achieved a pixel-level F1 of nearly 0.91 on the validation chips. After vectorisation, a human reviewer found 27.32% of the footprints (1,137 of 4,162) on a separate site to be wrong (bad boundaries, merged shapes, etc). That doesn’t even include the buildings that were never drawn at all. Pixel scores and a clean database are two very different things.
Is a confidence score the same as accuracy?
No. The confidence score is the model’s own confidence, not an error rate. Google’s Open Buildings assigns each footprint a score between 0.65 and 1.0. For each region, they also publish the score threshold that would be expected to give 80%, 85%, or 90% precision at an IoU of 0.5, and recommend starting by filtering out detections below the 90% threshold. Overture’s December 2023 release used that 90% threshold to separate the high- and low-precision Google buildings when combining the sources.
That is a useful filter. Google’s paper, though, makes it clear that the impact of thresholding on recall is unknown, so a filter can improve precision while quietly reducing recall. Use the threshold to clean the layer and then resample.
A Practical QA Workflow Before Anything Is Published
This is how I’d set it up. It’s a suggested workflow rather than a formal standard, so you can adapt it to the risk of your particular project.
- Write down what “good enough” means. Set an error tolerance per feature class and per use. A roof layer for solar screening can have a very different error tolerance to a layer which is going to be used for cadastral work.
- Triage automatically. Flag the tiny polygons, area outliers, odd shapes, overlaps and duplicates, and the low confidence features. In the UAV preprint, a decision tree screen based on shape and neighbourhood context measures picked up 87.34% of the erroneous footprints on the previously unseen site and retained 98.31% of the correct ones, reducing the residual error from 27.32% to 4.62%. One preprint from one region with RGB drone data, so treat it as a direction rather than a guarantee.
- Sample the remainder for precision. Take a stratified random sample of those that passed triage and calculate precision with a confidence interval.
- Measure recall on reference areas. Map the sample tiles independently and see what the AI has missed.
- Decide per feature class. Publish, publish with a quality flag, send for manual edit, or re-run the model. Store the results with the data (precision, recall, sample sizes, reference used, and imagery date). The imagery date is important, as a model run on 2023 imagery will have no knowledge of a building that’s appeared in 2025.
The same applies to QA/QC as the real bottleneck in drone mapping. Getting the data is the easy bit. Trusting the data is where the time is spent. That’s the same trust question raised by why Google pulled the AI satellite imagery tool, and it’s the very same issue with a change detection map that shows change. You still have to go and verify that the ground actually has changed.
What to Take Away
Three things. A detection count is not an accuracy measure. Precision and recall (checked per feature class and per area) are what tell you what any one accuracy figure is really worth. A few hundred features reviewed and a few independently mapped tiles will tell you far more than any vendor’s claim.
AI can produce the map in the course of an overnight run. It takes a few days of checking to trust it and I think that’s a reasonable price to pay. If you’d like some help in setting up your own check on your own data then do get in touch with the team via the Remote Sensing page.