Methodology
Over the course of summer 2021, Aurora conducted a study to validate the accuracy of Aurora AI using a dataset of 4,000 residential homes across the United States. Only locations with LIDAR data and high quality (HD) imagery were included in the dataset.
Note: The accuracy of Aurora AI is heavily dependent on the quality of its data inputs. For this reason, Aurora AI is designed to perform best when using HD satellite imagery. Although Aurora AI will still run when using standard resolution imagery, the accuracy of the AI degrades by around 20%. Aurora estimates that around 80-95% of solar buyers are covered by Aurora’s HD map imagery data and around 98% of the US population is covered by LIDAR data.
Methodology — Details
There were three main groups in the study:
1. Control: Human-produced model designed on Aurora’s Design Mode
2. Aurora AI: AI-produced site model
3. Human-produced: Human-produced model designed on Aurora’s Design Mode
Over the course of summer 2021, Aurora conducted a study to validate the accuracy of Aurora AI using a dataset of 4,000 residential homes across the United States.
For the human-produced datasets, Aurora had two users create roof models using Aurora’s platform. All of the human-produced models were deemed accurate by the Aurora team, and one of the models was assigned to be in the control group and the other model was placed into the human-produced dataset. Because every site had two unique human-produced models, the comparison between the Human-Produced and Control cohorts demonstrates the variability and error that occurs through manual modeling.
Since the dataset consists only of Aurora user designs that were deemed accurate, they are not a representative sample of all geographies and roof types.
The roof models generated by each cohort were compared the control group using three different metrics:
- Per-Face Intersection Over Union
- Total Area Intersection Over Union
- Height & Pitch
We take a detailed look at Intersection Over Union and Height & Pitch below.
Intersection Over Union
Intersection Over Union, or IOU, is the fraction of correctly predicted areas divided by the total area. There are three types of errors related to IOU:
- Mismatch: Occurs when ridge lines along roof faces are misplaced
- False Negative / Positive: Occurs when a part of a roof face is missed or added to where it does not belong
- Split / Merge: Occurs when two separate roof faces are identified as a single roof face
Per-Face IOU is the correctly predicted area divided by the total area, by roof face. All three types of error were considered in the Per-Face IOU comparison.
On the other hand, Total Area IOU is just the correct predicted area divided by the total area. Mismatch and split/merge errors are ignored and only false negative/positive errors are considered.
Height & Pitch
The Height and Pitch were measured with a metric called RMS_Z, which is the root-mean-square deviation (RMSE) of height values that are within both the ground truth and predicted roofs. If either the heights or pitches do not match up, this metric will detect the error, so the greater the accuracy, the smaller the RMSE.