Results
Overall, all three metrics indicated that Aurora AI generated designs are just as accurate, if not more accurate, than human-created designs.
For Per-Face IOU and Total Area IOU, both the mean and median IOU were slightly higher for Aurora AI models. For the Height & Pitch, Aurora AI models had smaller RMSEs than human-produced models, indicating that the heights and pitches matched up better in the Aurora AI models.
Per-Face Intersection Over Union (IOU)
Aurora found that the human-produced model mean and median for Per-Face IOU were 86% and 91% respectively.
The Aurora AI model’s mean and median were both higher, being 88% and 92% respectively.
Aurora considered 85% IOU as the cutoff for an acceptable roof model (only 20% of roofs were found to fall below that cutoff), meaning that both the human-produced and Aurora AI produced models are acceptable roof models.
Figure 1. Per-Face Intersection Over Union median performance. Left: Ground truth. Middle: Error mask. Right: Prediction
Figure 2. Median and Mean Performance, Per-Face Intersection Over Union.
Total Area Intersection Over Union (IOU)
Aurora found that the human-produced model mean and median for Per-Face IOU were 92% and 95% respectively. The Aurora AI model’s mean and median were both higher, being 93% and 96% respectively, indicating that Aurora performed better than human level accuracy. In general, only 12% of AI-generated roofs have errors larger than +-10% of their total area.
Figure 3. Median performance. Left: Ground truth. Middle: Error mask with total area errors in blue and orange (not red). Right: Prediction
Figure 4. Median and Mean Performance, Total Area Intersection Over Union.
Figure 5. Median performance.
Height and Pitch
Aurora found that the human-produced model mean and median for Height & Pitch were 0.934 meters and 0.40 meters respectively.
The Aurora AI model’s mean and median were 0.28 meters and 0.16 meters respectively, indicating that the height and pitch of the models generated by Aurora AI were more accurate.
Figure 6. RMS_Z performance of height and pitch.