Harris Deep Learning Technology
Authors: Michel Ehrlic, Bryan Justice, Braxton Baldridge, NV5
This whitepaper was originally created several years ago to summarize Harris Geospatial Solutions’ research & development in deep learning technology. Since then, we have become NV5 Geospatial Solutions and have released the ENVI Deep Learning module. Please refer to the Documentation Center to learn more about this module and our current technology.
Introduction
“Deep Learning is an algorithm which has no theoretical limitations of what it can learn; the more data you give and the more computational time you provide the better it is.”
- Geoffrey Hinton (Google)
To say deep learning has been a revolutionary advancement in machine learning might seem like hyperbole, but it’s far from it. Although the concepts that drive deep learning have been around for decades, it wasn’t until the recent advancements in processing power, specifically the availability of graphical processing units (GPUs), that the development of applications for deep learning were truly practical – and the effects of that are being seen everywhere. From autonomous vehicles to advancements in proactive healthcare, deep learning is accelerating the advancements of artificial intelligence (AI) unlike anything else.
NV5 recognized this potential early on, and with more than five years of experience and a multimillion dollar investment in deep learning research and technologies, we are at the forefront of this expansion and have positioned ourselves as both a leader and an innovator.
Practical Applications for Deep Learning
At NV5 we looked at AI, and more specifically at deep learning as an enabling technology to help solve real-world customer problems. To that end, we have developed a highly tuned process, relying less on the volume of label data and more on reliable training models and high-performance computing, that allows us to solve those real-world problems quicker and more cost effectively than ever before.
Figure 1: Networked hierarchy of abstract representations needed to find a target object (in this case, faces).
Where We Started
Too many pixels and not enough eyes.
Humans have no problem identifying objects in images. Even when identifying objects in hundreds of photos, humans can still do so with excellent precision. But what about tens of thousands of satellite images or millions of frames of video? Or even billions of points in point clouds? That is where AI, machine learning and deep learning can help.
Figure 2: Comparative large size of target objects (cats and dogs) to the overall canvas.
The first problem we set out to solve was finding objects in an overhead image either from satellite, airborne vehicle, or unmanned aerial systems (UAS). This is a completely different exercise than a more common example of object detection, such as finding a cat in an image.
What is different? For starters, the overall canvas is much smaller, and the cat is typically large compared to the scene it is in. The next difference is that the background is typically uncluttered with cat-like objects. This lack of similarly cluttered data means that there is less data that might confuse the algorithm. All of these factors make it easier to answer the question, “Is there a cat in this picture?”
Now, compare that to a geospatial query: “Is there a plane in this satellite imagery?” While the question is similar in that it is about finding an object in an image, the specifics in the second example make it a far bigger challenge. In terms of the canvas, the scale is massive. Whereas a cat might represent 25 to 40 percent of the pixels, in a satellite image the plane may only consume 1 half of .01 percent of the pixels. Add in the potential for confusion resulting from similar features in the image, and finding the answer becomes much more difficult. Determining the existence or position of an object in an image is one of the most common problems people look to deep learning to solve.
Automated Target Recognition
NV5 Geospatial’s deep learning technologies excel at automated target recognition, obtaining near-ceiling performance on Panchromatic (Pan), RGB, Multispectral Imagery (MSI), Hyperspectral Imagery (HSI), Synthetic Aperture Radar (SAR), LiDAR, and derived point cloud data sets.
The following are a sampling of successful automated target recognition tested on Pan imagery:
-
Airplanes
-
Storage Tanks
-
Sports Stadiums
-
Athletic Fields
-
Smokestacks
-
Cooling Towers
-
Clouds
-
Swimming Pools
-
Buildings
-
Paved Roads
-
Overpasses
-
Tollbooths
Figures three through six are examples of applying NV5’s deep learning research to solve customer problems.
NV5’s deep learning technologies have remarkable success even with extremely limited training data.
Figure 3: Tokyo International Airport, IKONOS Pan, 20/20 planes detected, 1 false positive.
Figure 4: Baseball fields, Orbview Pan, 5/5 full-sized fields detected, 0 false positives.
Figure 5: Storage tanks, ~140 training examples, hi-res Pan satellite data, resolution: 1 m to 2.5 m / pixel, accuracy of 91 percent, tested against hi-res Pan imagery (IKONOS, QB, WV-2, DigitalGlobe).
Figure 6: Similarly shaped “confusers” in urban environments correctly identified as non-storage tanks.
Use of Synthetic Training Data
NV5’s deep learning technologies have also been successful in the use of synthetic training data for model development for automated target recognition.
In a test on Pan imagery, 100 percent of the training data was synthesized using CAD models of fighter jets and a WV-2 scene simulator using Pan imagery. The trained model was then applied to real imagery to perform automated target recognition.
Figure 7: Three examples of the six CAD models used to synthesize training data.
NV5’s technologies were then used to generate synthetic data using different parameters (see figures 8 and 9).
Figure 8: 10 am, 7 degree look angle, Jan 1, Scene Azimuth 0.
Figure 9: 2 pm, 7 degree look angle, Jan 1, Scene Azimuth 225.
Figure 10: Initial results of these combined NV5 Geospatial technologies were very successful.
Classification
NV5 Geospatial’s deep learning technologies also improved classification mapping and produce superior semantic pixel classifications over older spectral techniques. In another example, our technology was used to build a land-cover classification map. The results were excellent. When compared to best practices Spectral Angle Mapper (SAM) techniques, building classification improved by 14 percent and transportation classification improved by 25 percent.
Figure 11: Original image. HarrisImageLinks, 8-bit 4-band WV-2, Melbourne, FL.
Figure 12: This is the result using SAM – best practices spectral mapping, pansharpened, Quick Atmospheric Correction (QUAC).
Figure 13: More precise results from NV5's deep learning.
Condition Assessment – Assesses the State of a Scene or Sub-scene
Another use of NV5's deep learning technologies is condition assessment – understanding the state of a scene instead of just finding targets. As an example, our technology was applied to real-time ground weather intelligence systems in order to determine the conditions of the roads such as wet vs. dry. The model used for this test performed at a 95 percent accuracy and is in use today in a commercial product.
Figure 14: Wet detected roads
Figure 15: Dry detected roads.
Have a Problem that Needs Solving?
We are delivering deep learning solutions to our customers today. We have demonstrated increasingly accurate results while driving down the need for labeled training data. Through transfer training techniques we have reduced timelines and cost. To put NV5’s deep learning technologies to work for you solving your target detection, feature extraction, and classification challenges, contact us today.
AUTOMATE ANALYTICS
Automate analytics with deep learning for fast, accurate results.
CASE STUDY
Taiwan uses ENVI Deep Learning to calculate arable lands lost to development.
SHORT VIDEO
See how ENVI Deep Learning can be used for environmental applications.