detect mis-registered OSM roads (by default in Delaware) (trailbehind#46

) * fix odd issue with tflearn/pil imports * fix radnomization issue too * undo test line changes * fix flake complaint * add a method to bundle of image, label, and location data for the UI/database * didnt need that dockerfile change * pep cetera * code to enumerate false negatives and false positives for all naip tiles * this seems to make list properly now, let's runn it on gpu * running this on gpu with an 80% accurate net to see how many false pos/negs that generates in all 170+ delaware NAIPs * typo * pep * more pep * use normal minibatching technique * add terrapattern to README * switch computers * correct false-pos/neg loops * pep8 * set defaults that do the delaware false positive analysis quickly as possible * update README to say what the script does by default * typos and such in README * fix bug from line spacing * add development seed's project to the readme
hzy-zg · May 31, 2016 · 92983f0 · 92983f0
1 parent 3d6aba9
commit 92983f0
Show file tree

Hide file tree

Showing 6 changed files with 156 additions and 61 deletions.
diff --git a/README.md b/README.md
@@ -1,37 +1,39 @@
 # DeepOSM
 
-Detect roads and features in satellite imagery, by training neural networks with OpenStreetMap (OSM) data. This code lets you:
+Classify roads and features in satellite imagery, by training neural networks with OpenStreetMap (OSM) data. DeepOSM lets you:
 
 * Download a chunk of satellite imagery
 * Download OSM data that shows roads/features for that area
 * Generate training and evaluation data
+* Display predictions of mis-registered roads in OSM data, or display raw predictions of ON/OFF
 
 Running the code is as easy as install Docker, make dev, and run a script. 
 
 Contributions are welcome. Open an issue if you want to discuss something to do, or [email me](mailto:[email protected]).
 
 ## Default Data/Accuracy
 
-By default, DeepOSM will download the minimum necessary training data, and use the simplest possible network.
+By default, DeepOSM will analyze about 200 sq. km of area in Delaware. DeepOSM will
 
-* It will predict if the center 9px of a 64px tile contains road.
-* It will only use the infrared (IR) band, not the RGB bands.
-* It will be about 65% accurate, based on how the training/test data is constructed.
-* It will use a single fully connected relu layer in [TensorFlow](https://www.tensorflow.org/).
+* predict if the center 9px of a 64px tile contains road.
+* use the infrared (IR) band and RGB bands.
+* be 75-80% accurate overall, training only for a minute or so.
+* use a single fully-connected relu layer in [TensorFlow](https://www.tensorflow.org/).
+* render, as JPEGs, "false positive" predictions in the OSM data - i.e. where OSM lists a road, but DeepOSM thinks there isn't one.
 
-![NAIP with Ways and Predictions](https://pbs.twimg.com/media/CiZVcu8UgAIYA-c.jpg)
+![NAIP with Ways and Predictions](https://pbs.twimg.com/media/Cjk6fADUYAE0wvh.jpg)
 
 ## Background on Data - NAIPs and OSM PBF
 
-For training data, DeepOSM cuts tiles out of [NAIP images](http://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/), which provide 1 meter per pixel resolution, with RGB+infrared data bands.
+For training data, DeepOSM cuts tiles out of [NAIP images](http://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/), which provide 1-meter-per-pixel resolution, with RGB+infrared data bands.
 
-For training labels, DeepOSM uses PBF extracts of OSM data, which contain features/ways in binary format, which can be munged with Python.
+For training labels, DeepOSM uses PBF extracts of OSM data, which contain features/ways in binary format that can be munged with Python.
 
 The [NAIPs come from a requester pays bucket on S3 set up by Mapbox](http://www.slideshare.net/AmazonWebServices/open-data-innovation-building-on-open-data-sets-for-innovative-applications), and the OSM extracts come [from geofabrik](http://download.geofabrik.de/).
 
 ## Install Requirements
 
-DeepOSM has been run successfully on both Mac and Linux (14.04 and 16.04). You need at least 4GB of memory.
+DeepOSM has been run successfully on both Mac (10.x) and Linux (14.04 and 16.04). You need at least 4GB of memory.
 
 ### AWS Credentials
 
@@ -68,14 +70,14 @@ make dev
 
 Inside Docker, the following Python scripts will work. This will download all source data, tile it into training/test data and labels, train the neural net, and generate image and text output. 
 
-The default data is eight NAIPs, which gets tiled into NxNx4 bands of data (RGB-IR bands). The training labels derive from PBF files that overlap the NAIPs.
+The default data is six NAIPs, which get tiled into 64x64x4 bands of data (RGB-IR bands). The training labels derive from PBF files that overlap the NAIPs.
 
 ```
 python bin/create_training_data.py
 python bin/train_neural_net.py
 ```
 
-For output, it will produce some console logs, and then JPEGs of the ways, labels, and predictions overlaid on the tiff.
+For output, DeepOSM will produce some console logs, and then JPEGs of the ways, labels, and predictions overlaid on the tiff.
 
 ### Jupyter Notebook
 
@@ -105,38 +107,41 @@ Images](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.232.1679&rep=re
 best/recent paper on doing this, great success with these methods
 * Similar Efforts with OSM Data
     * [OSM-Crosswalk-Detection](https://github.com/geometalab/OSM-Crosswalk-Detection) - uses Keras to detect crosswalks, a class project (Fall 2015)
-    * [OSM-HOT-ConvNet](https://github.com/larsroemheld/OSM-HOT-ConvNet) - attempted use for disaster response, author thinks it's only 69% accurate at pixel level (Fall 2016)
+    * [OSM-HOT-ConvNet](https://github.com/larsroemheld/OSM-HOT-ConvNet) - attempted use for disaster response, author thinks it's only 69% accurate at pixel level (fall 2016)
+    * [Terrapattern](http://www.terrapattern.com/) - (spring 2016) - detect similar images, such as pools, boat wakes, or other patterns journalists/researchers might be interested in - Carnegie Mellon researchers, open source
+    * [Skynet Data](https://github.com/developmentseed/skynet-data) - (spring 2016) - data pipeline for machine learning with OpenStreetMap
+
+
 * [Parsing Natural Scenes and Natural Language
 with Recursive Neural Networks (RNNs)](http://ai.stanford.edu/~ang/papers/icml11-ParsingWithRecursiveNeuralNetworks.pdf)
-* Links from the Tensorflow site
+* Background on Neural Networks and Deep Learning
     * [MNIST Data and Background](http://yann.lecun.com/exdb/mnist/)
     * all the other links to Nielsen’s book and [Colah’s blog](http://colah.github.io/posts/2015-08-Backprop/)
-* Deep Background
     * [original Information Theory paper by Shannon](http://worrydream.com/refs/Shannon%20-%20A%20Mathematical%20Theory%20of%20Communication.pdf)
 
 [Also see a work journal here](http://trailbehind.github.io/DeepOSM/).
 
 ### Papers - Relevant Maybe
 
-* [Uses a large window to improve predictions, trying to capture broad network topology](https://www.inf.ethz.ch/personal/ladickyl/roads_gcpr14.pdf)
+* [Uses a large window to improve predictions, trying to capture broad network topology.](https://www.inf.ethz.ch/personal/ladickyl/roads_gcpr14.pdf)
 
-* [Automatically extract roads with no human labels. Not that accurate, could work for preprocessing to detect roads in unlab](https://www.researchgate.net/publication/263892800_Tensor-Cuts_A_simultaneous_multi-type_feature_extractor_and_classifier_and_its_application_to_road_extraction_from_satellite_images)
+* [Automatically extract roads with no human labels. Not that accurate, could work for preprocessing to detect roads.](https://www.researchgate.net/publication/263892800_Tensor-Cuts_A_simultaneous_multi-type_feature_extractor_and_classifier_and_its_application_to_road_extraction_from_satellite_images)
 
 ### Papers - Not All that Relevant
 
 * [Uses map data and shapes of overpasses to then detect pictures of the objects? Seems like a cool paper to read if it was free.](http://dl.acm.org/citation.cfm?id=2424336)
 
-* [New technique for classification of sub-half-meter data into different zones](http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=6827949&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D6827949)
+* [New technique for classification of sub-half-meter data into different zones.](http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=6827949&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D6827949)
 
-* [Couldn't access text, focused on usig lidar data](http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=6238909&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D6238909)
+* [Couldn't access text, focused on using lidar data.](http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=6238909&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D6238909)
 
 * [Proposes a way to extract network topology, and maybe this can be used as a post processor?](http://www.cv-foundation.org/openaccess/content_cvpr_2013/html/Wegner_A_Higher-Order_CRF_2013_CVPR_paper.html)
 
-### Papers  to Review
+### Papers to Review
 
 Recent Recommendations
 
-* FIND - have you seen a paper from a few years ago about estimating osm completeness by comparing size of compressed satellite images vs number of osm nodes
+* FIND - have you seen a paper from a few years ago about estimating OSM completeness by comparing size of compressed satellite images vs number of osm nodes
 
 * READ - this presentation on using GPS traces to suggest OSM edits (Strava/Telenav): http://webcache.googleusercontent.com/search?q=cache:VoiCwRHOyLUJ:stateofthemap.us/map-tracing-for-millennials/+&cd=3&hl=en&ct=clnk&gl=us
 

diff --git a/bin/create_training_data.py b/bin/create_training_data.py
@@ -33,11 +33,11 @@ def create_parser():
                         help="how much data to allocate for training. the remainder is left for "
                              "test")
     parser.add_argument("--bands",
-                        default=[0, 0, 0, 1],
+                        default=[1, 1, 1, 1],
                         nargs=4,
                         type=int,
                         help="specify which bands to activate (R  G  B  IR)"
-                             "--bands 0 0 0 1 (which activates only the IR band)")
+                             "--bands 1 1 1 1 (which activates only all bands)")
     parser.add_argument(
         "--label-data-files",
         default=[
@@ -57,10 +57,9 @@ def create_parser():
                         help="turn on this arg if you don't want to get NAIPs in order from the "
                              "bucket path")
     parser.add_argument("--number-of-naips",
-                        default=5,
+                        default=6,
                         type=int,
-                        help="set this to a value between 1 and 14 or so, 10 segfaults on a "
-                             "VirtualBox with 12GB, but runs on a Linux machine with 32GB")
+                        help="the number of naip images to analyze, 30+ sq. km each")
     parser.add_argument("--extract-type",
                         default='highway',
                         choices=['highway', 'tennis', 'footway', 'cycleway'],

diff --git a/bin/train_neural_net.py b/bin/train_neural_net.py
@@ -8,8 +8,8 @@
 # src.training_visualization must be included before src.single_layer_network,
 # in order to import PIL before TFLearn - or PIL errors tryig to save a JPEG
 from src.training_visualization import render_results_for_analysis
-from src.single_layer_network import train_on_cached_data, predictions_for_tiles
-from src.training_data import CACHE_PATH
+from src.single_layer_network import train_on_cached_data, predictions_for_tiles, list_findings
+from src.training_data import CACHE_PATH, load_training_tiles
 
 
 def create_parser():
@@ -20,16 +20,19 @@ def create_parser():
                         type=int,
                         help="tile the NAIP and training data into NxN tiles with this dimension")
     parser.add_argument("--bands",
-                        default=[0, 0, 0, 1],
+                        default=[1, 1, 1, 1],
                         nargs=4,
                         type=int,
                         help="specify which bands to activate (R  G  B  IR). default is "
-                        "--bands 0 0 0 1 (which activates only the IR band)")
+                        "--bands 1 1 1 1 (which activates all bands)")
+    parser.add_argument("--omit-findings",
+                        action='store_true',
+                        help="prevent display of predicted false positives overlaid on JPEGs")
     parser.add_argument("--render-results",
                         action='store_true',
                         help="output data/predictions to JPEG")
     parser.add_argument("--number-of-epochs",
-                        default=100,
+                        default=5,
                         type=int,
                         help="the number of epochs to batch the training data into")
     parser.add_argument("--neural-net",
@@ -46,7 +49,23 @@ def main():
     with open(CACHE_PATH + 'raster_data_paths.pickle', 'r') as infile:
         raster_data_paths = pickle.load(infile)
     test_images, model = train_on_cached_data(raster_data_paths, args.neural_net, args.bands,
-                                              args.tile_size)
+                                              args.tile_size, args.number_of_epochs)
+    if not args.omit_findings:
+        for path in raster_data_paths:
+            print path
+            labels, images = load_training_tiles(path)
+            if len(labels) == 0 or len(images) == 0:
+                print("WARNING, there is a borked naip image file")
+                continue
+            false_positives, false_negatives, fp_images, fn_images = list_findings(labels, images,
+                                                                                   model)
+            path_parts = path.split('/')
+            filename = path_parts[len(path_parts) - 1]
+            print("FINDINGS: {} false pos and {} false neg, of {} tiles, from {}".format(
+                len(false_positives), len(false_negatives), len(images), filename))
+            render_results_for_analysis([path], false_positives, fp_images, args.bands,
+                                        args.tile_size)
+
     if args.render_results:
         predictions = predictions_for_tiles(test_images, model)
         render_results_for_analysis(raster_data_paths, predictions, test_images, args.bands,