To protect privacy, we detect and blur faces and license plates in images on Mapillary. The goal is to blur all identifiable faces and license plates. At the same time, we need to preserve important information in the images for mapping. This makes it a challenging problem.

Faces and license plates can be very small, with various visual appearances (e.g. different skin tones, license plates that differ by country and/or typeset), partially occluded or behind transparent material like windows, or similar to other non-privacy relevant objects like text on traffic signs or billboards where blurring should be avoided.

Mapillary’s automatic privacy blurring around the globe (click on an image to view in full resolution)

In order to tackle these challenges and automatically blur faces and license plates in hundreds of millions of images, we make use of computer vision technology. Previously, we used more traditional detectors, where the pixel patterns as input (i.e. combinations that are characteristic to faces and license plates) are learned only from local patches in images.

In our new blur algorithms, we make use of semantic segmentation, the task of assigning a semantic label (e.g. face, license plate, and background) to each pixel in an image. Unlike for traditional detection methods, the context of the objects is naturally encoded during the algorithm’s learning process, leading to much higher accuracy. For instance, by learning the coexistence of a vehicle and a license plate, the neural network is able to differentiate a license plate from a line of text on a road sign.

To train and evaluate our blurring algorithms, we generated a new dataset. This dataset consists of more than 1.7 million manually blurred images by our community and more than 27 thousand images from an extended version of the Mapillary Vistas Dataset where we added annotations of faces and license plates.

The resulting privacy blurring model is able to detect 99.9% of all identifiable faces and license plates while wrongly blurring only 0.078% of the pixels. This means that on average, we miss just one out of 1,000 identifiable faces and license plates.

To make sure we can use this algorithm on the bulk of images that flows into Mapillary every day, we optimized this model for large-scale processing. As a result, we were able to process all Mapillary imagery with this super-accurate detector in just three weeks.

Developing the technology

To train and evaluate our blurring model, we collected a new large-scale dataset which contains more than a million annotations for faces and license plates in more than 1.7 million street-level images all over the world. This dataset exists largely thanks to our community who have been manually fixing blurs (adding missing ones and removing false ones) on Mapillary images since the beginning.

In particular, our community has edited blurs on 1,742,967 images, adding 962,569 rectangles to be blurred. On top of that, we added more images to our Mapillary Vistas Dataset, annotating 40,081 faces and 83,549 license plates in 27,131 images. Combining the annotations by our community with our Mapillary Vistas privacy extension, we can make use of more than 1,086,199 annotated faces and license plates to advance our privacy protection algorithms.

Many thanks to our community! All this data together is helping us to train and evaluate our blurring models and get the results we can now experience on all Mapillary images.

Manual blurs by the Mapillary community (click on an image to view in full resolution)

Examples from the Mapillary Vistas, Privacy Extension (click on an image to view in full resolution)

As a baseline method, we chose our novel semantic segmentation approach which is top-ranked in scientifically recognized benchmarks for semantic understanding of street scenes. This leading model is optimized for accuracy at a fixed image length, not for fast processing of high-resolution images. So we adapted it to run at full resolution to better recover small faces and license plates, and optimized the runtime for large-scale processing.

Last, by using NVIDIA’s TensorRT library and AWS P3 instances equipped with NVIDIA Tesla V100 graphic cards, we accelerated the computation even more during inference, i.e. applying the model to previously unseen images.

Best privacy blurring for your street-level imagery

With the new GDPR laws to be enforced in the EU on 25 May, privacy is becoming more into focus than ever, and that includes images that different companies and organizations own. On Mapillary, your imagery will be blurred with the best available technology so you don't have to worry about privacy. If you're interested in bringing your imagery to Mapillary, get in touch!

/Gerhard