When talking to our users and customers, we quite often encounter the question: 'Yes, yes, it is amazing, but how accurate is your service?'. Simply put, this is not a straighforward question to answer. Mapillary relies on the geographic positions that come along with photos, and in general can only be as accurate as those coordinates are.

How do we know if the photos and objects are accurately put on the map? (Example from the JOSM editor)

To come up with a reliable measure of accuracy, we would need to compare each photo in our system to their true location. Without this ground truth, it is very hard to estimate the offset, which is what generates the issues of accuracy.

An easy approach would be to assume that photos should be on roads and compare the offset between the underlying road system and the photos that have been uploaded to Mapillary. A problem with this case is that it does not consider one of the big advantages of Mapillary - to be able to take photos from anywhere.

For example, a photo sequence can be taken on the sidewalk (or better yet, far from the sidewalk on an open field) and another sequence facing the same direction can be near the median of the road. In this exreme case, an algorithm comparing photos to roads would yield an inaccurate scenario. However, in reality these are just different use cases of Mapillary.

GPS Positioning

To understand the reasons behind inaccurate photo sequences, we first need to look at smartphone geolocation. Modern smartphones integrate different systems to determine their location on the Earth's surface:

• (Assisted) GPS
• WiFi positioning
• Cellular positioning

While cellular triangulation and Wifi positioning can give you an approximate location, in most cases the most accurate position can be achieved when the GPS receiver is turned ON. Unfortunately, there are many factors that influence this accuracy.

For example, GPS signals have to travel thousands of kilometers through different conditions before reaching your phone. While travelling this long distance, some delay is inevitable, which directly affects your positional accuracy. The constellation above you (i.e. the angle between your device and the satellites) as well as the number of satellites seen vary from time to time resulting in different levels of accuracy.

You can expect better results when out on an open field than being around tall buildings (a.k.a. urban canyons) or under heavy tree canopy that blocks the sky from your device. Placing your phone inside a car also results in lower accuracy as the metallic frame of the car can block signals.

The perfect treeless, buildingless conditions for getting great GPS signal - hardly an everyday for most people

These factors above affect each receiver the same way, but another set of errors come from your device itself. Different phones and receivers are equipped with different chipsets with varying sensitivity.

Common scenarios

It's easier to understand all of the above when you look at some maps. The first map showcases an urban canyon with skyscrapers blocking clear views of the sky in Downtown San Francisco. Here your device might have trouble finding enough satellites for precisely locating itself. It's obvious that photo sequences (i.e. their GPS tracks) can get messy due to a lot of uncertanity in the positioning and signal loss (which eventually prevents the phone from taking photos).

The other side of the figure illustrates a more ideal case, still in an urban environment west from downtown, but photo sequences are now smoothly following the roads. With clear views of the sky, devices were able to acquire accurate locations for each photo, resulting in a clean map.

One city, different conditions: skyscraper area on the left, clear skies on the right (San Fransisco)

The iOS vs Android debate won't get resolved by Mapillary. Most devices seem to perform similarly when conditions are similar. However, the Android market is so diverse that the phones can't be treated as a homogenous category. Some manufacturers create excellent devices with accurate positioning, while for others it proves to be difficult to do so. In any case, a modern smartphone is capable of providing an accuracy of around 3m - if conditions allow. This level of accuracy is suitable for a lot of geospatial applications.

Virtually no difference between the accuracy of iOS and Android on average

To improve positioning, it's a good practice to experiment with your setup. For example, you can try different positions for your windshield mount. If you have an external GPS receiver, it is a good idea to mount that outside of the car and connect it with your phone. You can do the same with action cameras with built-in GPS receivers to increase their sensitivity.

Correcting the photo collection

Our awesome Computer Vision team uses a technique called Structure from Motion (SfM) to find connections between photos and reconstruct their surroundings. This technique is also used to estimate the correct positions of the photos along with the camera angles. These corrections often allow to place misaligned photos to their original positions, allowing for other systems like object recognition to extract features where they would appear in reality.

In general, the more photos you feed into the system from an area, the better the results can be. An SfM-corrected sequence is highlighted below. You can easily notice the original photo locations (red) swinging from one side of the road to the other, ocassionally crossing buildings. If there are many overlaps between those photos, these corrections can be very promising. Green dots symbolize the corrected locations - now fully aligned with roads. Eventually this process results in smoother transitions between photos and better object recognition.

The more photos, the better the accuracy - Structure from Motion doing its corrections

Another thing that can be done to improve accuracy is to manually edit the sequences. Some members of our community are already doing this before uploading to the servers. This step is also useful for indoor mapping, but a more general use-case is to correct messy sequences, like the one shown on the first map. Also, with the new editor on the Mapillary website, it is now easier to edit existing photos by just dragging them to their real position. Every edit you make improves accuracy and the quality of our service.

As you can see from all the above, we heavily rely on all the different kinds of devices our community uses, so there is no easy way for us to quanitfy our "accuracy". Being a crowdsourced project, these noises in the data are normal. However, with the evolution of positioning services we do expect some major improvements in the coming years, all leading towards a better Mapillary experience.

So get capturing, get editing, and let's improve the accuracy of Mapillary together!

/Levente & the Mapillary team