Update: the Verifier tool is now part of Verification projects, which enable you to review machine-extracted map data in your area of interest by adding human validation to the workflow. Learn more in the announcement and refer to our Help Center for the latest information.

A while ago we added a feature to the Mapillary web that enables you to visualize object detections produced by our AI technology. While our results compare well against benchmarks, there are always those few tough cases that stand between us and 100% accuracy.

People have been asking whether there is anything they could do to help us improve our algorithms. Today, we can reply to this with a yes. We’ve built a tool that lets anyone help improve Mapillary data accuracy by verifying the automatic detections from our AI algorithms. The Verifier tool is part of our Human in the Loop concept, which in short means that we’re including human feedback into the learning loop of the machine, thus speeding up the learning process needed for improved data accuracy.

The Verifier tool

The first release of the Verifier is a simple tool for providing a true/false verification to traffic sign detections. You will be working on a task regarding a specific object and shown detections that have been classified as this object by our algorithms. Your task is to approve or reject the detection for each case.

Here is the really fun part: the Verifier tool actually functions as a game. As you approve and reject, you will see two numbers increase. One is simply counting how many cases you have reviewed. The other one is your game score that will give you the chance to shine in the task leaderboard.

The score is related to the purpose of the tool, which is data verification. To ensure high data quality, each detection needs to be correctly verified twice, i.e. by you and another person. That is when you also get a point in the game—when an object that you verified has received the same verification from someone else.

You can keep an eye on your score and rank in the leaderboard right next to your work area. You can also obtain the link to the specific task from within the tool so that you can easily share it with others and invite them to compete with you.

For more information and options, please head over to our Help Center.

The past and the future

For those who have been with Mapillary for a couple of years, this concept might sound familiar. Mapillary used to have something called the Traffic Sign Game which also enabled people to give feedback to traffic sign detections. However, the way the game was built could not keep up with the amount of data and processing on the Mapillary system, which is why we needed to deprecate it.

The Verifier tool is the first step along the way in our new approach to incorporating human feedback. Bit by bit, we integrate additional functionality and build new tools to create a comprehensive system to improve our computer vision technology. The way that the back end has been built is different from how it used to be in the old Traffic Sign Game. A lot of focus has been put on making sure that it is able to scale and handle the ever-growing amount of data.

The next step with the Verifier tool is to provide a user interface to choose the type of object you want to verify and the geographic area that the detections would be from. This way you can, for example, create a task for verifying stop sign detections in Dublin. The task will have a unique URL so you’ll have the chance to share it via the link and get other people to help you with your task.

We’re keen to see the leaderboards take off and look forward to making use of the verifications that you create with this tool, so that we can provide more accurate map data to everyone from OpenStreetMappers to HERE Map Creators to cities and automotive companies. Ready, set, go!

/Andrew