Adding fire hydrants to OpenStreetMap using 360 degree imagery from the City of Detroit on Mapillary

Origins of Deriviste

In October 2018, a new OpenStreetMap tool called Deriviste was introduced by Richard Fairhurst. This tool presented the user with a way to directly create OSM data from user images on Mapillary, including by clicking directly on the image. Thanks to an experimental feature in the open-source MapillaryJS library, Deriviste meant that a click in the Mapillary image was translated to a point on the map. Richard borrowed some code from the OpenStreetMap iD editor in order to allow searching for OSM tag presets, then used simple user authentication to allow submission of the newly created data as an OSM changeset.

The tool received mixed reviews on Twitter, with many praising the ease of adding new data derived from photos, while others pointed out the erratic behavior of the Mapillary viewer in regards to map position, resulting in unreliable accuracy. The truth is that neither this tool nor the Mapillary feature behind it are intended to function with perfect precision.

Instead, these are meant to enable the user to add new data to OSM with an enhanced ability to extract data, and the ideal environment for tinkering with that data in both the image and the map views. The tool also isn’t ideally integrated elsewhere (such as into OSM iD), but instead is a very simple, straightforward application of its own with one purpose: deriving new data from Mapillary imagery.

I wish ID could integrate Deriviste tool that lets us edit by simple clic on pictures. https://t.co/9d4rjzUxp1 This way of editing would be welcome for non mappers. This is in my idea a really innovant& intuitive way to create POI.
— J-Louis ZIMMERMANN (@JLZIMMERMANN) December 5, 2018

While the original tool was a simple experiment on Richard’s part, I decided to make an update to build off of my own extensive experience with making experimental Mapillary tools for dozens of use cases. One of the key additions I made was to show the current marker—the one being edited on the map—in the Mapillary viewer. This means that the map and the viewer both reflect the context of newly created data, even when clicking and dragging the point to adjust its position.

What Deriviste brings to the table

Before you get started with Deriviste, I first want to introduce a list of the complete features as of today:

• View OpenStreetMap in a Leaflet web map
• Search for your area of interest with the Leaflet Geocoder (Nominatum based)
• View Mapillary image locations in green
• Mapillary image layer disappears at close zoom (helpful for clarity, even if not intentional)
• Single click the green Mapillary layer to load the image at that location
• Double click the Mapillary viewer to add a point to the map
• Observe the red marker and line on the map to estimate image-map correspondance
• Click and drag the point on the viewer or on the map to add to a new location
• Use the search box to find a preset tag, such as fire hydrant or bicycle parking
• Manually add new tags to the table at bottom right
• Login to your OSM account and click submit to add a changeset
• All changesets are tagged as using Deriviste to create data, with Mapillary as a source

Inside the image

It’s crucial to understand the limitations not only of this tool, but of the cameras, GPS, and even the users capturing the photos. There are many factors that can erode or complicate the relationship of a Mapillary image and the map that accompanies it. When the Mapillary viewer has the marker component enabled, it can receive a longitude and latitude in order to display that in the image as a marker—also known as placement tools.

Mapillary makes a rough estimation of the ground plane in each image. Mapillary then computes a corrected geographic coordinate and camera angle, based on the image’s relationship with other nearby images. Based on this, we relate the pixel over which the mouse hovers to a longitude and latitude. If the mouse position can be matched to a geographic point on this rough estimation of the ground plane, within a 100 meter radius of the image location, then we display a corresponding position in both the map and the viewer. If the position of the mouse hover does not match the ground plane within that 100 meter radius, then it is not possible to place a marker.

If an image is 640 by 480 pixels, then the mouse may hover at a pixel coordinate of (250,250) which corresponds with a map coordinate (EPSG:4326) of (-115.176,36.116). This means you are looking at Caesar’s Palace in Las Vegas in the image, with the mouse hovering over it, and the map is displaying a marker at this position as well. If you hover on the map over Caesar’s Palace while viewing a nearby Mapillary image, the same effect happens in reverse.

Matching the image to the map

How does the Mapillary viewer know where in the image this geographic coordinate should be? To start with, every image on Mapillary has a geographic coordinate and a camera angle, called the metadata of the image. With this we know where the image is located and in what direction it is facing. The next step involves what is currently a best guess: where the mouse cursor is in regards to the horizon of the image.

Generally, when you hover on the image, we are assuming that as the mouse moves upward it approaches the horizon, and when you hover on the top half of the image the cursor has moved over the horizon based on the image’s position. Moving to the left and right, we use the image’s focal length to estimate the total width on the map where the cursor would be visible.

Limitations on precision and accuracy

The resulting precision is highly dependent on the image’s geotag also being precise and accurate, as well as the camera angle being correct. Due to GPS interference in urban areas, variation between GPS in different devices, and different reactions of the device to camera tilt and yaw, the GPS position and camera angle can often be less than perfect. This means the marker component and placement tools in MapillaryJS can have a skewed effect.

As an example, clicking on a fire hydrant on a street corner in a Mapillary image may add a point on the map that is in the middle of the street. This is due to the imagery not quite matching the map, and it’s often that the map is wrong as much as the image metadata is wrong—sometimes it’s both. This can be a similar effect to offset problems in satellite images, but with more complexity.

How Deriviste should be used

With that out of the way, let’s now consider how this tool should be used. What you should not expect:

• clicking on an image to perfectly line up with the map
• clicking on the map to perfectly line up with the image
• similar alignment when clicking and dragging

Deriviste is all about heuristics: trial and error, estimation, and a quick way to get data from images onto the map before making some adjustments. Deriviste makes it easy to start inventorying and adding data, but relies on you as the user to use a keen eye and common sense to correct the data before submitting it to OSM. It should be treated the same as when you add data that is hand-drawn from Field Papers, or as data collected using mobile apps like Go Map!! or Vespucci.

In these cases, the first place the data is collected is in the eye of the mapper, then it is translated onto a map with some relative placement, and then the user adjusts the position to make sure it looks correct based on other nearby data, satellite or aerial imagery, and local knowledge. In Deriviste, the user is the filter for data quality, just like everywhere else.

A practical exercise

Next, let’s do a quick practice round. We’ll start by opening Deriviste at https://osm.cycle.travel/deriviste/. When opening the tool, we see a world map, and that we are not logged in. We don’t need to log in until we’re ready to submit a changeset, so as a first step let’s choose a place to edit. Where better than sunny Santa Monica, California?

Once I’ve typed in Santa Monica, the map will zoom me to the city. We can now see the green lines and points representing Mapillary sequences and images. Below the geocoder search icon, you can see a layer switch option—use this to get a satellite imagery context.

Let’s choose an image in downtown Santa Monica—single click anywhere where the map is green! The map will now show a location icon in orange and yellow, indicating the current camera angle and position. In the viewer, we see the image. On the right side on my selected image, we see a fire hydrant and a few trash bins. The trash bins seem dynamic, so it probably isn’t a good idea to map them on OSM. The fire hydrant, however, will be a welcome addition.

Hovering on the image, a red halo shows my mouse position. The map also shows a red circle, plus a red line showing the camera angle and how far along its axis my cursor moves. Zooming in, the Mapillary layer disappears and we can check that there isn’t already a fire hydrant on the map here.

It looks all clear. We see the image is on the street, and there is a lot of grey space where there should also be a sidewalk, the curb, and grass. The red point on the map appears to be far away from the street itself, rather than right up against it as in the image, but we should remember that in OSM streets are lines and not polygons. Try switching to a satellite basemap and see how the image hover compares to the map. Next, double-click on the fire hydrant in the image, and we’ll see a red marker appear on the map.

Once the marker is on the map, we can search for the appropriate tag below the Mapillary viewer. I searched for hydrant saw fire hydrant as an option, and clicked it. Now it’s populated on the Edit tags table. We could also manually add other details such as colour=yellow. In case of a mistake, click the red Delete selected node button. To adjust the position, click and drag on the circle inside of the marker in the Mapillary viewer, or click and drag the marker on the map.

Moving down the street, we can find another fire hydrant and repeat the above steps. Once we’ve added the second fire hydrant, this latest one will be a red marker on the map, and the previous will be blue—red indicating the current selection. The first fire hydrant will no longer be visible in the viewer if it is not selected. In order to delete a marker it must also be selected.

When finished, type in your OSM username and password, and click the blue button to upload. A dialog will request a required changeset comment, and the changset will then be submitted with reference to Mapillary as the source and Deriviste as the editor.

That’s it! You’ve learned how to use Deriviste. Try it out in areas with different types of imagery, such as 360 degree images in Detroit and Amsterdam. If you have ideas and feedback, feel free to open an issue on the Deriviste Github repository, and as always the community is happy to see more developers contribute to the project.

/Christopher Beddow, Solutions Engineer

Update: For more information on Deriviste, check out Richard Fairhurst’s interview with the OpenCage Data Blog.