When the real, 3D world is captured with cameras, it is projected onto 2D images. Depending on the camera, this process can introduce some errors. One of them is radial distortion, which causes straight lines in the real world to look bent in the 2D image.

Using computer vision, it is possible to compensate for this issue, and this is something we have been working on to improve the viewing experience of Mapillary images. With the recent update of the MapillaryJS library, we have started undistorting all image textures. Now, straight lines in the real world will be straightened when rendered in the Mapillary viewer.

Lines that are straight in the real (3D) world, e.g. corners of houses, look bent in this distorted action camera image

Radial distortion is different for every camera. To reconstruct a good 3D model from 2D images, as we do on Mapillary, we need to determine the distortion. Mapillary uses OpenSfM as the technology for 3D reconstruction, which calculates radial distortion parameters during the process. These parameters are then used by MapillaryJS to undistort the image texture when rendering the 3D scene in the MapillaryJS viewer. As a result, lines that are straight in the real world will now also be straightened in MapillaryJS.

The same action camera image as above, now undistorted by MapillaryJS (lines are straightened)

Besides making images look more realistic, undistortion also has several other benefits. In general, image pixels are now correctly related to 3D positions in the viewer. This results in better alignment between the pixels of different images and, therefore, smoother transitions and less artifacts when navigating between images.

It also means that map markers will be more precisely positioned relative to the image pixels when added or placed in the viewer (see an example of MapillaryJS placement tools here).

You can try both the previous and the new version of MapillaryJS below to see the difference. Change the render mode by clicking the buttons and pan around to see how the images in the new version have been undistorted.

LetterboxFill

Try the previous MapillaryJS version with distorted images (top) and the new version with undistortion (bottom). Click the Fill or Letterbox buttons to change the rendering mode (filling the viewer window or showing the whole image).

As can be seen in the example above, image borders are not straight anymore after undistorting the image. While this does not cause any problems, it may take some time to get used to.

Undistortion also means that when you tag objects in Mapillary images, the tag rectangles will be rendered with bent sides if the original images are distorted. This occurs because the rectangle shape is represented with straight lines on the distorted 2D image and has to be undistorted when rendered in MapillaryJS.

As you may notice, there is room to improve the radial distortion estimation near the edges of the images. We will keep working on OpenSfM to achieve that, which will result in better straightening of lines in those parts of the images in the future.

The update has been implemented on the Mapillary web and will soon be available in our mobile apps as well. You can also integrate the new version of MapillaryJS in your own applications—take a look at the MapillaryJS examples page to get started.

/Oscar