Tech Sneak Peek: Navigating Your Photos in 3D

Ever since we started the OpenSfM project, we have been exploring the potentials of the 3D reconstructions built using photos contributed by Mapillary users.

We are using a technique called 'Structure from Motion' to reconstruct the relative positions of cameras and the 3D structure of the scene using only photos. See this blogpost for an example.

As a first experiment, we want to improve the transitions from one photo to another by using the scene structure. We generate a triangular mesh to approximate the scene geometry and map the images onto it. This way, we can render the approximate 3D geometry while we move from one camera to another, giving a better perception of the motion.

Here are two example sequences: a forward motion on a street and a sideway motion along the water.

You can move either by using the navigation buttons or by using the arrow keys on your keyboards. Press ESC or click Fly to enter the Matrix mode. Inside the Matrix, you can see all the cameras and 3D point clouds. Double-click on a camera to see through it.

As you can see we are getting nice and smooth transitions between photos. The occasional artifacts are to be expected since we use only an approximate scene geometry, so we take that as a good start.

Another goal we have is to enable natural navigation of the photos in 3D space. We rely on the superior accuracy of the camera positions recovered from SfM compared to those from GPS and compass. To experiment with the navigation, we have reconstructed an area with approximately 800 photos in Malmö, Sweden (will take some time to load). As you start navigating, you may notice that these are photos from different users and in different seasons. This is exactly what we are aiming at - to build a global fabric of photos both in space and in time!

While the results are still work in progress, we would really like to share these with you and hear your feedbacks (Twitter or Github)!

/Yubin and Pau

Continue the conversation