This is a guest post from Mapillary user Chris McNally on using video files for contributing to Mapillary. Until there is video upload support this workflow will help you process video files to create photos that you can submit to Mapillary. Thanks a lot to Chris for putting this together and posting it here!
Using a video file to create photos for Mapillary upload
I’ve been using videos recorded while riding my bike to create stills for upload to Mapillary. I have a Nabi Cube HD front facing action camera and a rear facing safety camera, the Fly6.This tutorial describes how I did it, using VLC to split the video into frames, then along with a GPS track file recorded at the same time, I used the Mapillary tools to add dates, and GpsPrune to view my images on a map, adjust times and add geo location. Finally, I use the Mapillary tools to add or change the direction/heading tag of the photos, which is important for images from the rear facing camera.
Tool of the mapping trade: Nabi HD, GlobalSat Bluetooth GPS and Fly6 rear facing bike camera.
The gear mounted on the bike
There is a similar tutorial by Craig Taverner for GoPro images.
In their case the camera had a timestamp, which was used to link the photos to the GPS track, with adjustment, since the GPS track and the camera would have slightly different times. If you have a camera with “time lapse mode” or “continuous shooting” like the GoPro or my Nabi, then then continue with this tutorial, but your job is easier, you can skip ahead to GpsPrune, it can display your images and GPS track on a map or on satellite imagery, adjust times if needed and write GPS data to the photos. However If you want to use stills extracted from a video, there are some additional steps.
When breaking a video into frames, there will be no timestamp, so you have to create Exif dates for the images that will line up with the GPS track. I use GpsPrune to view my gpx track against a satellite view or map tiles. This app lets me see the track, find track points and see the time of the track point. Then I can use scripts to write that date/time stamp to my photos. Once I have that, I use GpsPrune to view the photos on a map and add Lat and Lon info.
I performed all these steps on my Ubuntu desktop, however I think most of the tools would work on any platform.
To start you will need a gpx track recording app. I use MyTrails on my Android phone and tablet. It will save your tracks and upload them to Dropbox for convenient transfer to a pc. As you are recording your track you should be shooting the video for the source of the images. Just before you approach a landmark it would help to open the clock app on your phone and hold it in front of the camera for a few seconds. This will help you line up the photos with the track but is not mandatory. Another trick is to slow down, pause, or make a waypoint, which will be visible in the GPS track.
I would start with about 3 minutes of video, along some interesting route that has some easily identifiable landmarks, or streets and intersections you can identify. I picked the Belt Parkway in Brooklyn where a bike path runs along the waterfront with a highway next to it. I stopped at a pier and there was a large pedestrian overpass that I could see from the satellite view. These would help me align the photos to the GPS track points.
Break Video into Stills
The first step is to break up a section of your video into stills. For this I use VLC Media Player. On Ubuntu you go into the Tools menu --> Preferences, on the bottom left click the Show Settings “All” radio button, on the left scroll down to Video, Filters, and choose Scene Filter. Expand it and here you can put in the path to a folder which will hold your photos, I use /home/chris/glen/pics. This folder should already exist and you must specify the full path. Then enter the frame rate factor. The Fly6 and Nabi both record at 30 frames per second, so I enter 60 to get a photo every 2 seconds.
Press Save, but then go back into Filters, don't expand it but instead click the word Filters and look to the right panel. Make sure “Scene Filter” is checked and then click save again.
Using VLC, open the video you want to extract frames from. If you already had it open, you may need to open it again for the scene filter to take effect. Advance to the start of your segment and let it play. Photo files will start appearing in the specified folder. The first few might be from earlier in your video stream so delete them until the start of your segment. When you have the 300 or so photos you should hit stop in VLC. The photos generated do not have any Exif data yet.
Visualize your route on a map
Install GpsPrune. It’s a Java application, so it should run on any platform. I used the Ubuntu Software Center tool to install it. It uses Exiftool. If you are not using Linux, you may need to install this dependency separately. See their homepage for info. GpsPrune is feature rich app for visualizing and editing GPX files and writing GPS data to the photo Exif. It will display your GPS track on a map or display satellite imagery, view data about the track points, such as the date and time.
Start GpsPrune. Open your GPS track with File → Open. Your track will be displayed. Go into Setting → Set Map Background, and choose Mapnik to see your track on a map. If you want to use Mapbox Satellite view, you will have to add the Mapbox server as an option. Go into Settings → Set Map Background and click the Add New button. Create a new source named mapbox, and paste in the following into the URL
Check the Openstreetmap radio button, set the max zoom to 18. This will allow you to download imagery from the Mapbox server. In either case, it is a good idea to change your settings to save the tiles to disk you can cache the images and it will be faster in the future and be less of a burden on map servers.
The GpsPrune app. Entering the Mapbox tile server url for satellite imagery.
In GpsPrune you can zoom in and click on points and see the GPS info, such as the date and time. Note that the time is in the GMT timezone. That is what we will use later so you don’t have to make any timezone adjustment.
Now you need to look at your photos and attempt to match the time of the first photo with a GPS track point, or any photo in the set with a GPS track point, utilizing landmarks etc. Since I started at the pier, and my GPS settings in My Trails does not record when I pause, I can tell the exact point where I started moving again, leaving the pier. It’s easy to see the pier in satellite view. I delete any photos prior to that time so I can line up the first photo with a time and all subsequent photos will be 2 seconds apart. If you don’t have a good landmark at the start, find any sequence of photos in the middle and line it up with a GPS point. Now you have the timestamp for a single photo, count how many photos until the beginning of your sequence and subtract that number of seconds. You really don’t have to get the time perfect because you can shift all the times later.
Now that we have a time for the first photo, we’re ready to tag the photos.
Add dates using Mapillary tools
We need to add dates to the JPG files and we can do this with a Python script. Once the dates are added, we can use GpsPrune to correlate the photos to the GPS Track. Another option is to just use GpxExif to do all of it. GpxExif is a Ruby script, it does just what we need and more but it is not trivial to install. If you want to use it, I will include some instructions later. Instead I modified one of the Mapillary Tools scripts, and they were good enough to incl noude it in their package. To get the Mapillary Tools, do the following.
Fetch the scripts by cloning the Mapillary tools repository:
git clone https://github.com/mapillary/mapillary_tools.git
This will create a folder mapillary_tools.
You may need to install a few dependencies for the Mapillary tools. I did the following
sudo apt-get install python-setuptools sudo easy_install pip pip install exifread pip install upload_file
A comment in one of the Python scripts tells me we need to install a dependency, pyexiv2 & gpxpy.
sudo apt-get install python-pyexiv2
Then get gpxpy and build it
git clone https://github.com/tkrajina/gpxpy.git cd gpxpy sudo python setup.py build sudo python setup.py install
Use the add-fix-dates.py script, supply the directory of the images, the datetime of the first image and the number of seconds apart. This will add dates to help us correlate the photos to the GPS track using GpsPrune. When entering the date, use the GMT time, but enter it without any time zone info, as if it were a local datetime. Use a space between the date and time and use quotes around it.
cd mapillary_tools/python python add_fix_dates.py ~/glen/pics/ '2014-09-07 11:17:40' 2
Note my actual time was 7:17 a.m., but I entered the time in GMT, and GMT will be what you see in GpsPrune as well.
Once it is done, you can view the results in GpsPrune, make adjustments to the time and have GpsPrune correlate and write the lat and lon Exif tags to your photos.
Add photos to the Map using GpsPrune
Back in GpsPrune, open the Photo menu and click Add Photos.
Go back into the Photo menu and select Correlate Photos. A dialog appears. You do not need to adjust the times, so just click OK. If the OK button is disabled, try clicking preview then OK.
GpsPrune will display your photos as yellow dots along your track. Click on the yellow dot and your thumbnail will appear to the right. It can be hard to click on the yellow dots, so you can use the list of photos to the left and click on the file name. The red crosshairs will focus on the location of the photo and the thumbnail appears on the right. You can see the date of the photo and the date of the nearest track point.
Check if the photos are lined up to the map or off. You can click on the zoom tool near the photo and it will open in a new window. Resize the window to enlarge the photo if you need to, in order to help you locate it.The best landmark I could find was a large highway sign followed by a pedestrian overpass over the highway. Looking at my photos, I could see that my times were off by about 12 seconds. The images were 12 seconds later then the track points they should have corresponded to. I closed GPS Prune (Don’t save) and ran the add-fix-date.py script again with the new time. I subtracted 12 seconds from my first attempt.
python add_fix_dates.py ~/glen/pics/ '2014-09-07 11:17:28' 2
I then repeated the steps to open the track in GpsPrune, add the photos and correlate.
Once you are sure the photos line up to the track, save the Lat and Lon data to the photos by clicking on the Photo menu and selecting Save to Exif. It will take a little time.
Calculate Direction or Change direction for Rear Facing Camera
Your photos don’t have a direction Exif tag. You can use the Mapilary provided script interpolate_direction.py to fix this. I already told you how to install the Mapillary tools in the section Add dates using Mapillary tools above.
Now run the script to calculate your bearing assuming the camera is mounted front facing. This script uses the lat-lon of adjacent photos to calculate a heading. Supply the directory path of the pics to update
cd ~/mapillary-tools/python python interpolate_direction.py /home/chris/glen/pics
For my rear facing camera, I can use this script to adjust the bearing 180 degrees. If you mounted the camera sideways, you could use 90 for a right mounted or or -90 for a left mounted camera. Look at the scripts in a text editor for more information.
You are ready to upload your photos to Mapillary using one of the upload scripts or the web browser tool. Using the web browser upload should be easy. I use the upload_with_authentication.py script following this tutorial on the Mapillary blog
After all this, here is the result from the rear camera for one of the sequences, shown in a widget (press the link under the picture to see the full Mapillary site):
Note about using gpx2exif.
Gpx2Exif is a Ruby Gem you can use to correlate your photos to a gpx track, adding datetimes with a number of seconds between photos. The directions for installing Gpx2Exif is in here, near the end of the Craigs post.
Once you determine the time of the first photo, you can supply that to the script, along with the number of milliseconds between the photos and give it a GPX file. It will add the correct time and GPS location data to the images. The following geotag command will add the date tags to each JPG file starting at time 2014-09-07T22:30:32Z and incrementing the time by 2 seconds for each subsequent photo. Then it will analyze the track and assign lat and lon to your images.
cd /home/chris/glen/pics geotag -g belt.gpx -v *.jpg -T2014-09-07T22:30:32Z -G 2000
2000 is 2 seconds or 2000 milliseconds. It will make multiple passes, and you will see a lot of output if you use -v. On the second pass, it will use the calculated times to correlate the JPG files to your GPS track and calculate the lat and lon. The third pass over your images to write the tags takes a long time, for me about 4 seconds per file. Be patient.
Note about GpsCorrelate
Most tools will not overwrite your Lat and Lon numbers. Sometimes I want to change the values after looking at photos after I’ve written the Exif data. Gps Correlate will clear out the data for you and let you overwrite it. I did not use it in this tutorial, but it has been very helpful. You can install it with apt-get or using the Ubuntu software center. It will clear out GPS tags and re-correlate your photos. Once you have dates in your photos it might be a better tool for adjusting datetimes.