GeoTagging photos with FOSS

Since I got a crackberry cellphone that geotags images when I take them using its built in GPS, I have become a photo snapping lunatic. Of course one’s lunacy needs some way to manifest itself using FOSSGIS so I am going to decribe here a workflow to enable you to enjoy the thrills of geotagged imagery in QGIS even if you aren’t the proud owner of a completely proprietary crackberry phone! The process I describe below will allow you to geotag a directory full of images using a gpx track that you collect using your gps when you are out in the field. If your camera / phonecam already geotags your images, you can do step 1 and then skip straight down to step 10 below!

Step 1: Install dependencies:

sudo apt-get install gpscorrelate exiv2 python-pyexiv2

Step 2: Clean up your gpx file.

Be careful of using the QGIS editing tools to cut away any ropey looking or unneeded features – QGIS will delete the vertex attributes in the track! Rather use a text editor (like VIM!).

Step 3: Clean any existing geotags from the images:

gpscorrelate -r *.jpg;

Step 4: Find the timestamp of your first image:

exiv2 IMG_0594.jpg | grep timestamp

Image timestamp : 2009:07:31 13:02:53 IMG_0594.jpg

Step 5: Find the timestamp of the first trkpnt in the gpx file that matches the position where the photo was taken.

2009-07-31T10:58:29Z

Step 6: Compute the time offset betweenn your camera and your gps

13 02 53 <-- photo
10 58 29 <-- gpx
 2  4 24 <-- difference

+ 2:00 hours gmt <-- timezone offset
-264 photo offset <-- difference (4min 24sec) in seconds, negative to show gps is behind the camera

Another example:

Here was a point at which we knew we had taken a picture (and we know the filename of the picture):

  685.07
  2009-11-08T06:49:54Z
  ACTIVE LOG
  22:10 11-Jan-10
  eTrex Venture
   0
   0

And then we got the exiv2 dump for the image:

File name       : IMG_2709.jpg
File size       : 960453 Bytes
MIME type       : image/jpeg
Image size      : 2816 x 1880
Camera make     : Canon
Camera model    : Canon EOS 1000D
Image timestamp : 2009:11:08 08:55:14
Image number    :
Exposure time   : 1/250 s
Aperture        : F8
Exposure bias   : 0 EV
Flash           : No, compulsory
Flash bias      : 0 EV
Focal length    : 70.0 mm
Subject distance: 0
ISO speed       : 100
Exposure mode   : Landscape mode
Metering mode   : Multi-segment
Macro mode      : Off
Image quality   : Normal
Exif Resolution : 2816 x 1880
White balance   : Auto
Thumbnail       : image/jpeg, 7605 Bytes
Copyright       :
Exif comment    :

From that we can work out something like this:

08 55 14 <-- photo
06 49 54 <-- gpx
 2 05 20 <-- difference
 +2hrs   <-- gmt offset
  320s   <-- second offset

Step 7: GeoTag your images using gpscorrelate:

For the first example:

gpscorrelate --timeadd +2:00 -g /tmp/knp_gpx_tim_interp.gpx --degmins -O -264 -m 1 *.jpg

Sometimes you need to make some manual adjustments to get the timestamps just right. For the images in our second example we ended up using:

gpscorrelate --timeadd +2:00 -g ../../KZN_tim.gpx --degmins -O -348 -m 30 *.jpg

Step 8: Exiv2 Image GeoTag display:

This is just to show what is going on internally - you can use the -pa option to exiv2 to see the geotags that have now been embedded in the images e.g.:

exiv2 -pa IMG_0700.jpg

produces output which includes something like this:

Exif.Image.GPSTag                            Long        1  8984
Exif.GPSInfo.GPSVersionID                    Byte        4  2.0.0.0
Exif.GPSInfo.GPSLatitudeRef                  Ascii       2  South
Exif.GPSInfo.GPSLatitude                     Rational    3  25deg 4.26000'
Exif.GPSInfo.GPSLongitudeRef                 Ascii       2  East
Exif.GPSInfo.GPSLongitude                    Rational    3  31deg 12.74000'
Exif.GPSInfo.GPSAltitudeRef                  Byte        1  Above sea level
Exif.GPSInfo.GPSAltitude                     Rational    1  675.5 m
Exif.GPSInfo.GPSTimeStamp                    SRational   3  13:25:59
Exif.GPSInfo.GPSMapDatum                     Ascii       7  WGS-84
Exif.GPSInfo.GPSDateStamp                    Ascii      11  2009:07:31

Step 9: Bonus material

gpscorrelate includes an interpolation option (enabled by default) that will derive a relative position between two vertices based on the timestamps. You can also use gpsbabel to interpolate extra vertices into your gpx file. For example I put in a vertex for every 1 second interval into the gpx file:

gpsbabel -i gpx -f knp_gpx_tim.gpx -x interpolate,time=1 -o gpx -F knp_gpx_tim_interp.gpx

Step 10: Loading in QGIS

I have written a simple plugin which will create a shapefile from a directory full of geotagged images. You need to have exiv2 and python bindings for exiv2 in order for it to work, and the geotags must currently be written in degrees / decimal minutes format (which is achieved by using the --degmins option in the examples above). The plugin can be downloaded here. Point the plugin at your directory of geotagged images (it will recurse nested directories too) and afterwards a shapefile will appear in your map view. You can then query the image locations using the EVIS query tool (the EVIS plugin is included by default with QGIS).

You may also want to check out the Easy GeoTagger Project for manual options for embedding geotags into images. EasyGT also includes a vector driver which will treat a directory of geotagged images as vector layer in QGIS.

From the ‘that was fun’ dept…

By the time a release of QGIS makes it out the door, most of us developers have long since forgotten about it and taken to the green fields of being able to add features again in QGIS trunk (i.e. ‘the fun place’). There is however a kind of sigh of relief to have made another milestone in the project and to have reduced the delta between what users want in a desktop GIS and what we provide. This (1.4) release announcement resulted in a bit of a mob rush to the QGIS website which caused quite a bit of downtime for the site over the last 24 hours. Thankfully Chris Schmidt and Frank Warmerdam (and probably others) were on hand to give the server the needed poke with a pointy stick to get it to behave better.

I decided to host the QGIS standalone installer exe on linfiniti.com for this release in order to try to get a better idea of download stats. Despite the server outages (meaning people didn’t have the link to download QGIS off my server) we have done around 900 downloads in the 24 hour period since the announcement. Considering that many people will share a download amongst friends and colleagues, especially here in South Africa where bandwidth is limited, and that there are numerous types of QGIS packages which aren’t tracked, 900 downloads probably means many times that actually installed onto people’s desktops.

QGIS 1.5 should be the final release before we start breaking API compatibility to make way for the 2.0 release. Breaking API compatibility lets use get rid of cruft from the code base and refactor the way we have designed QGIS without the overhead of having to support a large number of existing plugins and custom apps based on QGIS. I am looking forward to the rest of this year and all the new QGIS goodies the developer team will bring to the table!

QGIS 1.4 ‘Enceladus’ released

I just sent out the official release announcement for QGIS 1.4 ‘Enceladus’….lets hope everyone loves it! The QGIS team has done a brilliant job for this one and I can’t wait for 1.5 because things are only going to get better! Read the QGIS Blog for details.

I put the standalone windows installer on my own server this time so I can track downloads – it will be interesting to see just how many people grab it….I’ll post some stats here later if there is anything worth reporting…

GPS Tracker tool in QGIS trunk

Marco Hugentobler and I have been working on integrating C++ based GPS integration into QGIS core.

New functionality

There are two parts:

- a gps background process implemented by Marco that communicates with a GPS and extracts nmea strings and then propogates them to any listing classes
- a user interface, implemented by myself, as a dock panel that shows position, signal strength bar chart, satellite polar chart, options

Similar to Martin Dobias’s GPS project, the GPS ui can be used to capture features (and attributes using the normal attribute dialog mechanism). The ui part uses Qwt for the charting functions.

How is it used?

- Add any reference layers to your project (e.g. satellite imagery backdrop)
- Create or add zero or more vector layers (lines, points, polygons) and enable editing
- Connect the GPS via a serial port (our test systems use serial to usb adaptors)
- Tick the auto-add vertices box in the options pane for easy use
- Now get on the train / bus / car

As you drive along, focus one of your vector layers, and click the Add feature button.

If it’s a point layer a new point will be added, the add attribute dialog will appear and the track will not be destroyed.
If it’s a line layer (should work with poly too, I didn’t test yet), the add dialog will appear and after closing it the feature is added to your dataset and the track will be destroyed – ready to start creating the next feature.

For more control of the vertices assignment you can disable auto-add vertices and click the add vertex button when you want a vertex. Here is what the main panel looks like with auto-add vertices disabled.

The Add Vertex button hides when auto add vertex check box is enabled. The add vertex and add feature buttons grow to use all available vertical space to make them easier targets to hit when trying to operate the laptop in a moving vehicle.

The track marker rubber band colour can be specified making it easier to see (e.g. on dark reference maps you can set the track marker to a light colour). For similar reasons, the rubber band width can be specified.

The reset feature button next to the Add feature button is there so that you can discard collected track data if you want to start your feature from your current location rather than your last saved feature’s end.

Other features

The tool can show a simple histogram showing signal strength:

It can also show a polar chart showing relative satellite positions:

An options panel allows you to specify your device port and various other choices. The map can be panned to be always centered on the GPS position, or recenter when the GPS cursor will leave the current view extents (so minimising refreshing), or never following the GPS. This panel is also where you specify the track width and colour.

When you click the Add feature button, the normal attribute dialog mechanism is invoked allowing you to capture attribute data for your feature. Here is a screen shot showing point capture.

In the field

Here is a pic of my in car setup while testing the software:

Note that the dog is still learning to operate the laptop so at the moment I keep her responsible for looking out the window for points of interest.

QGIS Described in 5 min

A friend asked me for a presentation about QGIS that provides a quick overview of the project and the software. I quickly knocked this presentation together from various other presentations. Obviously there is a lot more detail that one could give about the project but hopefully it gives a basic introduction well enough.

Batch convert a directory of tiffs to ecw

Today I wanted to batch convert a directory of .tiff images to .ecw (MrSid wavelet compressed). Our server has 8 cores so it would be nice to use them all right?

Here is the quick & dirty way I do this kind of job in parallel.

#!/bin/bash
mkdir ecw
for FILE in *.tif
do
  BASENAME=$(basename $FILE .tif)
  OUTFILE=ecw/${BASENAME}.ecw
  echo "Processing: ${BASENAME}.tif"
  if [ -f $OUTFILE ] #skip if exists
  then
    echo "Skipping: $OUTFILE"
  else
    /usr/local/bin/gdal_translate -of ECW -co LARGE_OK=YES $FILE $OUTFILE
  fi
done

The script is extremely simple and is set up so that you can run it multiple times without problems because if looks to see if the output file already exists before trying to write it. If it does exist, it skips straight on to the next image.

To run 8 parallel processes I simply do this at the command prompt (I did mine in a screen session):

./toecw &
./toecw &
./toecw &
./toecw &
./toecw &
./toecw &
./toecw &
./toecw &

Afterwards you can fire up top and watch ‘em go!

top - 18:21:04 up  6:41,  4 users,  load average: 10.29, 9.83, 6.69
Tasks: 216 total,   1 running, 215 sleeping,   0 stopped,   0 zombie
Cpu0  : 56.5%us, 22.5%sy,  0.0%ni, 15.7%id,  4.9%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu1  : 53.3%us, 31.6%sy,  0.0%ni,  8.9%id,  6.2%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  : 50.7%us, 37.5%sy,  0.0%ni,  4.9%id,  6.6%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu3  : 46.6%us, 38.4%sy,  0.0%ni,  4.9%id,  9.8%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu4  : 44.0%us, 29.8%sy,  0.0%ni,  8.7%id, 17.2%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu5  : 30.7%us, 57.4%sy,  0.0%ni,  1.7%id,  9.6%wa,  0.0%hi,  0.7%si,  0.0%st
Cpu6  : 58.3%us, 23.8%sy,  0.0%ni,  9.4%id,  8.5%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  : 46.1%us, 38.6%sy,  0.0%ni, 10.1%id,  4.6%wa,  0.0%hi,  0.7%si,  0.0%st
Mem:  16227956k total, 16144508k used,    83448k free,  1739140k buffers
Swap: 62492832k total,        0k used, 62492832k free, 13383020k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12717 timlinux  18  -2  197m  85m 5384 D  104  0.5   0:55.49 gdal_translate
12536 timlinux  18  -2  171m  77m 5384 S  102  0.5   1:08.95 gdal_translate
12705 timlinux  18  -2  195m  65m 5384 D  100  0.4   0:52.58 gdal_translate
12737 timlinux  18  -2  194m  64m 5384 D   97  0.4   0:40.78 gdal_translate
12549 timlinux  18  -2  195m 103m 5384 S   95  0.7   1:12.68 gdal_translate
12751 timlinux  18  -2  165m  66m 5384 S   88  0.4   0:37.46 gdal_translate
12561 timlinux  18  -2  166m  67m 5384 D   69  0.4   1:03.91 gdal_translate
12528 timlinux  18  -2  164m  65m 5384 S   16  0.4   0:18.24 gdal_translate

One thing to note – I ran this with the data sitting on a storage array – if your data all lives on a single drive you may have serious IO issues doing the above….

GeoDjango 3 Day Course

This week I conducted a 3 Day GeoDjango course. We had 11 particpants consisting of SITA (South African State Information Technology Agency) employees and interns, and other non-SITA employees. The course was given free in order to promote skills development in Open Source GIS, and the lab facilities were donated by SITA, Perseus Park, Pretoria, South Africa.

It was really great to meet the attendees and I hope the course sowed a seed of interested in them that will prompt them to go off and make beautiful open source web sites powered by GeoDjango!

Compiling QGIS Under Ubuntu 9.10

A lot of people are scared off from compiling QGIS on Ubuntu thinking the process is too complicated. Actually its pretty easy and you can do it in just a few steps. This weekend I am updating my desktop PC from Ubuntu Jaunty 9.4 to Ubuntu Karmic 9.10 by way of a clean OS install, so I thought I would document the steps needed to get a working QGIS build environment set up in the hopes that others might like to try. Building QGIS from source lets you be an early adopter, trying out the many new cool features that have made their way into subversion (our source code repository), but that are not yet in our mainstream releases.

Installing Dependencies the easy way

Firstly add the ubuntu-gis repository to you apt sources list:

sudo su -c "echo 'deb http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu karmic main' >> /etc/apt/sources.list"
sudo su -c "echo 'deb-src http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu karmic main' >> /etc/apt/sources.list"

Now update your package database:

sudo apt-get update

Now tell Ubuntu to install all build dependencies for QGIS:

sudo apt-get build-dep qgis

Now say ‘Y’ at the prompt and if you are in South Africa wait a few hours while the bits crawl across the wires and deplete your bank account. On my clean system it downloaded around 124mb of packages – pretty much everything you need to build QGIS including the compiler environment, the dependent libraries and their header files etc.

Building QGIS

Building QGIS is well documented in our INSTALL document that comes included but here is the quick version:

First install subversion and cmake console, cmake qt4 gui’s and a few other goodies:

sudo apt-get install subversion cmake-curses-gui \
cmake-qt-gui gdal-bin libgdal1-1.6.0-grass python-gdal

Now we can go on to build QGIS. I am telling CMAKE to install my hand built copy into my home directory (under a subdirectory called apps) to that it does not conflict with with any other copies of QGIS that may be on the system – although you may run into some issues with having a system level install of QIGS and a local copy unless you have tightly managed your library search paths.

cd
mkdir -p dev/cpp
cd dev/cpp
svn co https://svn.osgeo.org/qgis/trunk/qgis
cd qgis
mkdir build
cd build
ccmake -DCMAKE_INSTALL_PREFIX=${HOME}/apps \
-DCMAKE_BUILD_TYPE=Debug -DENABLE_TESTS=True ..

Set your library search path

Note This section was added in an update on 31 Nov 2009

If you want the grass plugin to load in QGIS, you also need to ensure your library path is updated. I do it system wide. First edit /etc/ld.so.conf as root and add this line:

/usr/lib64/grass64/lib/

Now do:

sudo ldconfig

Running QGIS

I add a launcher to my Gnome panel that points to /home/timlinx/apps/bin/qgis. After that your hand build QGIS will be a simple click away. QGIS source code is always under development and we try at all times to ensure the code in SVN will build cleanly. If you want to get the latest updates, simply do:

cd
cd dev/cpp/qgis
svn update
cd build
make install

Assuming you got no error messages during the build, you QGIS launcher icon should now launch the latest build.

Getting up and running with PostGIS in a jiffy

I’ve posted this before on my old blog so this is a repeat for those looking to get going with PostGIS in a hurry. This procedure should work on Ubuntu Jaunty or Ubuntu Karmic and possibly earlier versions.

sudo apt-get install postgresql-8.3-postgis
sudo su - postgres
createuser -s -d -r -l -P -E timlinux
createdb gis
createlang plpgsql gis
psql gis < /usr/share/postgresql-8.3-postgis/lwpostgis.sql
psql gis < /usr/share/postgresql-8.3-postgis/spatial_ref_sys.sql

Having done that, you should be able to connect to the database by creating a new postgis connection in QGIS. After that you can start to load data into your spatial database using SPIT (QGIS plugin to import shapefiles into QGIS) or the shp2psql command line tool.

Some more sneak peeks into QGIS trunk

We will be going into feature freeze for QGIS 1.4 soon. There has been so much work going on in QGIS for this release, I thought I would take a moment to show the world what you might expect rendering-wise from the upcoming release.

Symbology-ng

The symbology-ng merge has brought in many improvements to the quality of cartography we can do in QGIS. Take a look at this first screenshot (click on it to get the full image):

In the previous incarnations of QGIS, in order to make this map I would have loaded the polygon layer for country boundaries three times. Each instance would have been given a border with a different thickness, so producing the black, grey, blue outline effect you can see in the image. With the new symbology infrastructure, one can now create a custome polygon symbol type that overpaints the lines numerous times. In the screenshot below you can see how the symbology for my countries was constructed.

The key here is the concept of ’symbol layers’. Each symbol layer can have its own fill, outline width and colour etc. They are drawn from bottom to top when the feature is rendered. The ’symbol levels’ button you can see in the background window in the above screenshot lets you dictate if the symbol layers are drawn per feature or for all the features in the view extent. If layers are drawn per feature, then you get something like the screenie below. If you draw symbol layers for all features in the view extent then the bottom most symbol layer is drawn for all features, then the next layer up and so on. This gives you an effect as shown in my first screenshot above.

Labelling

I’ve blogged before about the new labelling engine provided as a plugin for the upcoming 1.4 release. It will be a plugin because it does not yet support data driven label placement and a few other things that we need before it can become the default labelling engine.

In the screenshot above, you can see how neatly the engine places the town labels – none of them overlap and the result is quite a readable, easy on the eye map.

Map Composer

Another area of QGIS that has been receiving a lot of attention is the map composer. It now supports rotating maps, annotations with arrows and shapes and many other improvements. Unfortunately, the new labelling engine is not yet supported in the map composer, and the new symbology framework does not produce a proper legend in composer. We will see if and which of these limitations will be addressed for the 1.4 release. In the screenie below you can see some of the new features at work.