gdal

Creating coloured rasters with GDAL

One of the most popular posts on my blog ever is my article about creating GDAL hillshades: ‘A workflow for creating beautiful relief shaded DEMS using GDAL‘. In the techniques outlined in the aforementioned article, colours are assigned across a continuum to and superimposed on top of a hillshaded model. The GDAL tools in QGIS have come along nicely thanks to the work of Guiseppe Sucameli – to the point that you can pretty much do the workflow from the aforementioned article within the QGIS gui environment now (barring some simple text file editing activities).

Sometimes one wants to simply assign arbitrary colours to pixel values of a raster to create a colour map. You can do this with QGIS, but the results are not persistent in the dataset, which I needed because I wanted to create my own colour map and then merge that raster with a shaded relief model. There was one extra requirement I had – I wanted to create my colour map based on polygonal areas at the beginning of my workflow. In this article I describe how I went about creating a hillshaded relief model using a polygonal overlay to define my height classes. I’m also going to show the mis-steps I took to give you a more realistic view of the process of doing such work. Read on for details…

Read more »

Selecting GDAL Drivers on the fly with QGIS

There is quite an old ticket in the QGIS bug tracker relating to the need to be able to select which GDAL drivers to use. The issues is this: GDAL in some cases provides multiple drivers for the same image type. For example, JPEG2000 datasets can be opened using ECW’s proprietary driver, KAKADU (gotta love that name!), OpenJPEG etc. The problem is that there isn’t actually any way to choose which driver GDAL should use for a given image format – it works on a first come, first served bases. This is a bit of a problem sometimes when you actually want to be using a different driver. In some work I did recently for SANSA (the South African Space Agency), I built a custom version of QGIS for them. They are particularly keen on the JPEG2000 format since it is an open standard and provides good compression. However the custom build of QGIS I made for them includes the ECW JP2000 driver which unfortunately chokes and dies (no fault of GDAL, it happens down in the ECW code itself) when trying to open their JP2 imagery. Also we would prefer to use a FOSS driver for JP2 in keeping with the general ethos of being a FOSS project. So all this prompted me to implement the needed support to let you disable GDAL drivers at runtime in QGIS which is illustrated a little by the screenshots below.

 

By default GDAL want to use the ECW JPG2000 driver...(click to enlarge)

By default GDAL want to use the ECW JPG2000 driver...(click to enlarge)

Using the options dialog, you can now disable a driver....(click to enlarge)

Using the options dialog, you can now disable a driver....(click to enlarge)

After the driver is disabled (and restarting QGIS), the alternate JP2 driver is available (click to enlarge)

After the driver is disabled (and restarting QGIS), the alternate JP2 driver is available (click to enlarge)

Improvements to raster performance in QGIS master

Today I committed changes to QGIS master that I started working on at the Lisbon hackfest and finalised this week (along with a unit test yay!). The changes implement improvements to the way in which statistics are gathered for rasters. First a little history: the raster stats code in QGIS is some of the earliest work I have done in QGIS. The code was also hugely inefficient – basically it completely scans every pixel in the file twice! It does this because it extracts minima, maxima, stddev etc. from each band in the raster – and the algorithm used to calculate the stddev requires first calculating the sum of means, hence the second pass. Of course we use the underlying GDAL library to traverse the raster, but we completly bypassed the GDAL stats gathering functions. Now from my testing (see below) the difference of gathering stats in QGIS as opposed to directly in GDAL was not that great. However, the real killer is the fact that QGIS had no reasonable implementation to cache these statistics – either within a session or between sessions. This resulted in quite a lot of  complaints about the slow loading of rasters in QGIS and the spontaneous recollection of stats when making various changes to raster symbology etc.

With the release of 1.7, Radim Blazek implemented ‘proper’ support for raster providers so that they follow a similar model to the vector providers in QGIS. This opens up room for provider specific optitmisations and for implementing new providers in the future e.g. for PostGIS Raster, Terralib etc. At the Lisbon hackfest, I refactored the provider interface so that it provides a default implementation of stats gathering which can be overloaded by specific providers, each then being able to use best practice for that datasource when gathering stats. I then updated the GDAL Provider to use gdal native calls to fetch statistics.

When GDAl gathers stats, it stores them in a file named after the source raster but with a .aux.xml extension. As a bonus, GDAL also stores histogram data in this file. Thanks to some excellent tips from Etienne (Sky?) on the QGIS mailing list my implementation was further improved so that it actually uses these cached stats from GDAL properly. So let’s see how performance is improved with these changes. I tested using a ~650mb tif hillshaded DEM from one of my clients.

 

Raster Load Performance in QGIS

 Added bonus: My changes also use a more efficient approach for obtaining min/max for a band (before all stats are gathered) and also histgram calculations are cached now. So after the initial gathering of stats and histograms, all activities when working with GDAL rasters should be near instant now. By the way the 1 in the graph above is rounded up – in my usage it was near instant in most cases.

I need to still evaluate whether it will be good to backport these changes to the 1.7 stable tree, so for now if you want to benefit from these changes, you need to be using a nightly build or build from source.

Every 6 months we hold a hackfest / QGIS meeting and many of our users generously donate towards it. I hope the above article gives those who do support our meetings a sense of the tangible benefits that arise from these meetings – there have been many other improvements coming into master that arose from the last hackfest that we all get to enjoy! If you would like to help towards the costs of the next hackfest, visit http://qgis.org and click the ‘Donate!’ button on the left hand side. Have fun!

 

Frank Warmerdam to become a Google guy

If you follow the osgeo planet blog aggregator, you may have noticed Frank Warmerdam’s recent blog post mentioning that he is off to work for Google. If I think back to the blog posts I have written, GDAL features in a great many of them – its a mainstay tool for me and barely a day goes buy that I don’t gdal_translate some file or what-have-you. Hell, I just finished running a batch job that ran for about 3 weeks comprised of nothing more than bash script and GDAL commands.

In QGIS too, GDAL is one of the key underpinnings of our project – it is only now 8 years or so into the project that we are starting to abstract the use of GDAL for reading rasters away a little to make space of other raster provider types. And of course OGR is one of the main ways that our users interact with vector data from within QGIS.

Personally, Frank has also been extremely helpful whenever I have wondered into the #gdal IRC channel looking for help. Reading his blog post made me think that Google doesn’t quite realise yet just how lucky they are to get him. If Google has any heart, they will make Frank a ‘minister without portfolio’ and just tell him to keep working on GDAL and enjoy the free food in their cafetarias. Here’s hoping at least that he still gets to spend a little time on the software that so many of use love using and depend on.

A workflow for creating beautiful relief shaded dems using GDAL

Sometimes I create hillshades using the QGIS hillshade plugin and then overlay the original DEM over it. I set the DEM to have a false colour pallette and set it to be semi-transparent to produce something like this:

Typical usage of a hillshade with false colour overlay

Typical usage of a hillshade with false colour overlay

That is all well and good but a bit impractical. It would be much nice to have the colour pallette permanetly assigned to the hillshade. Also I want to be able to clip and mask the resulting raster to the boundary specified in a shapefile. Fortunately, GDAL provides all the tools you need to make this happen. There are already some nice articles (like this one) that describe parts of this workflow but I am writing this because I wanted to note the additional steps that I took to make this work well for me.

Before you begin

Before you begin you should have:

  1. a raster containing digital elevation data (DEM) – in my case its called ‘alt.tif’
  2. a vector layer (typically a shapefile) containing the area of interest for your final product – in my case its called ‘tanzania.shp’

Create the hillshade image

The first thing we need to do is generate a hillshade. There is a nice python plugin for doing this in QGIS, you can do it in GRASS using the QGIS-GRASS plugin. But in this article I’m going for an all-GDAL approach so we will be using the handy gdaldem command line application.  I won’t be explaining the parameters I have used here as they are well explained in the gdaldem manual pages.

So to create our hillshade we do something like this:

gdaldem hillshade alt.tif shade.tif -z 5 -s 111120 -az 90

Which will produce something like this:

Our initial hillshade image

Our initial hillshade image

Create the terrain map

Here we want to create a pleasing colour palette for the terrain. There is a lot of content on colour theory available out there on the internet – googling for ‘colorbrewer‘ would be a good place to start if you want to learn more.  Another favourite site of mine is colourlovers.com and for this tutorial I decided to use a pallette from there to see how it would turn out.

Once you have selected a suitable colour pallette (around 5 or 6 colour classes should do it), the next thing you need to do is get some information about the range of values contained in your DEM. Once again you can easily point and click your way to this in QGIS, but here is the way to get it in gdal from the command line:

gdalinfo -mm alt.tif

The resulting output includes the computed min/max for Band 1 – which is what we are after:

Band 1 Block=1334x3 Type=UInt16, ColorInterp=Gray
 Computed Min/Max=1.000,5768.000
 NoData Value=65535

Ok so our minimum height is 1m and maximum is 5768m – Tanzania is the home of Kilimanjaro after all! So lets split that range up into 5 classes to match the ‘Landcarpet Europe‘ colourlover pallette I selected. I set nearly white as an additional colour for the highest altitude range.

65535 255 255 255
5800 254 254 254
3000 121 117 10
1500 151 106 47
800 127 166 122
500 213 213 149
1 218 179 122

The value in the first column is the base of the scale range. The subsequent values are RGB triplets for that range. I saved this as a text file called ‘ramp.txt’ in the same directory as my ‘alt.tiff’ dataset. You will notice that I made the value ranges closer together at lower altitudes and wider appart at higher altitudes. You may want to experiment a little to get pleasing results – especially if you have a relatively small number of high lying terrain pixels and the rest confined to lower lying areas.

Also note that I assigned the highest terrain ‘nearly white’ so that I could reserve white (RGB: 255 255 255) for the nodata value (65535) in this dataset. We will use the fact that white is only used for nodata to our advantage when we do a clip further on in these instructions.

Ok now we can use gdaldem to generate the terrain map:

gdaldem color-relief alt.tif ramp.txt relief.tif

This is what my relief map looked like:

The terrain colour map I produced

The terrain colour map I produced

Don’t worry about the fact that it does not resemble the colour pallette you have chosen – it will do in the next step!

Merging the two rasters

The next step is to combine the two products. I used Frank Warmerdam’s handy hsv_merge.py script for this purpose.

./hsv_merge.py relief.tif shade.tif colour_shade.tif

Which produced this:

The result of merging my hillshade and my terrain colour map

The result of merging my hillshade and my terrain colour map

You may have noticed that it is only at this point that the colours of our product resemble the original pallette we used.

One little gotcha with the hsv_merge.py script is that it throws away our nodata values, so what was sea before (and nodata in my original alt.tif dataset) is now white (RGB: 255 255 255).

Clipping and masking

You may have everything you need from the above steps. Alternatively you can also clip and mask the dataset using a shapefile.

gdalwarp -co compress=deflate -dstnodata 255 -cutline Tanzania.shp \
          colour_shade.tif colour_shade_clipped.tif

My final image is now a compressed tif with nodata outside of the country of Tanzania and looks like this:

Final result: A false coloured elevation map for Tanzania

Final result: A false coloured elevation map for Tanzania

A final note

One of the things I love about the command line is the repeatable and reusable workflows it allows for. I can distill everything in this blog post into a sequence of commands and replay them any time. If you are still stuck doing things in a GUI only way, give BASH a try – you can really do powerful geospatial data processing with it!

Batch clipping with GDAL and bash

I know the little bash scripts I write and post here are popular so here is another one.  The script sequentially unzips worldclim future climate scenario datasets, and then clips them to the bounding box of Tanzania using gdal. After clipping, it removes the extracted files again so you are left with just your original downloaded zip files, and the clipped files in compressed geotif format.

#!/bin/bash
mkdir clip
for FILE in *.zip
do
  unzip $FILE
  cd 30s/
  for BIL in *.bil
  do
    CLIP=../clip/$(basename $BIL .bil).tif
    gdal_translate -of GTiff -co COMPRESS=LZW -co TILED=YES -projwin 29 0 40 -11 $BIL $CLIP
    rm $BIL
  done
  cd ..
  rm -rf 30s
done