QGIS

Using valgrind, kcachegrind to evaluate QGIS performance

Valgrind is a nifty too to do code instrumentation – tell you how much time gets spent in different parts of your code tree. KCachegrind is a visualisation tool for valgrind outputs. They are two separate tools that together can provide useful insight into where you code is slow or doing too much work. To set them up on ubuntu is simple:

sudo apt-get install kcachegrind valgrind

After installation, you can lauch QGIS under valgrind like this:

cd /tmp
valgrind --tool=callgrind --dump-instr=yes --simulate-cache=yes \
 --collect-jumps=yes /usr/local/bin/qgis

Note that everything you do in QGIS will take 10x longer because of the overhead valgrind places on it. Now load up a project or do something that you feel is taking longer that it should in QGIS. Then exit QGIS. You will be left with a nice big unintelligible xml file in your /tmp directory. Not to worry though, kcachegrind can open it an make sense of it for you. The file will be called something like:

callgrind.out.18364

So to open it with kcachegrind you can simply do:

kcachegrind callgrind.out.18364

KCacheGrind will take a few moments to digest it and then show you something like the screenshot below:

Application call graph for QGIS in KCacheGrind (click for larger image)

Application call graph for QGIS in KCacheGrind (click for larger image)

 

The callgraph can be displayed in a number of different ways and provides extremely useful information if you are trying to identify bottlenecks. In the image above, you can see that when QGIS starts, almost 50% of startup time is spent loading python plugins. There are also views that show how many times a given function is called which can provide some helpful hints as to what to optimise.

 

Valgrind can also be used to detect memory leaks, but unfortunately for GUI lovers, there isn’t a nice GUI environment to explore the leak report data. Don’t let that put you off though – you can get extremely useful feedback from the tool. Here is the syntax to do a leak check on QGIS:

valgrind  --tool=memcheck -v --error-limit=no --leak-check=full /usr/local/bin/qgis

When it finishes running QGIS you will get an extremely verbose report (printed to stdout) that lists all the potential issues. Here is a short snippet of what you might see:

==3144== 627 errors in context 151 of 997:
==3144== Invalid read of size 1
==3144==    at 0x4C2A06A: __GI_strcmp (mc_replace_strmem.c:540)
==3144==    by 0xA3080BA: _nl_load_locale_from_archive (loadarchive.c:161)
==3144==    by 0xA30750D: _nl_find_locale (findlocale.c:107)
==3144==    by 0xA306B38: setlocale (setlocale.c:379)
==3144==    by 0x680F13F: QgsCoordinateReferenceSystem::setProj4String(QString) (qgscoordinatereferencesystem.cpp:753)
==3144==    by 0x680C262: QgsCoordinateReferenceSystem::loadFromDb(QString, QString, QString) (qgscoordinatereferencesystem.cpp:271)
==3144==    by 0x680B9B3: QgsCoordinateReferenceSystem::createFromSrsId(long) (qgscoordinatereferencesystem.cpp:213)
==3144==    by 0x6776E2B: QgsDistanceArea::setSourceCrs(long) (qgsdistancearea.cpp:64)
==3144==    by 0x6776CF9: QgsDistanceArea::QgsDistanceArea() (qgsdistancearea.cpp:45)
==3144==    by 0x67C06DE: QgsMapRenderer::QgsMapRenderer() (qgsmaprenderer.cpp:46)
==3144==    by 0x71B8D7B: QgsMapCanvas::QgsMapCanvas(QWidget*, char const*) (qgsmapcanvas.cpp:115)
==3144==    by 0x5314EC: QgisApp::QgisApp(QSplashScreen*, bool, QWidget*, QFlags<Qt::WindowType>) (qgisapp.cpp:419)
==3144==  Address 0x144ed1f1 is 1 bytes inside a block of size 12 free'd
==3144==    at 0x4C282ED: free (vg_replace_malloc.c:366)
==3144==    by 0xA306BC8: setlocale (setlocale.c:173)
==3144==    by 0x680F0B0: QgsCoordinateReferenceSystem::setProj4String(QString) (qgscoordinatereferencesystem.cpp:743)
==3144==    by 0x680C262: QgsCoordinateReferenceSystem::loadFromDb(QString, QString, QString) (qgscoordinatereferencesystem.cpp:271)
==3144==    by 0x680B9B3: QgsCoordinateReferenceSystem::createFromSrsId(long) (qgscoordinatereferencesystem.cpp:213)
==3144==    by 0x6776E2B: QgsDistanceArea::setSourceCrs(long) (qgsdistancearea.cpp:64)
==3144==    by 0x6776CF9: QgsDistanceArea::QgsDistanceArea() (qgsdistancearea.cpp:45)
==3144==    by 0x67C06DE: QgsMapRenderer::QgsMapRenderer() (qgsmaprenderer.cpp:46)
==3144==    by 0x71B8D7B: QgsMapCanvas::QgsMapCanvas(QWidget*, char const*) (qgsmapcanvas.cpp:115)
==3144==    by 0x5314EC: QgisApp::QgisApp(QSplashScreen*, bool, QWidget*, QFlags<Qt::WindowType>) (qgisapp.cpp:419)
==3144==    by 0x52DFBE: main (main.cpp:607)

 

Just from starting QGIS and then immediately shutting it down, you will see a rather large list of errors – some of which originate from QGIS itself, and others which originate from some of the underlying libraries we use. The leak detection tool is a hand way to debug when you are getting application crashes and when it seems to be using more memory than you think reasonable. One of these days lets hope that it returns a big fat nothing when we run QGIS on it :-)

 

 

 

Selecting GDAL Drivers on the fly with QGIS

There is quite an old ticket in the QGIS bug tracker relating to the need to be able to select which GDAL drivers to use. The issues is this: GDAL in some cases provides multiple drivers for the same image type. For example, JPEG2000 datasets can be opened using ECW’s proprietary driver, KAKADU (gotta love that name!), OpenJPEG etc. The problem is that there isn’t actually any way to choose which driver GDAL should use for a given image format – it works on a first come, first served bases. This is a bit of a problem sometimes when you actually want to be using a different driver. In some work I did recently for SANSA (the South African Space Agency), I built a custom version of QGIS for them. They are particularly keen on the JPEG2000 format since it is an open standard and provides good compression. However the custom build of QGIS I made for them includes the ECW JP2000 driver which unfortunately chokes and dies (no fault of GDAL, it happens down in the ECW code itself) when trying to open their JP2 imagery. Also we would prefer to use a FOSS driver for JP2 in keeping with the general ethos of being a FOSS project. So all this prompted me to implement the needed support to let you disable GDAL drivers at runtime in QGIS which is illustrated a little by the screenshots below.

 

By default GDAL want to use the ECW JPG2000 driver...(click to enlarge)

By default GDAL want to use the ECW JPG2000 driver...(click to enlarge)

Using the options dialog, you can now disable a driver....(click to enlarge)

Using the options dialog, you can now disable a driver....(click to enlarge)

After the driver is disabled (and restarting QGIS), the alternate JP2 driver is available (click to enlarge)

After the driver is disabled (and restarting QGIS), the alternate JP2 driver is available (click to enlarge)

Building QGIS in windows from the command line

Here is a handy tip if you are building QGIS under windows using our official INSTALL documentation. The document recommends that you open MSVC Express and then right click the BUILD_ALL and INSTALL targets. I find using a graphical user interface for this kind of thing tedious and so I looked around for a way to carry out the build from the command line. Here is how to do it:

cmake --build . --config RelWithDebInfo
cmake --build . --target install --config RelWithDebInfo

You should run those commands from within your build directory. I made a little batch file called INSTALL.bat containing the above two lines which I just call when I want to build in windows now.


Norwegian Trekking with QGIS Server

I have blogged about the awesomeness that is QGIS Server before. Maybe you have seen some of the nice maps that Andreas Neumann has made in Uster, Switzerland. Over the past 3 months we were contracted to help prototyping a new mapping service for the Norwegian Trekking Association. We really got a chance to put QGIS Server (and the new QGIS Web Client project) through its paces. Why not take a little look for yourself. We had quite a lot of fun optimising the client’s data set for good performance and generally making sure everything works smoothly.

Note:

- the site is still a test site, but the fine folks from Norway gave me the OK to let people take it for a whirl.
- we have not yet implemented search support.
- We made a bunch of different print layouts for you to try out!

It’s now incredibly easy to get your QGIS cartography online. In the near future I plan to test out techniques for tilecaching with QGIS Server and once that works it will be the end of the line for my 11+ year love affair with UMN Mapserver for the most part I think.

Get your map on! QGIS Server in action (click for full image)

Get your map on! QGIS Server in action (click for full image)

 

 

 

GeoInformatics article on QGIS

Otto Dassau and I recently wrote and article for GeoInformatics about the work we have done in in Tanzania and the QGIS project in general. The article is now online here.

Extract projection tool for QGIS

QGIS got another useful feature a few days ago, courtesy of Alexander Bruy. Here’s the commit.

What this lets you do is to find out what projection your rasters are in. It then exports this to a .WLD file. It also allows you to create .PRJ files.

If you have GDAL Tools installed (and activated), you’ll find this new tool under Raster > Projections > Extract Projection. It gives you this efficient-looking dialog to choose your options in:

Extract projection dialog

The "extract projection" dialog.

The batch- and recursive modes are a great feature as well. You could easily find yourself in a situation where you have a whole project full of rasters, with undocumented projections, in countless subfolders, that got handed to you without comment. Whereas previously that would be a hassle, now you can just point this tool to the right directory and set it loose. Voila, instant projection information! And in your format of choice, no less. Ladies and gentlemen… a round of applause!

Shield Labels for QGIS

QGIS Server (the ‘map’ part of the name has been dropped) is an awesome way to publish maps and you can make really good looking cartographic products with QGIS. I have been building out a good looking online map for a client and I suddenly realised that we don’t have any way to do road shields. This weekend I hacked together basic support for label shields. It integrates with the labelling-NG tool in QGIS master. Currently it provides only basic options – distance from label at which to start the shield and the shield colour. Only rectangular (with slightly rounded corners) shields are currently supported. In the future we could do lots of other fancy shield types and allow setting outline colour, use a gradient for the background, different shapes and so on. In it’s current rather basic configuration though it suits my immediate needs and probably I will add to it as the need arises.

Shield options

Shield options (click for larger view)

Yes the screenshot is taken in windows – something for a future blog post :-P Here is a little sample of what you can do with shields  (click for larger image):

Some contrived examples of using the label shields

Some contrived examples of using the label shields

 

If you want to try it out, it is in my github clone here:

git remote add timlinux git://github.com/timlinux/Quantum-GIS.git
git fetch timlinux
git branch --track shield-labels timlinux/shield-labels

 

 

 

New hatching modes in QGIS Master (line pattern fill)

This week Marco Hugentobler (http://sourcepole.ch) committed a nice new feature to QGIS master branch: Line pattern fills. The default selection of hatches that can be used to fill polygons in QGIS is quite limited. Of course you can always use SVG symbols to extend that, but Marco has added another option. With the line pattern fill symbol style, you can create multiply symbol layers of pattern fills with different rotations, allowing you to create a wide range of fill styles. Rather than try to describe this, its easier to demonstrate with a few images.

 

 

 

 

Of course these new layer styles can be combined with existing layer styles (SVG Fill, Simple Fill etc.) to extend the already huge range of possibilities for symbolization in QGIS. Thanks Marco for another great feature!

 

Improvements to raster performance in QGIS master

Today I committed changes to QGIS master that I started working on at the Lisbon hackfest and finalised this week (along with a unit test yay!). The changes implement improvements to the way in which statistics are gathered for rasters. First a little history: the raster stats code in QGIS is some of the earliest work I have done in QGIS. The code was also hugely inefficient – basically it completely scans every pixel in the file twice! It does this because it extracts minima, maxima, stddev etc. from each band in the raster – and the algorithm used to calculate the stddev requires first calculating the sum of means, hence the second pass. Of course we use the underlying GDAL library to traverse the raster, but we completly bypassed the GDAL stats gathering functions. Now from my testing (see below) the difference of gathering stats in QGIS as opposed to directly in GDAL was not that great. However, the real killer is the fact that QGIS had no reasonable implementation to cache these statistics – either within a session or between sessions. This resulted in quite a lot of  complaints about the slow loading of rasters in QGIS and the spontaneous recollection of stats when making various changes to raster symbology etc.

With the release of 1.7, Radim Blazek implemented ‘proper’ support for raster providers so that they follow a similar model to the vector providers in QGIS. This opens up room for provider specific optitmisations and for implementing new providers in the future e.g. for PostGIS Raster, Terralib etc. At the Lisbon hackfest, I refactored the provider interface so that it provides a default implementation of stats gathering which can be overloaded by specific providers, each then being able to use best practice for that datasource when gathering stats. I then updated the GDAL Provider to use gdal native calls to fetch statistics.

When GDAl gathers stats, it stores them in a file named after the source raster but with a .aux.xml extension. As a bonus, GDAL also stores histogram data in this file. Thanks to some excellent tips from Etienne (Sky?) on the QGIS mailing list my implementation was further improved so that it actually uses these cached stats from GDAL properly. So let’s see how performance is improved with these changes. I tested using a ~650mb tif hillshaded DEM from one of my clients.

 

Raster Load Performance in QGIS

 Added bonus: My changes also use a more efficient approach for obtaining min/max for a band (before all stats are gathered) and also histgram calculations are cached now. So after the initial gathering of stats and histograms, all activities when working with GDAL rasters should be near instant now. By the way the 1 in the graph above is rounded up – in my usage it was near instant in most cases.

I need to still evaluate whether it will be good to backport these changes to the 1.7 stable tree, so for now if you want to benefit from these changes, you need to be using a nightly build or build from source.

Every 6 months we hold a hackfest / QGIS meeting and many of our users generously donate towards it. I hope the above article gives those who do support our meetings a sense of the tangible benefits that arise from these meetings – there have been many other improvements coming into master that arose from the last hackfest that we all get to enjoy! If you would like to help towards the costs of the next hackfest, visit http://qgis.org and click the ‘Donate!’ button on the left hand side. Have fun!

 

QGIS Training in Dar Es Salaam

I just got back from another week away, training people to use QGIS in Tanzania. There are few things more rewarding in life for a QGIS developer that looking out over a class of 30 trainees new to GIS (and QGIS) eagerly absorbing everything you have got to show them about your favourite FOSS GIS.  This was my second time running a training course in Dar Es Salaam and my third visit to Tanzania. What is really nice is that two of the attendees from the previous course I ran (Dr. Edward Kohi and Fadhili Bwagalilo) acted as training assistants this time around. It was nice on two counts. Firstly, the fact that they are now comfortable in a mentoring role for other users shows that the last training course was successful. Secondly, it was really great to have their help – anyone who has ever tried to train 30 demanding people in a computer lab for three days solid will attest to how tiring it is and how valuable it is to have roaming helpers in the room to absorb some of the queries that one gets bombarded with. Thanks Edward and Fadhili!

Hulda Gideon (TanBIF representitive), training assistant Fadhili Bwagalilo, myself and Dr. Edward Kohi (training assistant). (Click for larger image)

Hulda Gideon (TanBIF representitive), training assistant Fadhili Bwagalilo, myself and Dr. Edward Kohi (training assistant). (Click for larger image)

During the course we covered basic concepts of GIS, basic usage of QGIS and introduced the openModellerPlugins for QGIS that we wrote under contract to GBIF / TanBIF last year. Needless to say I also gave my introductory lecture on how wonderful the concept of Free Software and how bad proprietary software is. The latter really resonates with audiences in developing nations. The costs of proprietary software leave many intellectually  out in the cold. The ability to share FOSS GIS software freely and learn on an open system offers a real leg up in the world to someone who would otherwise be struggling to solve complex problems with an extremly limited set of tools.

A class full of QGIS converts! (click for full size image)

A class full of QGIS converts! (click for full size image)

I really look forward to doing further work in Tanzania and other African countries in the future.  If anyone has need of my services in their country, please feel free to contact me. In the future I hope to hold some training workshops for previous course attendees with the idea of introducing more advanced concepts and getting their knowledge to the point where they are comfortable conducting their own training sessions. With this model, before we know it every little village in Africa that has a computer could be putting QGIS to good use to solving problems that have a spatial twist to them! A big thanks to my hosts COSTECH and TanBIF, and personally to Hulda Gideon for working all the behind the scenes magic! I leave you with something to ponder while you sit in your airconditioned office in some cold place….

 

Another beautiful sunset over the ocean near Dar Es Salaam...

Another beautiful sunset over the ocean near Dar Es Salaam...