Linfiniti Geo Blog

GIS for Open Source People

I have always ejoyed ImageMagick – in particular (much like gdal) I get a wierd kind of pleasure from being able to edit images from something so innately non-graphical as the command line. One common activity I do is resizing images. In this case, I wanted every image in a directory to be 200px wide whilst it’s height should be proportionately resized to match the width.

I also wanted to remove pesky spaces from the file names, and make sure they were all writable. Here is a little one liner I came up with:

sudo chmod a+w *.jpg; for FILE in *; do mv "$FILE" $(echo $FILE | sed 's/ /_/g'); \
echo $FILE; done; for FILE in *.jpg; do echo $FILE; convert -resize 200x $FILE $FILE; done

I know it could have been done more efficiently in a single loop but since I was knocking this together in a jiff, it wouldn’t have been worth the time…

The real take-home command is

convert -resize 200x $FILE $FILE

which resizes the file to 200px wide and then saves it over itself.

pixelstats trackingpixel

Horst Duester and I spent the week in Johannesburg (South Africa) doing FOSSGIS training. As I mentioned in a previous article, Horst is visiting South Africa on a sabbatical. He has kindly donated his time to provide training services in order to promote and facilitate the use of FOSSGIS. We badly need more people evangelising FOSSGIS here – there are many people looking for low cost and good GIS solutions and few people with the time and know-how to show them how to proceed.

We drove up to Jhb from Swellendam in one day – 1400km – a first for Horst who calculated that with an equally long trip from his home town he would have landed up in Russia!

Our first port of call was at Wits University (the University of Witwatersrand, Jhb). Devlyn Hardwick convened the course at Wits and kindly provided us with accommodation for the week. We spent the monday afternoon installing the Ubuntu VMWare image we had prepared onto all of the computers in the lab. At the end of the day it was a real thing of beauty to see so many Ubuntu workstations.

From Tuesday to Thursday we (ok mainly Horst) presented an enlightening overview of QGIS and GRASS to the participants. He covered the process you need to follow to produce a professional quality map – from importing or digitising your date, to symbolising that data to conducting an analysis and finally culminating in the use of the QGIS map composer to create a great map. Much of the development of the map composer was funded by the SO!GIS group at Solothurn and it was really great to revisit just how far QGIS has come along in its ability to produce a high quality map.

Training at Wits University
Training at Wits University

Devlyn and Horst enjoying sumptuous fare at the Ethiopian restaurant
Devlyn and Horst enjoying sumptuous fare at the Ethiopian restaurant

Horst putting QGIS through its paces
Horst putting QGIS through its paces

On friday our activities moved to the CSIR Meraka institute in Pretoria (which is very close to Johannesburg). At Meraka we held a one day PyWPS workshop which was convened by Graeme McFerren. During the workshop we walked the participants through the process that you need to follow to get PyWPS set up on your server, and then followed that with some examples of how to write a PyWPS process. The workshop was quite technical in nature. Those particpants with less technical knowledge were given an understanding of what the OGC WPS standard sets out to facilitate, while those with deeper technical knowledge left with the building blocks they needed to go off and deploy their own PyWPS processes.

PyWPS training on the go
PyWPS training on the go

One of Horst’s motivations for spending his sabbatical here in South Africa was to improve his English. So it was quite a feat on his behalf to conduct his training sessions entirely in English for four days straight.

All in all it was a great and successful week and we are now en-route back to the Cape – though we are breaking up the trip with an extremely pleasant stay over at the Karoo National Park. In true geek fashion we are sitting in our chalet using e2fsck to repair Horst’s dodgy hard disk which seems to have thrown a wobbly after being exposed to our South African electrical system.

Horst in the Great Karoo
Horst taking a break from riding through the desert on a horse with no name

As an aside for those interested, the way to check your disk under Ubuntu / Linux and mark bad sectors so they don’t get used is this:

e2fsck -c /dev/sda5

In the weeks to come Horst will be presenting more free and commercial FOSSGIS training workshops here in South Africa. Hopefully in the future others who feel like a break from the daily grind will come and pay a visit to South Africa and follow suite – the effort is greatly appreciated here!

pixelstats trackingpixel

I’ve been surveying various grid controls for a web project. I thought I would save others from doing the same trawling and point you to jqGrid. Its beautiful, flexible and powerful – try some of their demos to see what I mean!

pixelstats trackingpixel

Its been an enlightening few days. Horst Duester (of SO!GIS fame) is paying me a visit for a two month sabbatical. Horst is here to provide valuable training and knowledge exchange to South Africans interested in FOSS GIS (more on that to come in future blog posts no doubt!). Its been really great having an opportunity to sit down with a fellow FOSSGIS geek and exchange notes – and today Horst showed me something really cool. Horst has written a PostGIS function to clean geometries within the database. Normally I use GRASS if I want to clean the topology of my vector dataset. With Horst’s approach this is no longer needed which is a great timesaver. Here’s what to do in a nutshell:

Assuming you already have a spatial database created (with lwpostgis, spatial_ref_sys etc. sql scripts run on it), download and install Horst’s clean geometry function. Here is on way to do that:

wget http://www.sogis1.so.ch/sogis/dl/postgis/cleanGeometry.sql
psql mydatabase < cleanGeometry.sql

Now lets create a test table and show you how to use it. I used Martin Dobias' excellent PostGIS Manager plugin (available in the python plugin repositories) to create a table as shown in the screenshot below (click for a larger version). As an aside, Martin's tool is a great way for new users to get started with building their own spatially enabled database.

create_spatial_table

Next I created a self-intersecting polygon (in this case a figure of eight-ish shape) as shown below:

borken-poly

The little green cross in the middle of the polygon shows the self-intersection. To clean the geometry, I then ran a little sql command that uses Horst's cleangeometry to fix all the geometries in the table like this:

update testpolys set the_geom=cleangeometry(the_geom);

The result is a multipart polygon with no self intersections but preserving the original shape and attributes. In the image below I've pulled the connecting vertices in the center apart so you can see that two sub-polygons were created:

mulitpolygons-after-clean

pixelstats trackingpixel

While looking at the options for apt-cache today I noticed ‘dotty’ as an option. I presumed this could be used to make grahpviz diagrams of package dependencies. After a bit on googling on how to use that, I came across debtree:

sudo apt-get install debtree graphviz

Of course the first command I ran with it was this :-)

debtree -I --rdeps-depth=3 qgis | dot -Tpng -o qgisdeps.png ; display qgisdeps.png

Take a look at the beautiful (ok and just a teeny bit complex) diagram it produced. Its quite amazing really to see how many different components there are out there in the FOSS world that can be combined to produce cool software like QGIS. It really lets you see that QGIS is just as much an integration effort as it is a bespoke software development effort.

pixelstats trackingpixel

The more I use OpenLayers, the more I enjoy using it. I often refer to the OpenLayers examples page to look at the source of examples as inspiration for my work. Today I happened across another great website providing some more brilliant examples of how to get the most out of OpenLayers, so I thought I would share the resource here in case others also find it useful.

Enjoy!

pixelstats trackingpixel

A little while ago I posted here that the videos we created for the Gentle Introduction to GIS series were online at dotsub.com. I received feedback from some readers that the videos were still locked and so subtitles could not be added. I believe they have been unlocked now and if folks would like to subtitle in their own language (which would be great!) you can go ahead and do this. Currently all the videos should be available with both English and Spanish subtitles and it would be great to see more languages added to the list. Here are the links to all the vids:

· No 1 http://dotsub.com/view/a14b1c1f-b781-41a6-81a4-b567f9baa0f0
· No. 2 http://dotsub.com/view/18f347b4-1e77-4af2-8991-1ff6dc77310c
· No. 3-1 http://dotsub.com/view/4f0be755-6dfa-4787-80fb-56e94b982c70
· No. 3-2 http://dotsub.com/view/a603e8ca-8a78-4a2f-bfed-89c3df43a73d
· No. 4 http://dotsub.com/view/c2fe42a4-84d5-4144-9cbf-43009b4451da
· No. 5 http://dotsub.com/view/46678a27-fb19-448f-b9b6-8ba81b2be2c7
· No. 6 http://dotsub.com/view/6da33c08-0ce0-4540-91d8-2e6c7191bc8c
· No. 8 http://dotsub.com/view/8b44b731-70c6-4fbe-9534-39d3de5a4b87
· No. 9 http://dotsub.com/view/e665a587-3bea-4e14-aef1-a6a346d20208
· No. 10 http://dotsub.com/view/f135cb79-3e04-477b-8ffb-9af912fe935b

pixelstats trackingpixel

The recent release of the 2.9.x OpenLayers javascript library has seen it grow still fatter 857kb for the OpenLayers.js if you use it out of the box from the OpenLayers download.

If you live in a bandwidth impoverished country like South Africa, those 857kb equate to a lengthy wait time while a page with a map on it first loads.

I’ve never really delved much into the inner workings of OpenLayers before – I’m just a contented user – but with this new release I thought I would take a little poke around to see if it was possible to trim some of the fat off the library.

The fine folks at OpenLayers already minify the javascript (which removes redundant white space, compacts var names etc), so my best option was to look at removing parts of the code base I don’t really use. OL supports many data formats that I’m not using e.g. WFS, bing maps, etc.

It turns out its pretty easy to create your own custom build. What I did was unpack the OL 2.9.1 tarball and then do:

cd build

To enter the build directory. Then I created a file called tim.cfg that looks like this:

# This file includes the OpenLayers code to create a build for everything that
# does not require vector support. build.py uses this profile if no other one
# is specified.

[first]
OpenLayers/SingleFile.js
OpenLayers.js
OpenLayers/BaseTypes.js
OpenLayers/BaseTypes/Class.js
OpenLayers/Console.js
OpenLayers/Util.js
OpenLayers/Events.js
OpenLayers/Handler.js
Rico/Corner.js

[last]

[include]
OpenLayers/Format/WKT.js
OpenLayers/Format/KML.js
OpenLayers/Format/WFS.js
OpenLayers/Format.js
OpenLayers/Handler/Point.js
OpenLayers/Geometry/Point.js
OpenLayers/Geometry/Surface.js
OpenLayers/Geometry.js
OpenLayers/Layer/Vector.js
OpenLayers/Control/Attribution.js
OpenLayers/Control/Button.js
OpenLayers/Handler/Box.js
OpenLayers/Handler/Click.js
OpenLayers/Handler/Drag.js
OpenLayers/Handler/Feature.js
OpenLayers/Handler/Hover.js
OpenLayers/Handler/Keyboard.js
OpenLayers/Handler/MouseWheel.js
OpenLayers/Handler/Path.js
OpenLayers/Handler/Point.js
OpenLayers/Handler/Polygon.js
OpenLayers/Handler/RegularPolygon.js
OpenLayers/Control/ZoomBox.js
OpenLayers/Control/ZoomToMaxExtent.js
OpenLayers/Control/DragPan.js
OpenLayers/Control/Measure.js
OpenLayers/Control/MouseDefaults.js
OpenLayers/Control/MousePosition.js
OpenLayers/Control/KeyboardDefaults.js
OpenLayers/Control/PanZoom.js
OpenLayers/Control/ArgParser.js
OpenLayers/Control/Permalink.js
OpenLayers/Control/Scale.js
OpenLayers/Feature/Vector.js
OpenLayers/Renderer
OpenLayers/Renderer/Elements.js
OpenLayers/Renderer/SVG.js
OpenLayers/Renderer/VML.js
OpenLayers/Renderer.js
OpenLayers/Map.js
OpenLayers/Layer/WMS.js
[exclude]

Once that is created, you can build the library then copy it into your project to test:

python build.py tim.cfg
cp OpenLayers.js [your project dir]

Then reload your web page with a map on it. You can incrementally remove features while testing your page still works properly until you reach the happy medium of size vs functionality. In my case my custom OpenLayers.js came out to a much smaller size – 342kb – and given time I could probably strip it down further.

Thanks to the OpenLayers devs for a great library!

Update After posting this I realised that I hadnt removed WFS and KML – after removing those two lines in the above file, the library now weighs in at 290kb!

pixelstats trackingpixel

Recently, a friend pointed me to Python nose, a package providing a very easy way to build tests, based on unittest. I still have to write some code on my own to prove this, but I am too impatient to talk about it! Stay tuned for some code snippets in the next future..
Have a look at nose‘s page:

http://pypi.python.org/pypi/nose

Happy Python coding,

–anne

pixelstats trackingpixel

I have a cron job that backs up my PG databases and emails them to me everynight. Today I wanted to upgrade my PostgreSQL 8.3 databases to PG 8.4 so I made a few modifications to my script so that I could dump out my PG data, and then restore it under PG 8.4. In case you are wondering I am doing this because Ubuntu Lucid now ships with PG 8.4 (yay!). I also made the script generate another script to restore the databases. So basically the procedure is to run the script below on your 8.3 cluster, shut that down and bring up your 8.4 cluster, and then restore your databases into that. Here follows my little script:

MYDATE=`date +%d-%B-%Y`
MYBACKUPDIR=/home/timlinux/sql_backups
MYRECIPIENT=tim@linfiniti.com

DBLIST=`psql -l \
  | awk '{print $1}' | grep -v "+" | grep -v "Name" | \
  grep -v "List" | grep -v "(" | grep -v "template" | \
  grep -v "postgres"`
for DB in ${DBLIST}
do
  echo "Backing up $DB"
  FILENAME=${MYBACKUPDIR}/PG_${DB}.${MYDATE}.sql.tar.gz
  pg_dump -f ${FILENAME} -x -O -F tar ${DB}
  #If you want to email the database uncomment
  #below (will cause issues if backups are large)
  #mpack -s "Daily $DB PG backup for ${MYDATE}" $FILENAME $MYRECIPIENT
done

echo "Procedure to restore one of your backups:"
echo "createdb "
echo "pg_restore -F t .sql.tar.gz |psql "
echo "Or to restore in a batch make a script like this:"
echo "for FILE in /home/timlinux/sql_backups/*; do DB=\$(echo $FILE | \"
echo "  sed 's\/home\/timlinux\/sql_backups\/PG_//g' | sed 's/.${MYDATE}.sql.tar.gz//g'); "
echo " 'Restoring: \$DB'; createdb \$DB; pg_restore -F t \$FILE |psql \$DB; done"

Update 02 May 2010 Uncommented pg_dump line which was inadvertantly commented in my original post.

pixelstats trackingpixel