GeoDjango

Django : Faking it with South

So we use the super awesome South to manage schema changes when we change our django models. Basically South works in the background to make sure that your database is always in a consistent state with your models. Recently I have been adding South to a project and deploying these updates to my server. One of the tricky things I found is letting South know when your database is already in a consistent state and that it should mark its migration history as being up to date. Today I had a few moments to properly figure out how to do this. You can substitute RTFM for ‘figure out’ as you prefer. The trick is to do a fake migration like this:

python manage.py migrate someapp 0001 --fake

With 0001 being the migration state that you want to tell South that you are now at. This is documented here. One other noteworthy tip is that in the course of our updates, we are switching from Django 1.2.1 to 1.3.x and South gave me some pretty confusing error messages like this:

Traceback (most recent call last):
  File "manage.py", line 11, in <module>
    execute_manager(settings)
  File "/home/web/sac/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/home/web/sac/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/web/sac/python/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/web/sac/python/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/home/web/sac/python/lib/python2.7/site-packages/south/management/commands/schemamigration.py", line 83, in handle
    if migrations.app_label() not in getattr(last_migration.migration_class(), "complete_apps", []):
  File "/home/web/sac/python/lib/python2.7/site-packages/south/migration/base.py", line 307, in migration_class
    return self.migration().Migration
AttributeError: 'module' object has no attribute 'Migration'

I did’t really find any useful help on google, but eventually resolved the issue by removing the .pyc files from the app/migrations/ folder and then everything just worked like magic.

 

Django development with Eclipse and PyDev

Eclipse is a Java IDE right? Wrong! You can use it for other languages too! I have been needing to improve my debugging techniques for python and Django (more on that in the future). This post outlines how you can set up Eclipse with PyDev to work on an existing Django aplication. The benifits? A complete IDE environment with a debugger that will let you set breakpoints anywhere in your app and interactively step through the code from that point.  You also get other nice things like refactoring support, code completion and so on. I’m still a dyed in the wool command line user since I can use my same development environment both locally and on a remote server effortlessly,  but for training courses and those who prefer to use a GUI, PyDev on Eclipse will probably make a compelling alternative. lets see how to get is set up (after the break below)…

Read more »

How to move a django project and its virtual environment

We do a lot of django work here at Linfiniti and always use a python virtual environment with each project so that the runtime requirements are isolated to that project. Our typical project looks like this:

 

projectfolder
+-- django_project
   +-- django_app
+-- python
   +-- bin
   +-- include
   +-- lib
   +-- local

 

…with an outer folder containing a python virtual environment and a django project with one or more django apps in it.

This works out great…until you need to change the name of your project. You can simply rename everything like this:

 

newfolder
+-- new_django_project
   +-- django_app
+-- python
   +-- bin
   +-- include
   +-- lib
   +-- local

 

Using mv and git mv. But the problem is that virtualenvs are not portable – they have the filesytem paths hard coded into them. Usually I just destroy the virtualenv and then recreate it, but in our low bandwidth office that takes quite a while. So today I came up with a formula to rename everything without rebuilding. There are four steps:

1) rename the outer project folder e.g.

 

 
mv projectfolder newfolder

 

2) rename the django project e.g.

 

git mv django_project new_django_project

 

3) use the linux ‘rpl’ command to replace all references to the old folder names (run from the root of the newfolder directory:

 

 rpl -R "projectfolder" "newfolder" *
 rpl -R "django_project" "new_django_project" *

 

4) remove all old pyc files:

find . -name "*.pyc" -exec rm -rf {} \;
This last step is the important one – if you don’t do it python binary files will contain stale references to the old directory structure and you will see something like this if you try to do a runserver:
ValueError: bad marshal data (unknown type code)
Needless to say you should create a backup before attempting the above.

Django: serving an image from a remote server

For a current project (using django), I wanted to get a dynamically generated image from a remote map server and return the image to the django view as an object (as opposed to returning a URL as a string).

Here’s how it’s done (logic derived from this code snippet):

import urllib
import urllib2
import mimetypes

def serveMapImage( theArea ):
  # omitted: procedure to define the variable zoomExtents
  # from user-specified, database-derived variable theArea
  URI = "http://example.com/mapofsouthafrica-" + zoomExtents
  contents = urllib2.urlopen(URI).read()
  mimetype = mimetypes.guess_type(URI)
  response = HttpResponse(contents, mimetype=mimetype)

  return response

Listing the number of records in all postgresql tables

I love bash and the gnu tools provided on a Linux system. I am always writing little oneliners that help me do my work. Today I was generating some fixtures for django (yes I am finally properly learning to use unit testing in django). I wanted to know how many records where in each table in my database. Here is the little one liner I came up with:

 

for TABLE in $(psql foo-test -c "\dt" | awk '{print $3}'); do \
psql foo-test -c "select '$TABLE', count(*) from $TABLE;" | grep -A1 "\-\-\-\-" | tail -1; done

 

It produces output that looks something like this:

auth_group |     0
auth_group_permissions |     0
auth_message |     0
auth_permission |   273
auth_user |   366
auth_user_groups |     0
auth_user_user_permissions |     0
etc.

 

 

 

Help wanted: FOSS GIS and Python Geek in Africa

Are you a python programmer adept at django and other technologies in the web mapper’s toolbox? We have an exciting project on the go and are looking for someone to help us during the months of February and March 2011. You can work remotely if needed, or here in Swellendam, Western Cape, South Africa if you prefer. Please note applicants must already be available full time for these months (or a good part of them) and already up to speed with these technologies – there will be no time for ‘learning on the job’. Please send me your CV to tim [A T] linfiniti.com if you are interested.

Hope to hear from you soon!

Tim

Bending django flatpages to your will

I’ve been working on a web site for a client that uses django flatpages. Flatpages is a built in component that allows users to create static content within the admin interface and publish it to the web front end. Think of it as a basic content management system component for django. Out of the box django’s flatpages have some limitations though – you cannot easily dictate a sort order or heirachy for the pages. Let me demonstrate…

The old way:

Typically one might show a list of your flatpages on your website you would do something like this

{% load flatpages %} {# Custom tag defined in lib/templatetags/ #}
{% get_flatpages as flatpages %}
<ul>
{% for page in flatpages %}
 <li><a href="{{ page.url }}">{{ page.title }}</a></li>
{% endfor %}
</ul>

Which might generate some html rendered out like this:

Basic flat page listing

Basic flat page listing

Extending the FlatPages model:

We can use model inheritance to add a couple of fields to the flatpages model as explained here. To do that I simply add something like this to my models.py:

from django.db import models
from django.contrib.flatpages.models import FlatPage
class ExtendedFlatPage(FlatPage):
 show_after = models.ForeignKey('ExtendedFlatPage', \
   null=True, blank=True, default=None, \
   related_name="flatpage_predecessor", \
   help_text="Page that this one should appear after (if any)")
 child_of = models.ForeignKey('ExtendedFlatPage', \
   null=True, blank=True, default=None, \
   related_name="flatpage_parent", \
   help_text="Page that shis one should appear under (if any)")

So my new model adds two fields to the flatpages model that will allow me to define ordering and heirachy. The show_after field means that when I create a new flatpage, I can specify which other page the new page should be shown after. The child_of field can be used to state that the new page being added is a child of another page. A quick run of

python manage.py syncdb

Will go ahead and create the new table for your extended flat pages model. If we want to migrate existing flatpages into the new model, you can run a little query from the postgres command prompt like this:

insert into localsite_extendedflatpage (flatpage_ptr_id)
  select (id) from django_flatpage;

Where localsite_extendedflatpage is the table that was generated for your model (its name will vary depending on the name of your django app).

Registering the model in the admin interface:

To allow the user to administer the extended flat pages, you need to register your custom flatpages model with the admin interface and deregister the existing one (in admin.py):

from django.contrib import admin
from django.contrib.flatpages.admin import FlatpageForm, FlatPageAdmin
from django.contrib.flatpages.models import FlatPage

from models import ExtendedFlatPage

class ExtendedFlatPageForm(FlatpageForm):
 class Meta:
   model = ExtendedFlatPage
class ExtendedFlatPageAdmin(FlatPageAdmin):
 form = ExtendedFlatPageForm
 fieldsets = (
 (None, {'fields': ('url', 'title', 'content', \
         'sites', 'show_after', 'child_of' )}),
 )
admin.site.unregister(FlatPage)
admin.site.register(ExtendedFlatPage, ExtendedFlatPageAdmin)

After restarting your web server, you should now see the ExtendedFlatPages model listed in your admin interface, and it should show you our two custom fields which we use to define order and heirachy.

Admin interface for our customised flatpages model

Admin interface for our customised flatpages model

Creating a custom template tag:

The next step in our journey is to create a custom templatetag that will render our listing of flatpages according to their heirachy and order. I added this to <myapp>/templatetags/local_tags.py :

from django import template
from localsite.models import ExtendedFlatPage
register = template.Library()
@register.simple_tag
def show_ordered_flatpages():
 flatPages = ExtendedFlatPage.objects.filter(child_of__isnull=True).order_by('-show_after')
 myString = ""
 for myPage in flatPages:
   myString += get_ul_for_page( myPage )
 return myString

def get_ul_for_page( thePage ):
 flatPages = ExtendedFlatPage.objects.filter(child_of=thePage).order_by('show_after')
 if len(flatPages) < 1:
   return """<li><a href="%s">%s</a></li>""" % ( thePage.url, thePage.title )
 else:
  myString = """<li><a href="%s">%s</a>""" % ( thePage.url, thePage.title )
  myString += "<ul>\n"
  for myPage in flatPages:
    myString += get_ul_for_page( myPage ) #recursion
  myString += "</ul>\n"
  myString += "</li>\n"
  return myString

So the template tag uses a simple recursive function to generate a heirachical nested collection of unordered lists (ul). The last thing I need to do is update my template that shows a listing of available flatpages to look something like this:

{% load local_tags %}

<div class = "flat-page-list">
   {% show_ordered_flatpages %}
</div>

The result:

After making the above changes, my contrived list of flatpages now looks like this when rendered:

Flatpages rendered using user defined order and nesting

Flatpages rendered using user defined order and nesting

A couple of gotchas:

My examples above do not check to see where the logged in user has the rights to view a page. Also my ExtendedFlatPages model needs to have some checks added to ensure that show_after and child_of fields can not be populated with a reference back to themselves (which would probably cause infinite recursion in my templatetag code).

jqGrid: Hands down the nicest grid control for JQuery

I’ve been surveying various grid controls for a web project. I thought I would save others from doing the same trawling and point you to jqGrid. Its beautiful, flexible and powerful – try some of their demos to see what I mean!