Andy McKay's blog

Sadly Matt Berg won't be able to make it DjangoSki. Far too busy at the moment in Uganda working on the next version of Child Count. That's a shame, we'll get him next year when he's had time to catch up on his skiing a bit ;).

Very fortunate to have Brian Leroux, a self described "software hack" from Nitobi coming to speak in Matt's place. Brian works on mobile apps and things like PhoneGap, XUI and Lawnchair.

Checklist for DjangoSki:

• Laptop
• Passport (for any non-Canadians that is)
• Skis / snowboards and gear
• Hot tub gear

We have a few sprint topics planned on the wiki, but if you've got your own, please come ready with those ideas.

Pleased to say that DjangoSki picked up a couple of good speakers.

Andy Smith is a developer at Google, where he works on Google App Engine. He'll be speaking on Google App Engine and how to develop apps for it. Also he'll be talking about the status of non-relational backends for App Engine. A handy person to be sprinting with.

Michael Richardson is one of the brains behind Urban Airship and Bac'n.com. He'll talk about django-piston and building an API in Django which Urban Airship use to do cool things for the iPhone.

Also pleased to note that Rob Hudson is making his way up from Portland. Rob is the writer and maintainer of Django Debug Toolbar. Possibly the most useful debug tool you'll ever use in Django. As a connoisseur of fine Portland beer, I hope you'll be buying him a beer.

A new version of the Django documentation iPhone app has been pushed to iTunes. Updates are:

• Updated to 1.2 alpha 1 documentation
• App now remembers the page you were last on and re-opens it at the start.
• Code samples when page is rotated widen.
• Some copyright notices added.

Thanks to sponsorship from the Python Software Foundation we've got some free tickets to give away to people from charities or non-profits.

Do you know someone who uses Django in a charity? Or are you at a charity? Do you use Django or want to learn about it? Like skiing, snowboarding or just hanging out in Whistler? Then nominate someone, or yourself. We've got a limited number of tickets and time is short - so hop to it.

Fill out the form here.

Note: the image isn't relevant, but was one of the first hits on Flickr for ticket. It's much colder in Whistler.

Update: people chosen and contacted.

Over the Christmas holidays the DjangoSki changed location from one hotel in Whistler to another. This was the reason for the registration and hotel information going offline over the Christmas holidays. We've got a new hotel sorted out and the conference will be at the new location, a few minutes down the road.

So we went for plan B, one of the other hotels. It's a good hotel with other benefits for example: they don't mind external food coming in, a kitchenette in the room, washer and dryer etc. In fact this hotel is directly on the Creekside chair. You can go directly from the conference room, walk to the hot tub, past the barbeque to the chair lift (or some combination of that).

The reason? During the process of sorting out the contract with the original hotel the contract changed quite dramatically against us. We proceeded hoping that the pre-registration numbers would solve that problem. As it turned out it wasn't enough and the amount of money that we were putting up for the conference was so large to be a really, really big concern for us. A conference is a risk, but we are a small company and the risk was too large.

We are sorry about the inconvenience and there's no more changes now, it's full on to the finish line of a great conference in Whistler. We think we've contacted everyone involved who's booked a room, but if this does affect you, then please contact us immediately.

Details are up on the DjangoSki website of the Legends hotel.

Registration is now open for DjangoSki. Early bird tickets are on a simple first come, first served basis and registration is here.

With our three keynotes set up and all the hairy contracts signed away, all that's left for us to do is plan the rest of the conference. And let it snow, snow and snow so that we've got a good base of snow ready for the conference.

The next date to worry about is in about 3 weeks time, which is the deadline for any scheduled talks. We've got a couple of interesting case study talks lined up and still looking for some good introductory talks eg: Introduction to Django, Pinax, Django-CMS and so on.

Been putting quite a few posts about DjangoSki up, sorry about that. It's very exciting for me and been quite a bit of work. Normal business of ignoring the blog and going back to Twitter will resume shortly.

For our final keynote I'm pleased to say that Matt Berg will be speaking at DjangoSki.

Matt Berg is the ICT Coordinator on the Millenium Villages Project at Columbia University. As part of that project he's been building technical systems in developing countries. He's involved with RapidSMS and projects include work in Kenya, South Senegal, Malawi and Rwanda that help children with malaria and malnutrition. And it's all using open source software and Django.

Not only does Matt do interesting work that is helping people right now, but he passionately believes that he can use Django and other technologies to make a difference. His passion and dedication is infectious, as you'll find out.

We have room for other speakers at the conference, so we'd like to call for other speakers to submit proposals to speak at the conference. Please apply here or see details here.

We are excited to announce that Jacob Kaplan-Moss will be giving one of the keynote talks at DjangoSki.

It's almost impossible to be around Django and not know who Jacob is. He's one of the original founders of Django, is on the foundation board, is the co-BDFL of Django and is the co-author of The Definitive Guide to Django. Let's face it, if it's got something to do with Django, Jacob has had some influence on it. Currently Jacob works at Revolution Systems where he gets to do lots more Django.

After missing Jacob at DjangoCon in Portland, it will be certainly good to have him around in Whistler. Perhaps even get some sprinting on Django 1.2 (which will be out a few days after DjangoSki). Jacob writes a great blog and is a great speaker.

I bet you've always wanted to parse your djangopeople.net account into JSONP so you can use it somewhere else on the web. Well your wait is over: djangopeople-jsonp.appspot.com will do just. Some examples of some famous and not so famous Django people. (and of course mr nobody).

We are pleased to announce the first keynote speaker for DjangoSki is David Ascher.

David Ascher is the CEO of Mozilla Messaging. He's been involved with Python for many years and has dabbled around in Django too. He co-authored Learning Python and worked on Komodo whilst at ActiveState (where he became the CTO). Most recently Mozilla Messaging released the rather cool Raindrop, which is written in Python.

David has done some great talks at Python conferences in the past, providing thought provoking and entertaining talks. We think he can ski too, but we'll find that out.

We would also like to thank our first sponsor for DjangoSki, OpenRoad Communications.

If you would like to sponsor DjangoSki and get your company in front of key Djangoers whilst supporting a great conference, contact ClearWind. For more information on the DjangoSki conference, go to djangoski.com.

We've released iPhone app that contains all the Django documentation. Basically we took the documentation, reformatted it a bit to try and fit on the iPhone and then pushed it all into PhoneGap. Unfortunately there was then a 5 week wait as Apple thought about approving it.

The app has just now been approved and is on the app store. There's more information on the Clearwind site, here.

There's a few things we'd like to improve like: reducing the memory footprint, remembering the last page and indexing in spotlight. I'm not sure how many of those things will be possible, but we'll keep the app to date with the latest docs.

And the best thing is, 30 cents out of every app sale goes to the Django Software Foundation.

I am very excited to announce the DjangoSki Conference in Whistler, March 2nd-4th 2010.

This is a conference that combines my favourite web application framework with one of my favourite sports, skiing. If you are going to run a conference, may as well do it somewhere with style right? The conference is slightly unusual in that it's styled a bit after an unconference. There are several keynote talks we are lining up and some introductory talks, but the rest of the conference is up to the attendees.

Each day will have sprints or ad-hoc talks created by the attendees. The real goal is to focus on the things I find enjoyable about a conference - meeting people and learning. Sprints are the ideal way to learn from other people. I'm not alone in this.

Details will follow as we get registration and keynotes organized.

So if you are planning a ski holiday this year and do Django, the choice has just become a whole lot easier. Come to Whistler and have a damn good time.

I'm pleased to announce that in January 2010 I'll be running Django Training in Vancouver. The four day training covers all the things you need to know about getting Django up and running. The training is very hands-on - a bit of talking and then a bit of writing code.

Since it's in Vancouver in the winter, we also thought we'd throw in a nights skiing up Cypress Mountain (where the Olympic free-style skiing and snowboarding will be held). After many years of skiing I still consider myself quite an amateur, so that will be another opportunity for me to learn from you.

After several years of Plone training, I'm really looking forward to doing some Django training. Hope to see you here.

Reprinted from Djangozen.

The other week I faced a real slow down on Arecibo due to a high number of errors being generated by a few other sites. At the time I started to find out how bad Postgres is compared to MySQL in terms of performance when doing a SQL count. MySQL does more work to make that faster.

In Arecibo, we present a list of all the errors that have recently occured. This page is paginated using the Django pagination class. To figure out how big the result set is and hence how many pages you have,  the default pagination class will call the method count on the queryset.

So how can we make this faster, the first obvious strategy, change count on your query set. To do this write a custom object manager and then create your own count. For example:

def count(self):
    # rather stupid count
    return 999999

Hmm that actually works pretty well on some larger Arecibo error pages, because we don't really need to do a count if you've got a lot of results. If you do this, then the pagination class just thinks the list is big and the user sees lots of "next" links.

How about caching it? That's straightforward too.

def count(self):
    # not fully tested
    if cache.get("count", None):
        return cache["count"]
    count = self.get_query_set().count()
    cache.set("count", count)
    return count

Hmm again, ok. The problem is that this is fast on the second hit. The first hit is slow.

But you've got the general idea. The problem with these is that they change the ObjectManager and hence the change all count calls for that model (unless I start having multiple ObjectManagers). Actually for me that's a problem in my application. I still want the django-admin and other parts of the site to work and this change broke quite a lot of unit tests as I quickly looked for a solution.

You could also overwrite count on a per query set basis. Since there was on key view in Arecibo that was causing the problem, I focused on that and wrote the following:

class CountProxy:
    def __call__(self):
        # do something clever here, cache, guess etc
        # to create a count
        return count

Next in your view, assign the CountProxy to the object.

queryset = Error.objects.filter(...)
queryset.count = CountProxy()
Paginator(queryset, ...)

In the end my CountProxy class got quite complicated as it did work to cache and optimise the query based on the filtering. The nice thing is that you have an opportunity to make count as complex and as custom you'd like at this point. Add in some Postgres index tuning and things turned out a lot better. The end result was that I got page that was taking sometimes around 8 seconds (it was a bad day), down to about 40ms.

The ability to do fine grained control on that count can be quite useful.

This was my first DjangoCon and the first conference I've been to where I felt I didn't know a large number of people. I'd always gone to conferences with more than one person, or where I knew a large amount of people. It wasn't quite "social skydiving" as my friend likes to call it, because as it turns out I knew lots of people there.

Tuesday was the first day of the conference and the first talk. I was surprised by Fake Jacob who gave a very witty first talk that had us all laughing - a good way to start the conference. The conference was a big room, enough for 200, with tables running across the hall. Everyone had a laptop and pretty much they were a Mac. In some ways I prefer not having a table to make people look at the speaker.

Some of the highlights were:

• "Almost a Vancouverite" Avi Bryant's talk, which was interesting on a few fronts. The last 5-6 sites I've built have all had just a REST and JSON interface, but for me the catalyst of this was jQuery, JSONP and Google App Engine which allowed you build and mashup sites really quickly. I covered this at Open Web Vancouver.
• Simon Willison's talk about the Guardian work was excellent. Based on stuff I knew already, there were still some interesting snippets in there.
• Going out for an excellent fondue with a whole bunch of new friends and then meeting up later with others for some fine Portland Ale. If I listed everyone, I would offend the people I left out. One person it was cool to see later was Michel Pelletier.
• I missed Ian Bicking's keynote, I'll blame it on the beer, but had a good chance to chat to him afterwards.
• That night a whole bunch of us got together to talk about the Django Software Foundation and that was cool. It's something I'm very keen to help out on and definitely an area I can contribute on. Not much was decided but lots of ideas and enthusiasm.
• Wednesday had some great highlights with Ted Leung's talk which was interesting as ever.
• I really liked the Custom Signals For Uncoupled Design talk. I'm a big fan of signals and the idea of just sticking them all of the place, such as pre and post form creation, seem like great ideas. And Justin Bronn's talk was great - short and informative. Both talks I can bookmark and know I'll come back to one day.

All in all it was a great conference and met a whole bunch of cool people there, I'll know next DjangoCon I can walk in and it will feel like meeting a bunch of old friends.

I did two talks (both in this blog) and they went pretty well, the malnutrition one seemed to particularly well received. For those in the audience on the debugging one I'm sorry - last time I used those font colours and combinations it worked really well. The projectors were quite overwhelmed by the lights and I couldn't see the slides on the project due to the setup. Wish someone had said, something. I spent a while making them clearer for the next one.

It did seem at this conference that quite a few things went wrong, with speakers not turning up and confusion about the talks. But Robert Lofthouse was around, helpful and responsive and that's great. Looking forward to the next one.

These are the slides of the Fighting Malnutrition with SMS and Django, it covers my work with the RapidSMS project in Kenya and Malawi and focuses on some of the technical bits.

RapidSMS is in need of Django programmers willing to volunteer - especially people who can speak Spanish and French. If you are keen to help out, you could find this a very rewarding projects to work on.

These are my slides from the talk at DjangoCon 2009. It was only 15 minutes long so had to go pretty quickly and skips over some details, but hopefully it's useful.

Update: met Rob Hudson after (of django-debug-toolbar fame) and he said that django-debug-logging does pretty much what my django-sql-profile does, so go check that out.

Call them best practices if you like, but here's the things I do when starting a new Django project.

• Create an "app" app. The app folder is the catch all for all the non-reusable parts of your site. The bit that actually does alot of the site specific stuff. Usually, but not always, it only contains an emptymodels.py (this is so that Django knows its an app).
• Create a templates folder. Inside the app project, this is for all the site specific stuff, as opposed to hopefully reusable parts in the specific folders.
• Create a template context processor. You know you will need one eventually, every project does. Don't forget to hook it into the settings file. Mine is normally called context.py and placed in the app project. It does absolutely nothing at this point, it's just waiting to be customised.
• Create a middleware file. You might not need it, but chances are you will. Don't forget to hook it into the settings file. Mine is normally called middleware.py and placed in the app project. Again it does absolutely nothing at this point.
• (Optional) Create a custom authentication backend. Called authentication.py and placed in the app project. Hook it up when you need it.

So that's the app project. Next let's set up the static content, that's easy:

In a couple of weeks it will be time to head down to Django Con. Looking forward to meeting a whole bunch of Django people. I'll be giving two talks and it would be great to see you there.

The first is entitled "What the Heck went Wrong" and is aimed at more novice developers and focuses on debugging techniques and tools. From how to do a simple print statement to tools like Django Debug Toolbar.

The second is "Fighting Malnutrition with SMS and Django" and covers the great work being done by the RapidSMS team and my brief involvement with it. It's a longer talk that covers an innovative use of Django and some sample code for using Django in a slightly different environment.

Also sad to note tonight that at the moment both djangopluggables and djapp are both offline. Yet djangozen is still chugging along.

In defiance of the basic wisdom:

Don't release anything on a Friday

I released some new stuff on a Friday.

Arecibo actually got the biggest release, which is a completely new database structure behind the scenes. On the front-end there's a new look and feel. With that release in place I can now roll out the new features that have been queueing up on the site. Once all the database is re-indexed and all good, I'll put out an announcement on the Arecibo blog about it. Just polishing those last few things now.

For doing the migration I used South and I have to say it's pretty good. Looking forward to using that in the future.

DjangoZen got a few little bits of polish. I've also added in a job board. It's slightly sparse at the moment, due to no jobs being present. But if you are looking for a django developer, now is a good time to enter a job, then send me a list of bugs about the problems with the job board :) Jobs are free at the moment.

In this project we are processing text messages. So peice of text comes into a method and needs to be routed to a particular view/method/class/thing. There's lots of ways of doing it, a big if might be one. RapidSMS includes a damn cunning way of doing it by using decorators.

Here's an example:

    @keyword(r'n(?:ote)? \+(\d+) (.+)')
    @authenticated
    def note_case (self, message, ref_id, note):
         [...]

I thought that was really neat, you can see the decorator here. Nice stuff!

However since I'm a GOD (Grumpy Old Developer) after using it for about 10 minutes I see there's a basic problem I'm not happy with: I find it hard to track what message goes where. There could be 100's of lines of code between each decorator, its hard to see them all at once.

Hmm. So how about using Django's urlpatterns? I actually like the url regex in Django because it's easy to see in one editor window what is going where.

In my case I made a file: msgs.py that simply maps the regex's to the view:

from django.conf.urls.defaults import patterns

urlpatterns = patterns('',
    (r'^country (?P\S+)', "apps.testy.views.country",),
    (r'^list (?P\S+)', "apps.testy.views.list",),
)

Then in my app I can use the URLResolver:

from django.core.urlresolvers import RegexURLResolver, Resolver404

resolver = RegexURLResolver(r'', "apps.testy.msgs")

In my class I'm able to use the resolver in a method, on any text:

       try:
            callback, callback_args, callback_kwargs = resolver.resolve(text)
        except Resolver404:
            raise ValueError, "There was no view found for: %s" % text
            
        response = callback(self, message, *callback_args, **callback_kwargs)

And now I can pass text through to different views based on those regex's. The only problem with it is that (understandably) it's using HTTP semantics. That Resolver404 inherits from a HTTPError, which isn't ideal for a non HTTP source. Also it's kind of annoying having to call it urlpatterns in msgs.py since they aren't really about URL's.

So then that's where I'd say something like: I've ripped urlpatterns out of Django, made it generic, placed it on pypi, then put a Django wrapper around it for URL'ishness and put it back into the Django project. Except I haven't. But if you were doing a project that included doing logic on a bit of text.... perhaps you could do that.

Added some posts to djangozen.com:

• Signals vs overriding save
• Content mirror and djangozen

I added in tagging for plugins (the tag cloud doesn't change the fonts because I hate that) and allowed anyone to own and unowned project. I also added in notifications to twitter, follow along at http://twitter.com/djangozen.

And finally I pulled out the plango source which powers a lot of the site and put the source online: http://djangozen.com/plugins/view/plango/.

Next the job board, oh and a bit of traffic would help.

If you are following this blog for Django things then it might get a little bit quieter. This weekend I started up djangozen.com a website devoted to lots of stuff about Django. There is already a Django website of course and lots of other sites with useful Django stuff on them, but on the whole there are some things I think are missing. If you want to know more about that sort of stuff, go read the about page.

Technically DjangoZen uses:

• Plone as the content writing and managing interface
• Django as the delivery interface, using Postgresql
• Contentmirror copies content (mostly) into Postgresql
• Used jQuery for lots of fun bits
• I got annoyed with all the OpenID libraries and implemented my own based off django-openidauth.
• I use the Blue Print CSS framework and was very happy with it.

Now to adding in more features and writing some content.

This just removes the contrib from mentions in the code and is a bit more explicit on a failed path expression.

Download: simpletemplate.0.5.zip

Rant, swear and slag off a framework. A lot. Then the people will keep coming back. Dammit I knew I was being too nice.

Well, in two days my blog has had 33k unique visitors. Django post got 20k so far. Very interesting.

...or how I used django-openid all wrong in the first place and I've just gotten it cleaned up.

A while back I dedicated myself to trying to do everything with OpenID since having all these logins all other the place is just awful. With sites dedicated to the technically minded it seems the right way to go. So I grabbed django-openid out and slapped it in. This handles the login and logout for you. When an authenticated request is made a variable is appended to request by its middleware: request.openid which in turn provides account information.

So then it was a simple matter of checking if that request.openid is there and finding the user. In the old version of my site that was very simple, just a decorator to check someone is logged in and I'm done.

In the next revision I need permissions, so I thought I'd use the Django permissions from contrib.auth. No problem... just make the current user model a user profile and I can use permissions. Except this is where it gets sticky. Because I was working around the Django contrib.auth with using openid, things weren't working. This came to the fore when I tried to use the test client to check that I can only view pages when logged in.

To do that you use the login method of the test Client, so here's what I wanted to check:

from django.test import TestCase                
from django.test.client import Client

class register(TestCase):
    def testLogin(self):
        client = Client()
        client.login(openid="http://some.user.id/")
        res = client.get("/listener/view/2/")
        assert res.status_code == 200

    def testFailLogin(self):
        client = Client()
        res = client.get("/listener/view/2/")
        assert res.status_code == 302

However there's a problem. The login code works with the authentication backend, django-openid doesn't define one and neither did I. After trying to bend the Client to my will I realised I was going the wrong way and use the django authentication system that's there. So let's snip out all the messing around in the middle and cut to the chase of what worked.

Firstly, define an authentication backend, so that's a matter of grabbing the openid from the request and then searching in my UserProfile:

from general.models.user import UserProfile
from django.contrib.auth.models import User

class OpenIDAuthentication:
    def authenticate(self, openid):
        user = UserProfile.objects.get(openid__exact=openid)
        if user:
            return user.user
        
    def get_user(self, user_id):
        try:
            return User.objects.get(pk=user_id)
        except User.DoesNotExist:
            return None

Then you add in:

AUTHENTICATION_BACKENDS = (
    'general.authentication.OpenIDAuthentication',
)

That then handles my authentication. Now in my code i can call:

authenicate(openid="some id")

All I had to do then was realise that most of my old login handling and checking methods were wrong and remove them or make them use the contrib.auth ones. Once that was all done, my test cases ran much more smoothly. Going back and looking at django-openid now, this makes much more sense and will hopefully be easier to switch to a more up to date openid version.

One thing that I looked at a while back, but didn't have chance to play with was Google Ajax Feeds. These give you an interface to easily read RSS feeds in Javascript.

A few years ago I tried this in Javascript using ClearRSS for Plone. The only problem with that was that I used a generic server side proxy (now redundant thanks to JSONP) and found myself hitting all the inconsistencies with feeds and programming that in Javascript. This API removes the first problem and solves the second by normalising all the data so that it is accessible consistently.

The result is that it's easy to read feeds in Javascript. As an example I just created a widget for Cleartrain, so that all trainings can be easily embedded on a page. For example all the Django training can be pulled using this:

<div id="cleartrain-feed" />
<script type="text/javascript" src="http://www.google.com/jsapi"></script>
<script type="text/javascript" src="http://media.cleartrain.ca/widget/cleartrain.js"></script>	
<script type="text/javascript">
    cleartrain.url = "http://cleartrain.ca/atom/topics/4/";
    cleartrain.content = false;
</script>

Cleartrain is itself a Django site and required no modification above and beyond having RSS feeds for everything. The Javascript to allow this to be embedable is really easy, all we do is read the Ajax API, loop through the entries and then add into the innerHTML. The source for that is here: http://media.cleartrain.ca/widget/cleartrain.js.

Definitely a neat API from Google and a very quick and easy way to create an RSS widget.

One of the nice things about Django tests is that if you are testing your views (and who doesn't) you can test the context that comes back. That is, when you make a call to a template, you could check the HTML if you wanted, but you could just check the right data was sent to the template.

Unless you are using my simpletemplate that is, since Django testing patches the internal Django templating to pass back the context. I've just added the same to simpletemplate, so that it does the same. At the top of your tests you need to add one line:

from django.contrib.simpletemplate import test

Then in a test we can do:

        c = Client()
        res = c.get("/")
        assert res.status_code == 200
        assert res.context != None, "Test that my simpletemplate patch is working"
        assert res.context["objects"], "Check that we have some objects"

That's in simpletemplate SVN.

Sigh, the more I use simpletemplate, the more I see all the silly things I did when I wrote it (like putting it in contrib) and need to clean that up one day .

One of the mis-conceptions with open source is that there's a lack of training. That's not true, it's just that there's a lot of training spread out all over the place through different vendors. There are a couple of sites that do list the training, but there a few things I want from them:

• Ability to rate the training so that other's can see the how good it is.
• Similarly give comments on the training.
• Get all the training in my calendar (which means iCal) and RSS.
• An API for accessing the training would be nice.

This sounds like it's a pretty straightforward Django site. So a few days later I'm announcing Cleartrain, a site to index training on open source projects. We've indexed quite a lot of the trainings we've been able to find. At first, we are specifically focusing on our favourites Plone, Django and Python. The latter two seem quite under represented, probably need to keep digging.

The site is using Django and jQuery and Page Templates. The Javascript is done using jQuery (despite earlier posts) and go done quite quickly, I do like the clicking on an item filters it in the list, i'd like to add in the reverse as well.

We'll keep adding into this site as we see training, but if you have training you'd like to add in, please add it in here.

I wanted to include a regular expression in my Django URL's, but I didn't want to that be passed as an argument to the view. The answer is in the Python regular expression documentation:

(?:...) A non-grouping version of regular parentheses

So in my case a URL like:

r'^(?:site/[0-9]+/)view/(?P<id>.*)/$

Would match the URL:

/site/1/view/2/

But by using (?: only the id is passed to the view.

There was an interesting post on Dion's blog about using Prototype. I have to throw my hat in the ring there too, I often prefer prototype and fluctuate between that and jQuery.

One application I'm working on revolves around an interface built in Ajax. It started off as complicated, but is rapidly getting simpler - but I moved from jQuery to Prototype. I found that while jQuery was good there were times when I just couldn't get it to co-operate. I was bind a DOM element to an event and then changing the DOM and no matter what I did could no rebind that event to the DOM.

I read the FAQ's, I asked in #jquery but couldn't get anywhere. That magical $ function which is so handy was just swallowing everything. I plopped back into Prototype and find it an easy switch that involved less documentation reading and odd errors. For a while. But I certainly found that Prototype has a place.

With some serendipity this post appeared. A while ago we made this decision at Blue Fountain, well let's rephrase, I made this decision. As it turned out, it made absolutely no difference, but the key reasons at the time running through my head were:

1. All the Ruby and Ruby on Rails developers have moved on leaving Python developers
2. We do Plone and there is more opportunity for an overlap between Django and Plone than Rails and Plone
3. Clients don't really care either way

The other issues like Django is faster and easier to develop and has great features like the admin interface, were there. But the above made it a slam dunk.

When tool or programming choices are made, they are rarely made for the reasons that you think. All the big comparison charts can be made, but in the end the reasons people pick A over B can be the oddest ones of all: "We've already paid for an Oracle license" for example.

Want a reason, for Django, here's a good one.

The other day I was putting on an iPhone skin for Arecibo. Just a HTML and CSS wrapping of the site using IUI. But the form rendering options for forms aren't quite right since IUI is very specific about the HTML is laid out.

There's a simple way to fix that though, add a custom method that outputs it the way I want:

class Form(forms.Form):
    ....
    def as_iui(self):
        return self._html_output(u"""
        <div class="row">
            %(label)s
             %(field)s%(help_text)s
        </div>
        """, u'%s', '', u' %s', True)

Calling form/as_iui gives us the form rendered just as IUI want's it.

These last few days I've been doing some upgrades to Arecibo. These include adding in new fields and converting times into a users local timezones - figuring out when an error occurred is now just that little bit easier.

I also moved from mod_python to mod_wsgi. I still have to tune those settings but it looks like a great win in terms of performance and mirrors what other people have been saying about mod_wsgi. There were helpful docs here and here. Was very easy to setup too, worked first time.

Also did some quick benchmarking with CoralCDN and just to confirm, its stated goal is to make a website available, but not fast. My site was way slower with CoralCDN, probably more available, but slower.

This has cropped up a couple of times on irc, so here's a recipe I have for restarting my Django instance during development. I'm not the sort of person who takes a long time to think it all through and can write a model and call it done, I iterate quickly and often. For the quickest way to do that during the initial development, I drop the db, re run syncdb and import my fixtures regularly.

Once a production version is out and running I then get rid of this script and do migrations.

There's no easy way from the command line to create a user neither syncdb --noinput or createsuperuser --noinput accept a password. Leaving lazy people like me wanting. I used to use pexpect but I find just creating a user object nice and easy. So for a project called collector, the following works well for me:

#!/usr/local/bin/python
import os
os.environ["DJANGO_SETTINGS_MODULE"] = "settings"

# dump the data to fixtures
os.system('python manage.py dumpdata collector --indent=2 > collector/models/fixtures/data.json')
# drop and recreate the db (this is for postgres)
os.system('dropdb authy')
os.system('createdb authy --encoding=UTF-8')
# sync the db
os.system('python manage.py syncdb --noinput')

# create a super user
from django.contrib.auth.models import User
u = User.objects.create(
    username='xxx',
    first_name='',
    last_name='',
    email='amckay@bluefountain.com',
    is_superuser=True,
    is_staff=True,
    is_active=True
)
u.set_password('xxx')
u.save()
print "User account created"

# load the fixtures back in
os.system('python manage.py loaddata data.json')
# run the server
os.system('python manage.py runserver')

I can then quickly get in the cycle of creating a model, entering data in admin interface, dumping it to a fixture and then writing unit tests.

This is a simple rip off of the DCWorkflow that Plone uses, DCWorkflow originally coming from Digital Creations many moons ago. Anyway I always liked it because its very simple and abstracts quite a few things away from the object, letting admin's decide how things happen. The goal of this state workflow is to:

• put any number of workflows on a model
• the workflow can contain mulitple states
• there can be multiple transitions between the workflows
• conditions (well scripts) are checked before a transition is run
• scripts are run in the transition

Workflows can be constructed through the admin interface, or through scripts. The admin interface is a little rough and you can't actually workflow things in it. Since it doesn't check which scripts can be assigned to which workflow, or look for changes, but since I don't want to use the admin interface, I'm not too worried about that.

Users don't actually get to write any scripts. Scripts are created by the admin and given descriptions, the admins can then decide which script to assign to which transition and when. There are two kinds of (optional) scripts:

• permission: this is to check the object can be transitioned and gets passed the object and the transition. If you want other things like request or user, you might have to use threadlocals or other hacks to get them. If any of the assigned permission scripts does not return True (or anything evaluating as True), then it fails.
• do: this is what happens when the script is run and involves changing things, logging, emailing that sort of thing. Script gets passed the object and the transition. Returns ignored, if there's a problem raise an error.

The unit tests explain this better than some text, but here's a brief synopsis. Supposing a news item workflow, three states: private, pending, published. Someone in the office needs to review it before it goes live. Also all the fields have to be filled out.

Here' an example permission script assigned to the review transition, ie you can't workflow to review until you fill these out:

def check_complete(obj, transition):
    for field in ["title", "description"]:
        if not getattr(obj, field):
            return False
    return True 

So here's our news item:

                                       
from stateworkflow.models.state import State

class NewsItem(models.Model):
    title = models.CharField(max_length=255)
    description = models.TextField()
    published_date = models.DateTimeField(blank=True, null=True)
    state = models.ForeignKey(State)   

You can do all this manually (see unit tests) or try the following WorkflowManager a nice wrapper around it all:

from stateworkflow.workflowmanager import WorkflowManager

self.one = NewsItem()
self.one.state = Workflow.objects.get(id=2).default_state 
# at the moment we can't transition
self.one.title = "Hello"
self.one.description = "This is a description"
self.one.save()

# we should now be to able transition
wf = WorkflowManager(self.one)
wf.do_transition("Request review")
# we are now pending
assert wf.get_state() == self.one.state
assert wf.get_state().name == "Pending"

For more its in SVN at:

svn co http://svn.clearwind.ca/public/django/stateworkflow

Hope its helpful.

You can move models.py and views.py into seperate directories called models and views. I also make a directory to stick all my forms in called, excitingly enough, forms. But you can make a tests directory and put your tests in there. Grrr. A bit of Googling didn't get me too far, so a bit of reading of source got me this far.

The key is that Django will look to see if tests.py has a "suite" method, if so it will run it. This allows us to do what we'd like with the test runner.

So, make a directory called tests and put in it your unit tests, make a __init__.py and in that reference each of the modules you'd like to run tests on. Contents of my __init__.py:

import unittest       
import browser
import site 

__tests__ = [browser, site]  

def suite():
    suite = unittest.TestSuite()
    tests = []                           
    for test in __tests__:
        tl = unittest.TestLoader().loadTestsFromModule(test)
        tests += tl._tests
    suite._tests = tests
    return suite

My tests directory has a browser.py and site.py that contain unit tests. Adjust the imports and the __tests__ lists as you see fit for your instance. When I add a new file, I register it here.

Django 1.0 has been released. Yay! Well done to all those people involved.

The weird thing I was fiddling in Django most of yesterday and today, spending lots of time in the documentation section on the website, but didn't notice until I saw it in my RSS feed.

I know this is old hat, but I still get a fact that I can still run HTML Tidy over my Page Templates and fix up indenting and a few other things. Perhaps it's just because I've seen some horrible Rails templates that are a badly indented and hard to read. Well I've seen a few Page Templates like that, but hey at least I can clean them in one simple bash script:

tidy -i -m -xml $1

The login decorator for Django looks for a user, as if you'd logged in through the admin interface. That's not what OpenId gives you. For that you'd want something like this:

def login_required(fn):
    def new(*args, **kw):
        request = args[0]
        if request.openid is None:
            return HttpResponseRedirect("/login")
        else:
            return fn(*args, **kw)
    return new  

Then for any views that require OpenId login on them, slap the following before them just like normal:

@login_required

My colleagues got hit by this issue the other day. It's a definite annoyance, and just for the record here's my version, just on the off chance that os.curdir is different.

import os
this = os.path.dirname(os.path.abspath(__file__))

TEMPLATE_DIRS = (
    "%s/jobs/templates" % this,
)

On other things I integrated django-openid the other day into a Django site and its really nice. It worked really well, although I do have to find a way of altering the templates to go nicely into my Page Templated site. I do have to put in some work to the login screen. I have to explain the OpenId login, without making it too daunting and provide an easy way to create an account.

Finally I had a quick play with Google maps. Want to show a map based on a UK postcode? How about:

          var postcode = "L7 9NJ";
          if (postcode != "") {
              if (GBrowserIsCompatible()) {
                var map = new GMap2(mapnode);
                var lookup = new GClientGeocoder();
                lookup.setBaseCountryCode('uk');            
                map.addControl(new GSmallMapControl());                
                lookup.getLatLng(postcode, function(point){
                    map.setCenter(point, 12);
                });
              }
          }

Yay. Not the encoding has to done with setBaseCountryCode as UK, not .co.uk or gb. Otherwise it centres on Germany, not Liverpool.

I've got quite a few Plone sites floating around. And quite a lot of those sites I end up maintaining and I don't really want to, I'd really like it if someone else could. Further those sites aren't really suited to Plone. They are more external brochure ware sites. Mostly static with a small number of editors and no looking at the Plone interface at all. For those sites, Plone is a fine choice, but it's really not its main strength.

So after talking about how much I like Django I thought it was time to start actually doing something. I took one of our Plone sites and said: can I convert this to Django in a weekend? And the answer is simply no. I was hampered by sucking at coding and the family wanting attention. After a few nights of messing around I thought it was time to kick something out and see if I can get on to the next few things in my rather long queue.

First off, converting Plone to Django is a hopeless task, Plone is so big and has so many features that this is a very, very, very big project. All I was initially curious in was my site, could I convert over some content types, pull the data over, set up the URL's and put the views up. This would not mean doing things like security, i18n, users, properties, topics, gosh anything useful at all. So with that caveat, here goes.

Django needs some models, rather than hard code them, how about a little conversion script that goes and reads your Archetypes schema and outputs a Django model. I picked a few fields I really wanted and pulled out some key fields into a metadata (such as effective date), tell it the models you want and you'd get:

class Document(models.Model):
    uid = models.CharField(max_length=255, help_text="Should not  [snip]")
    title = models.CharField(max_length=255, blank=False)
    description = models.TextField(help_text="A short summary [snip]", blank=True)
    text = models.TextField(blank=False)
    ...

Because things like reference fields or lists are unclear, it just skips over those. All possible, but a little too much for the moment.

So you'll get a project and app which I called plango with a bunch of models. Next we want to run through the database and pull out the object and write them into Django, that's a matter importing Django inside a zopectl run script. The magic code to pull the fields out of the content types so I can get them from Zope is:

obj._meta.fields

...that gives you the Django fields. One thing that is important is that the URL's which are great in Plone are maintained. For that there's a urlpaths model that keeps track of the url for that object. It uses Django signals and we rip off and grab the normalize function from Plone to make a nice URL like about/my-company.

Once that's done, we've got our objects in and accessible through the Django admin interface. Wee. The changing is pretty easy, all the fields are there in the Django interface, once you click on edit. Oh sigh if now we could just get Kupu in, which is still the best editor out there.

Next there was a piece of middleware which grabs the url such as about/test and grabs the relevant page in urlpaths. There's a bit of magic that grabs the object and then looks up a default view for that object. The view is a page template, yay, for example in document.pt:

        <h1 tal:content="object/title" class="documentFirstHeading">
          Title or id
        </h1>

Using the templates was easy, I was able to just drop main_template in, and then spend an ages ripping everything out that I wasn't going to build out tonight. And the end result is extremely unremarkable screenshot to our right.

Phew. Well it was an interesting exercise, it's certainly something that needs a great deal of work to continue before its useable, although might be an interesting tool on top of things like entransit. Certainly I feel that if I to start upgrading those Plone sites, I will be working on this tool instead.

Disclaimer: this won't work, if it does I'll fall over with shock. But here's some code:

http://svn.clearwind.ca/public/django/plone-django

This comment from the Django documentation was pointed out to me tonight and it winds me up a bit:

Why use a text-based template instead of an XML-based one (like Zope’s TAL)? We wanted Django’s template language to be usable for more than just XML/HTML templates. At World Online, we use it for e-mails, JavaScript and CSV. You can use the template language for any text-based format.

Oh, and one more thing: Making humans edit XML is sadistic!

There's a few things in this comment that annoy me.

Most websites are written as HTML. Almost every site these days is XHTML. And XHTML is "A Reformulation of HTML 4 in XML 1.0". At this point yes I'm aware, XHTML isn't quite XML as this article points out, but it's as near as damn it. Everyone writes XHMTL these days. In fact look at the heading HTML of the Django admin interface:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
 lang="en-us" xml:lang="en-us" >

So how can you write an interface in XHTML and claim that forcing people to write in XML is sadistic? The difference is so small to be neglible..

Next, you use it for "e-mails, JavaScript and CSV". Excellent, I put together sites using XHTML, Ajax (XML), RSS (XML), e-mail, Javascript, CSV's and PDF's. I've almost never, in the many sites I've done, started formatting Javascript, CSV's or PDF's using a templating language - the only exception I can think of is DTML which CMF and Plone (and I didn't write it).

In fact in Plone e-mails are formed using Page Templates, here's a snippet:

From: "<span tal:replace="portal/email_from_name" />" 
<span tal:replace="structure lt"/><span tal:replace="portal/email_from_address" />
<span tal:replace="structure gt"/>

I would accept that is a little complicated and personally I've always found using python string formatting (the % operator) more than sufficient. So we don't need Django templating for that either. But let's face it, in most sites I have done, 95% of the templating demands are for HTML. I bet yours are too. So focus on what you need to do the most and build good tools that allow you to do it fast and simply with great flexibility.

Throwing out a great tool for an edge case, which it allegedly can't meet, is not a good answer. There's better reasons for not using Page Templates and I bet in any feature comparison, Page Templates would beat Django templates. Just say "I don't like Page Templates, it's ok".

This release adds in:

• Increased performance, but not recompiling on every request and utf-8 templates (thanks Mr Topf)
• More detailed errors through SimpleTAL patch

Download: simpletemplate.0.5.zip

Django provides flatpages but these won't inherit from your page templated pages, only the Django templating language one. Talflatpages is a very simple app that copies completely from flatpages, but changes the default template to be a tal one, using main_template in a macro (I'm sure you can change this to suit your site).

Only a small project so not bothering with a release, grab it from SVN here:

 http://svn.clearwind.ca/public/django/talflatpages/

Update: got the URL for SVN wrong, corrected.

There's a feature in Django to add in new values to a user record, use a profile. Here's an example: http://www.b-list.org/weblog/2006/jun/06/django-tips-extending-user-model/. So first thing I did was write a unit test for it, which failed.

This got me confused for a minute because I thought that when you created a user, it created the profile for you automatically. Duh! So for the record here's a simple unit test, create a user, set the company (a field on the Profile object), retrieve it and check the company is still there.

import unittest
from models import Profile
from django.contrib.auth.models import User
from django.contrib import auth

class TestSecurity(unittest.TestCase):
    def setUp(self):
        user = User.objects.create_user(username="bob",
                                        email="j@j.com",
                                        password="pwd")
        user.save()
        
        profile = Profile.objects.create(user_id=user.id)
        profile.company = "Test"
        profile.save()

    def testProfile(self):
        user = User.objects.get(username="bob")
        assert user.get_profile().company == "Test"

This tutorial is a re-writing of the Django Tutorial in the documentation on the Django website. In this I step through the same tutorial covering the templating sections, but focusing on how they would look using Page Templates instead of the Django templating language.

The demo code and so on is from that site and all credit to the authors of that tutorial. As ever all mistakes are mine.

To run the code in this demo, you'll need Simple TAL and Simple Template. Check out Simple Template from here and follow the readme contained within that directory:

svn co http://svn.clearwind.ca/public/django/simpletemplate

If you don't fancy typing everything in, then please checkout code used in these examples from:

svn co http://svn.clearwind.ca/public/django/polltutorial

Already to go. Then start following through the tutorial and through parts one and two. Neither of these touch on templating at all so nothing relevant to us there. Tutorial three is where the fun starts to happen, in the section Write views that actually do something.

Simple template provides key api calls for rendering templates, that you can use instead of the builtin ones. Instead of:

from django.template import Context, loader

Use:

from django.contrib.simpletemplate.public import get_template

Instead of the given example, use:

from django.contrib.simpletemplate.public import get_template
from django.template import Context
from mysite.polls.models import Poll
from django.http import HttpResponse

def index(request):
    latest_poll_list = Poll.objects.all().order_by('-pub_date')[:5]
    t = get_template('polls/index.pt')
    c = Context({
        'latest_poll_list': latest_poll_list,
    })
    return HttpResponse(t.render(c))

The key changes here are line 1, with that different import. Also on line 8 we are no longer using index.html, we are using a Page Template which follows a convention established from Zope of .pt as extension so we'll call ours index.pt.

The index.pt looks like the following:

<tal:block tal:condition="latest_poll_list">
    <ul>
        <li tal:repeat="poll latest_poll_list"
               tal:content="poll/question">question</li>
    </ul>
</tal:block>
<tal:block tal:condition="not: latest_poll_list">
    <p>No polls are available.</p>
</tal:block>

If you are familiar with TAL then this will hopefully make sense. The next step covers using "A shortcut: get_object_or_404()", this is relatively straightforward, to use the render_to_response replace the current import with the following:

from django.contrib.simpletemplate.public import render_to_response

Just remember to point to a .pt template, not a .html one.

And finally in tutorial three there is one more example that covers the detail view. Here's how detail.pt looks in Page Templates:

<h1 tal:content="poll/question">Poll Question</h1>
<pre>
<p tal:condition="error_message|nothing">
    <strong tal:content="error_message">Error</strong>
</p>

<form action="" 
         tal:attributes="action string:/polls/${poll/id}/vote/" 
         method="post">
    <tal:block tal:repeat="choice poll/choice_set/all">
        <input type="radio" name="choice" 
                tal:attributes="for string:choice${choice/index};
                value choice/id" />
        <label tal:attributes="for string:choice${choice/index}"
                tal:content="choice/choice">The choice<label>

And that's it, on to tutorial four in another day.

    

Yesterday saw the beginning of the end of our Rails applications as we started moving them over to Django. A simple one was our timesheet application (yes we have one internally) that was in Rails.

The rails application was around 1000+ lines of code (just in the app directory) of which I imagine a lot was the boiler plate scaffold. It used a plugin we developed called Toffee which did the user interface and then one for pretty user interface. It had quite a few bugs, took 2 days to write (which I still find hard to believe).

The django application (with the following exception) took 1 hour to write and had about 80+ lines of code (mostly because I write with lost of new lines). No bugs so far. The tricky bit that took me 2 hours is a) changing django to trust REMOTE_USER sent from our Apache which does LDAP auth and b) figuring out how to set this user on the timesheet model. I'm still not happy with the solution to the latter and need to research, I think it should be a manipulator so the admin interface uses that.

Today we worked on moving an old Zope application to Django, looking good so far.

It seems a rather big omission, so if I've missed it my apologies. But there doesn't seem to be any standard logging framework in Django, it's in the write your own land. So I've got a module, simpletempates that I put out there (don't think anyone uses it) and that sometimes wants to tell the person developing the site things like your template is not compiling or cannot be found, useful stuff. There isn't any standard way of doing this as far as I can tell. Just found lots of posts on "wouldn't it be nice if...".

The RSS feed for this blog has been a bit wonky in Safari for a while. Specifically Safari would randomly pick dates for posts, randomly showing old posts as new ones. I think the problem was I was missing a timestamp on each item. This was as simple as adding in:

    def item_pubdate(self, obj):
        return obj.timestamp

...on to my feeds class. Apologies if this has been messing your feed. Safari and Shrook now seem much happier.

This release includes quite a few changes including ones that break backwards compatability, please read the docs. It adds in:

• Support for django's default filters
• Easy access to MEDIA_URL
• A few shortcuts from normal templating like render_to_response

Download: simpletemplate.0.3.zip

Django applications have this thing that is very useful called MEDIA_URL specified in the configuration, that allows you to have a pointer to your static files, your css, js, images and so on. This is nice because it basically assumes that Django won't be responsible for serving your static stuff out to the world. Let Apache, IIS or whatever do that.

So in your templates, you can link to the MEDIA_URL to point to those files. On my development laptop it would be say localhost on my production server www.agmweb.ca/somethingorother. But wait the MEDIA_URL setting isn't available in a template, through a tag or anything. You have to change your views to use RequestContext and use a context preprocessor. If you do this at a beginning of a project its about 5 lines of code as detailed here.

So repeatedly people keep opening bugs on this eg: http://code.djangoproject.com/ticket/2532 and http://code.djangoproject.com/ticket/1278 and I can understand not having a template tag, or having all the settings exposed. But this is a useful tag, designed for this purpose, but hard to use. Why? There's no real explanation:

"Im marking this as a wontfix due to the slippery slope potential. If we added this, then people would want a context processor for another setting, and for another setting, and for another. We've already ruled out the idea of a "settings" context processor (see [1278]), so I'd recommend that people write their own two-line context processors if they really want this stuff."

People do seem to want it a lot, so it's being added to SimpleTemplate, the advantage of being able to do my own stuff. If there is another way people change the links of their static files, please let me know.

Django 0.96 is out. Get those shiny new features you know you want including the things I wanted: newforms, tests and fixtures.

Django now ships with a comprehensive set of testing tools. You can write tests based on doctest or unittest, test your views with a simple test harness, and load initial data ("fixtures") automatically.

Django 0.96 also ships with a brand new forms library, django.newforms. This is a replacement for django.forms, the old forms/manipulator/validation framework

• Release notes
• Announcement
• Download

Simple Template 0.2 is released. In a completely unprofessional manner I can't remember what I changed. Just that as I've been using it more and more I've made some changes. Still need to get a nice logging error mechanism in there, but the error handling is improved.

Simple Template yets you use SimpleTAL in Django.

Download here: simpletemplate.0.2.zip

After a little bit of confusion the first Norweb meeting got off successfully with a talk on Django followed by beers. A good time was had, I promised to post my slides and here there are as a PDF (I have to get S5 running here, on the bottom of a very long list.

Django - an Introduction, talk given by Andy McKay to Norweb Thursday 15th, only made sense if you were there really.

Good article on signals in Django:

Every model object used in a Django application will send these signals. And the beauty of it is that any one request can listen (connect in Django-speak) for a signal on one, more than one, or all of the object types in an project.

I'm sure someone has done this before, but I couldn't find it in searching. simpletemplate is a library for Django that allows you to use SimpleTAL in views. You can use TAL, metal in Page Templates and render them back. At this current time of writing, this site is using them.

Found I was getting this error in Django running tests:

Got an error recreating the test database: must be owner of database test_clearwind 

Did a bit of googling and took a while, so in case I get it again the answer is to run the following in psql:

alter user django createdb;