Andy McKay's blog

Taken at Morecambe beach, 2007

A great metaphor from the Wire for how messed up the police department was in the series. This never happens in any company. Warning: NSFW without headphones, the Wire has rather colourful language.

All the talk today is about Britain lowering its VAT rate. This is meant to stimulate us into jumping out of seats, driving down to one of the generic high streets full of the same generic shops and filling our cars and houses full of crap. It seems the answer to all our economic woes is for debt laden consumers to go out and buy more stuff.

I can see a reduction in prices helping to stimulate things a bit, but one of the things I really noticed coming back to the UK is the vast amount of out of town shopping areas that have sprung up. All selling the same rubbish. How much of this do we really need? At some point perhaps we could all just stop buying junk, buy food only and let the economy adjust and the environment breathe a sigh of relief.

As for me I'll renew my Christmas Manifesto and my Pledge to buy hand made again this year.

Canadian reporting on the housing prices (something I follow regularly) is terrible. 90% of the reports include a quote from an estate agent, or:

Despite the fall in sales, board president Dave Watt (of Real Estate Board of Greater Vancouver) remained upbeat about the Vancouver real estate market.

Fortunately it seems this trend is slowing down as perhaps even they have had little positive to say. Or perhaps the media have realised asking the salesman if you should buy their product is a bit stupid. But this one in the Vancouver Sun (who by the way, have got a terrible website) got me laughing:

"It's been a little bit surprising the consequences of the subprime crisis have been so far-reaching," said Hanley, 48, who has cut his asking price to $3.99 million

Add us to the list of people not buying that one then.

What I've seen about the current discussion focuses on giving the auto manufacturers lots of money because that's cheaper than them going under. Some comments on that:

• If they are looking for $25bn in a bailout, that won't last very long - GM alone, lost $39bn in 2007. So how long will that $25bn last?
• If they are looking for that much, you know they will want more. They will become dependent. Does the government get to take it over? Does the management change?
• This is a problem that's been in the making for a long time. From slow sales to pensions and healthcare GM is a troubled company:

  In fact, per car, GM is now spending more on healthcare than it does on steel.

  Rising benefits burden (2006)

I see these as businesses that are inherently bad and going to fail. The real question is do you spend money now and keep them alive or wait for them to fail in a few years. I haven't been able to find figures on actual volumes of sales, but here's GM share prices, notice a decline since their peak in 1999:

Yes it is precipitous in the last few months, but this is more than just a shock, it's a crap business. You might help them ride out the storm for a month or two, but they will be going under until they can get their union under control, focus on cars people want instead of crap, cheating or just plain disgusting cars.

Note: yes the PT Cruiser is by Chrysler and I mostly talk about GM. These three companies are generally not good corporate citizens and I lump them together.

Shane Hathaway is a great Zope developer from many years ago - well respected for great code. After four years of doing other stuff, he's just posted that he's back doing Python, Zope and Plone, that's great news.

This has cropped up a couple of times on irc, so here's a recipe I have for restarting my Django instance during development. I'm not the sort of person who takes a long time to think it all through and can write a model and call it done, I iterate quickly and often. For the quickest way to do that during the initial development, I drop the db, re run syncdb and import my fixtures regularly.

Once a production version is out and running I then get rid of this script and do migrations.

There's no easy way from the command line to create a user neither syncdb --noinput or createsuperuser --noinput accept a password. Leaving lazy people like me wanting. I used to use pexpect but I find just creating a user object nice and easy. So for a project called collector, the following works well for me:

#!/usr/local/bin/python
import os
os.environ["DJANGO_SETTINGS_MODULE"] = "settings"

# dump the data to fixtures
os.system('python manage.py dumpdata collector --indent=2 > collector/models/fixtures/data.json')
# drop and recreate the db (this is for postgres)
os.system('dropdb authy')
os.system('createdb authy --encoding=UTF-8')
# sync the db
os.system('python manage.py syncdb --noinput')

# create a super user
from django.contrib.auth.models import User
u = User.objects.create(
    username='xxx',
    first_name='',
    last_name='',
    email='amckay@bluefountain.com',
    is_superuser=True,
    is_staff=True,
    is_active=True
)
u.set_password('xxx')
u.save()
print "User account created"

# load the fixtures back in
os.system('python manage.py loaddata data.json')
# run the server
os.system('python manage.py runserver')

I can then quickly get in the cycle of creating a model, entering data in admin interface, dumping it to a fixture and then writing unit tests.

Last week a new version of Clouseau came out that fixes issues that occurred with using Clouseau with buildout, namely a locals directory didn't exist and users didn't have permissions to save to the local directory. These should both be fixed and hopefully the python setup.py incantation worked so it's on Pypi correctly.

Clouseau on Plone.org

This is a simple rip off of the DCWorkflow that Plone uses, DCWorkflow originally coming from Digital Creations many moons ago. Anyway I always liked it because its very simple and abstracts quite a few things away from the object, letting admin's decide how things happen. The goal of this state workflow is to:

• put any number of workflows on a model
• the workflow can contain mulitple states
• there can be multiple transitions between the workflows
• conditions (well scripts) are checked before a transition is run
• scripts are run in the transition

Workflows can be constructed through the admin interface, or through scripts. The admin interface is a little rough and you can't actually workflow things in it. Since it doesn't check which scripts can be assigned to which workflow, or look for changes, but since I don't want to use the admin interface, I'm not too worried about that.

Users don't actually get to write any scripts. Scripts are created by the admin and given descriptions, the admins can then decide which script to assign to which transition and when. There are two kinds of (optional) scripts:

• permission: this is to check the object can be transitioned and gets passed the object and the transition. If you want other things like request or user, you might have to use threadlocals or other hacks to get them. If any of the assigned permission scripts does not return True (or anything evaluating as True), then it fails.
• do: this is what happens when the script is run and involves changing things, logging, emailing that sort of thing. Script gets passed the object and the transition. Returns ignored, if there's a problem raise an error.

The unit tests explain this better than some text, but here's a brief synopsis. Supposing a news item workflow, three states: private, pending, published. Someone in the office needs to review it before it goes live. Also all the fields have to be filled out.

Here' an example permission script assigned to the review transition, ie you can't workflow to review until you fill these out:

def check_complete(obj, transition):
    for field in ["title", "description"]:
        if not getattr(obj, field):
            return False
    return True 

So here's our news item:

                                       
from stateworkflow.models.state import State

class NewsItem(models.Model):
    title = models.CharField(max_length=255)
    description = models.TextField()
    published_date = models.DateTimeField(blank=True, null=True)
    state = models.ForeignKey(State)   

You can do all this manually (see unit tests) or try the following WorkflowManager a nice wrapper around it all:

from stateworkflow.workflowmanager import WorkflowManager

self.one = NewsItem()
self.one.state = Workflow.objects.get(id=2).default_state 
# at the moment we can't transition
self.one.title = "Hello"
self.one.description = "This is a description"
self.one.save()

# we should now be to able transition
wf = WorkflowManager(self.one)
wf.do_transition("Request review")
# we are now pending
assert wf.get_state() == self.one.state
assert wf.get_state().name == "Pending"

For more its in SVN at:

svn co http://svn.clearwind.ca/public/django/stateworkflow

Hope its helpful.

Microsoft Corp. today announced the next generation of Windows Live, an integrated set of online services that make it easier and more fun for consumers to communicate and share with the people they care about most

Consumers today are creating online content and sharing it in many places across the Web

Well done Microsoft PR, if that does turn me off I don't know what will. I guess I don't want to communicate and share with the people I care about, since I'm not a consumer, I'm a person. Dammit. I can see why you want me to use your service.

Cost of cancelling our broadband:

• 20 pounds cancellation fee (which they just added a month ago)
• 14.99 pounds hook up fee which is waived after 12 months, we haven't quite been with them for 12 months
• 19.99 pounds for last months service

So ~55 pounds. Thank you plus.net, Virgin was terrible too. Golly as much as Rogers suck, I didn't realise how good they were in comparison. All ISP's and Telco's suck, it's one of those universal rules. Some just suck more than others.

I've seen this happen quite a few times and each time it makes me want to bang my head on a table. I saw it again today and it's just so frustrating yet it seems so basic. The scenario is simple:

• you have a database that has some records in it
• you want to display some records on a web page
• you want to paginate through the records

The developer throws a SQL query together like:

select * from log;

Someone throws together a pagination method that looks for the number of results in that record set and then splits it up. You throw together a nice UI to display it. To test it, you set the pagination at 20 records and put 45 records in the database, beautiful and it's all done.

Fast forward a few months... site has come to a crawl. Your log table now has 10 million records in it. To display 20 records on the screen, how many records is the database getting back for you, 20 or 10 million? Again and again, I've seen it where people return 10 million records and then throw all but 20 of them away.

The overhead of producing all those records, sending them over to your application server and then doing whatever needs to be done is of course completely pointless. It's not hard to stop this:

• paginate your SQL correctly, for example Postgresql has LIMIT and OFFSET
• use an ORM that evaluates lazily ensuring that it only queries once you've told it you want 20, for example in Django
• ZSQL Methods in Zope (I don't think many people use them anymore) have a limit of 1000 records to prevent this, if you are upping that number you know something is wrong.
• ZCatalog in Plone is generally ok since it's quite lazy, but there is an overhead. There's an options called limit and sort_limit ("An optimization hint to tell the catalog how many results you are really interested in"), your mileage may vary there.

And finally if you have a data source that you can't paginate, you have to ask yourself if you can apply the pagination style interface to it. One example was I was asked to create a browse interface to an old LDAP installation. This version of openLDAP did not have an offset meaning that there was no easy way to do it. Ugh.

So if you hear someone say "just list all X here" or "can you just add in pagination" think about the scalability down the line.