Andy McKay's blog

I've seen this happen quite a few times and each time it makes me want to bang my head on a table. I saw it again today and it's just so frustrating yet it seems so basic. The scenario is simple:

• you have a database that has some records in it
• you want to display some records on a web page
• you want to paginate through the records

The developer throws a SQL query together like:

select * from log;

Someone throws together a pagination method that looks for the number of results in that record set and then splits it up. You throw together a nice UI to display it. To test it, you set the pagination at 20 records and put 45 records in the database, beautiful and it's all done.

Fast forward a few months... site has come to a crawl. Your log table now has 10 million records in it. To display 20 records on the screen, how many records is the database getting back for you, 20 or 10 million? Again and again, I've seen it where people return 10 million records and then throw all but 20 of them away.

The overhead of producing all those records, sending them over to your application server and then doing whatever needs to be done is of course completely pointless. It's not hard to stop this:

• paginate your SQL correctly, for example Postgresql has LIMIT and OFFSET
• use an ORM that evaluates lazily ensuring that it only queries once you've told it you want 20, for example in Django
• ZSQL Methods in Zope (I don't think many people use them anymore) have a limit of 1000 records to prevent this, if you are upping that number you know something is wrong.
• ZCatalog in Plone is generally ok since it's quite lazy, but there is an overhead. There's an options called limit and sort_limit ("An optimization hint to tell the catalog how many results you are really interested in"), your mileage may vary there.

And finally if you have a data source that you can't paginate, you have to ask yourself if you can apply the pagination style interface to it. One example was I was asked to create a browse interface to an old LDAP installation. This version of openLDAP did not have an offset meaning that there was no easy way to do it. Ugh.

So if you hear someone say "just list all X here" or "can you just add in pagination" think about the scalability down the line.

Today was a frustrating day as Citrix kept messing me around. I found that if I ran Windows in a VMWare on my Mac, the client there ran much better. The only problem was finding the Windows client. So I've got an .ica file, you need to run. So you end up at the following page, where next?

One easy place? Yeah right, it's not obvious or easy. Answer it's Citrix XenApp (the new name for Presentation Server). As someone who doesn't know what all the product names mean, there's no way for me to know what the hell to do next.

The cost to the taxpayer of abandoned Whitehall computer projects since 2000 has reached almost £2bn

Nothing new in here, but this project:

It was being used to report around 30,000 crimes a year but began to falter when defence technology company, Qinetiq, was brought in to build a replacement in 2005. Costs began to spiral and the new system was deemed to be unfit for purpose, resulting in suspension of the service last March and final closure in December. The deal is now subject to a legal dispute, with Qinetiq claiming that it has completed the work it was contracted to do.

Reminds me of this project:

This is in sharp contrast to conventional contracts which attempt to pass on risk – or rather, as Mathew Riley, the company’s commercial director for Terminal 5, ‘the financial consequences of risk’ – to contractors. BAA realised that such contracts do not really work because ultimately either the risk would always fall back on itself, or the attempt to pass on risk would result in massive legal claims and counterclaims.

Although Heathrow is possibly as close as you can get to hell on earth, they do have something to teach us.