Andy McKay's blog

One thing I've enjoyed doing the last year is taking a bit of a step away from being blinkered solely on Plone and Zope and take a look around at other things out there. And no surprise - all these fancy new frameworks out there don't actually solve some problems. The exact same problems and patterns exist out there. All too often in IRC channels or from managers I've heard an answer like "so if move away from Plone we'll fix this problem". Chances are you won't, chances are other framework's have the same problem. Take for example the "pulling large amount of data from a remote source" situation. I've seen this now in Plone, Zope, Ruby on Rails and heard about it Drupal. The problem is simple, you've got a source that contains a large amount of records, such as a SQL server or and LDAP server. You write code to show the first 20 users and but the rest behind next links. Works with 10, 20, 200 users. Now throw 5 million in there and what happens. It falls over. It's a pattern that developers (like myself) need to learn and in LDAP's case its a limitation of the tool. OpenLDAP has recently started to support this kind of "give me records 200-300 out of this possible 1,000 records" but no fiddling on the framework is going to change that. I was in a Drupal birds of a feather and I remember hearing them say "oh yeah when our LDAP server has 20,000 users in (number made up) the admin tool for browsing gets really slow". Yep same thing. Many times Alan Runyan is right, one talk in particular that stuck in head was a talk at the last New Orleans conference. At that he was asking people to focus on reusability, not re-invention (although his point was probably deeper than that). And don't get me started on how Django is crippling itself with re-inventing yet another templating language and simultaneously natching defeat from the jaws of victory. Side note: Just to compound the problem in the Ruby on Rails app was that it was looking for a change on the input box every second and firing off a request to the server, returning a huge recordset every time. There's a reason Google doesn't have this on their home page you know. Okay rant over, on with the show.

Sorry. It's been about 6 years (eek) since I first went to Zope so it's hard to remember. These last few days I've been fixing up a Ruby on Rails application. More on that in later posts. Of course fixing up a complicated app as your first dive into any framework is the worst thing that you can do. Unfortunately I think this is what commonly happens to most developers. I've spent ages fighting my way through implicit variables, variables that don't exist in any source file but magically appear, Ajax that will never work in IE, badly thought out solutions relating to performance, lack of documentation, multiple ways of defining the same thing and well, Ruby. I can sympathise with new Zope users who probably say the exact same thing. It will get better, everyone's learning these lessons. Zope 2 get's better as you learn more, I'm sure Ruby on Rails will too. Stick with it.

This came out last week and sparked quite a few blog posts. I spoke to a few people on the weekend and still not sure what to feel about it really. In the end I hope it all works out for them. All I can really remember for some bizarre reason is an episode of the Wonder Years. The main character is working in a hardware store and wants a pay rise, but doesn't get one. The man running the store is an old friend (I think, it was many years ago). In the end after not getting a pay rise he leaves the store and goes to work in burger joint. Of course there is much heart wrenching, soul searching and so on. At the end in typical Wonder Years fashion there's soppy music and sad feelings as this man feels rejected by the kid leaving. Nobody notices him joining the burger joint and sure enough when he leaves a year or two later, no one cares less. The Java community won't blink an eyelid as Nuxeo joins. The Java community certainly won't care if Nuxeo leaves. But the Zope community will miss you.

Now features context sensitive Python prompt. Not sure about the API of an object? Just click the little Clouseau button and a Python prompt will appear with context mapped to the actual object you are looking at. Other changes:

• Add in utils library, for things like utils.commit()
• Cleaned up tooltips, no longer raise an error
• New tooltips code from pyShell
• Changed icons

Note: because I have no Windows computers anymore this is not tested on IE. May work, help needed for that. Upgrades: do an install and uninstall from the add/remove products section of the Plone control panel to get the context sensitive help. Download: Clouseau.0.3.zip

I was playing with dir in Clouseau today. The problem is that in all tools this useful function returns around 600-900 when run on a Zope object. The reason I thought was acquistion and then I realised it wasn't its just Zope 2's huge inheritance tree. So how can this be more useful?

• Integration with DocFinderTab to find say the dir of the top 4-5 or five classes, most of the time you don't need to go too deep.
• Easy introspection of an objects Schema for example:

  >>> utils.schema(context)
  [ 'setDescription', 'setTitle', 'Title',]
  
  

Any other things that are possible to do? ie: give me code or sane ideas please

I just completed the accounts for the the conference and sent the money to the PSF. As per the blog post:

The Vancouver Python Zope User Group has donated the proceeds of the 2006 Vancouver Python Workshop, amounting to CAD$5770, to the Python Software Foundation. The PSF thanks the user group for their support.

Thank you to everyone for coming. After two conferences, we are now totalling almost $10,000 in donations to the PSF.

Thanks to a helpful post from Sidnei, I'm almost there. I've got two zope.conf files now, a zope.conf and a zope-full.conf. The former has logging minimised, and deprecation warnings blocked.

<warnfilter>
    action ignore
    category exceptions.DeprecationWarning
</warnfilter>
<warnfilter>
    action ignore
    category exceptions.Warning
</warnfilter>

<eventlog>
  level error
  <logfile>
    path $INSTANCE/log/event.log
    level error
  </logfile>
</eventlog>

The latter is the default zope configuration file. I then copied runzope to runzope-full and pointed it to the new config file.

CONFIG_FILE="/opt/Plone-2.5/Instance/etc/zope-full.conf"
...
exec "$PYTHON" "$ZOPE_RUN" --configure=$CONFIG_FILE "$@"

So now I can:

• Do runzope and get only errors when the occur, here's a sample:

  /opt/Plone-2.5/zope/etc $ ~/plone-bin/runzope
  2006-09-14 10:33:54 ERROR Zope.SiteErrorLog http://localhost:8080/test/sampleError
  Traceback (innermost last):
    Module ZPublisher.Publish, line 115, in publish
    Module ZPublisher.mapply, line 88, in mapply
    Module ZPublisher.Publish, line 41, in call_object
    Module Shared.DC.Scripts.Bindings, line 311, in __call__
    Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec
    Module Products.PythonScripts.PythonScript, line 323, in _exec
    Module None, line 1, in sampleError
     - 
     - Line 1
  ZeroDivisionError: integer division or modulo by zero

• Do runzope-full and get all the errors, useful for testing before a release or debugging.

However one left on the plate: zopectl debug does the latter:

~/plone-bin/zopectl debug
Starting debugger (the name "app" is bound to the top-level Zope object)
2006-09-14 10:52:07 WARNING Init Class Products.ATContentTypes.content.base.ATCTFolderMixin has a security declaration for nonexistent method 'manage_copyObjects'
2006-09-14 10:52:08 WARNING Plone Deprecation Warning 
The CMFPlone.MemberData class will be removed in Plone 3.0
2006-09-14 10:52:08 WARNING Plone Deprecation Warning 
CustomizationPolicies are deprecated and will be removed in Plone 3.0. Please use GenericSetup extension profiles instead.
2006-09-14 10:52:08 WARNING Plone Deprecation Warning 
registerSetupWidget is deprecated and will be removed in Plone 3.0.
2006-09-14 10:52:08 WARNING Plone Deprecation Warning 
registerSetupWidget is deprecated and will be removed in Plone 3.0.
2006-09-14 10:52:09 WARNING Init Class Products.PloneMultisite.MultisiteTool.MultisiteTool has a security declaration for nonexistent method 'inheritDefaultSubscriptions'
2006-09-14 10:52:09 WARNING Init Class Products.PloneMultisite.MultisiteTool.MultisiteTool has a security declaration for nonexistent method 'inheritDefaultSubscriptions'
2006-09-14 10:52:10 WARNING Plone Deprecation Warning 
CustomizationPolicies are deprecated and will be removed in Plone 3.0. Please use GenericSetup extension profiles instead.
2006-09-14 10:52:10 WARNING Plone Deprecation Warning 
registerSetupWidget is deprecated and will be removed in Plone 3.0.
>>> 

...and the same with zopectl run. Still needs a bit more work.

Just added tooltips to Clouseau, so it shows tooltips on ( and removes it on ). Think I need to make the UI a little more funky and I'm sure the tooltip collection code could be improved. Here's the obligatory screenshot as well. Download: Clouseau.0.2.zip Note: only tested on Firefox, Zope 2.9 and won't work with CacheFu.

I don't know about you, but the deprecation warnings when Plone starts up are getting kind of annoying. Here's the trimmed down version:

2006-09-11 22:44:25 WARNING Init Class Products.ATContentTypes.content.base.ATCTFolderMixin has a security declaration for nonexistent method 'manage_copyObjects' 2006-09-11 22:44:28 WARNING Plone Deprecation Warning The CMFPlone.MemberData class will be removed in Plone 3.0 2006-09-11 22:44:30 WARNING Plone Deprecation Warning CustomizationPolicies are deprecated and will be removed in Plone 3.0. Please use GenericSetup extension profiles instead. 2006-09-11 22:44:30 WARNING Plone Deprecation Warning registerSetupWidget is deprecated and will be removed in Plone 3.0. 2006-09-11 22:44:30 WARNING Plone Deprecation Warning registerSetupWidget is deprecated and will be removed in Plone 3.0. 2006-09-11 22:44:37 WARNING Plone Deprecation Warning CustomizationPolicies are deprecated and will be removed in Plone 3.0. Please use GenericSetup extension profiles instead. 2006-09-11 22:44:38 WARNING Plone Deprecation Warning registerSetupWidget is deprecated and will be removed in Plone 3.0. .. and a few removed (see below)

Further if you set etc/zope.conf to only show errors and above you still get a few:

/opt/Plone-2.5/lib/python/ZServer/utils.py:33: DeprecationWarning: The zLOG package is deprecated and will be removed in Zope 2.11. Use the Python logging module instead. LOG('ZServer', severity[type], message) /opt/Plone-2.5/Instance/Products/Marshall/handlers/__init__.py:38: UserWarning: libxml2 not available. Unable to register libxml2 based marshallers warnings.warn('libxml2 not available. Unable to register libxml2 based ' /opt/Plone-2.5/Instance/Products/CMFSquidTool/Permissions.py:19: DeprecationWarning: The module, 'Products.CMFCore.CMFCorePermissions' is a deprecated compatiblity alias for 'Products.CMFCore.permissions'; please use the new module instead. from Products.CMFCore.CMFCorePermissions import ManagePortal /opt/Plone-2.5/Instance/Products/CMFSquidTool/__init__.py:38: DeprecationWarning: The product_name parameter of ToolInit is deprecated and will be ignored in CMF 2.0: CMFSquidTool icon="tool.gif", /opt/Plone-2.5/Instance/Products/CacheSetup/config.py:23: DeprecationWarning: The zLOG package is deprecated and will be removed in Zope 2.11. Use the Python logging module instead. LOG(PROJECT_NAME, INFO, msg) /opt/Plone-2.5/lib/python/OFS/Application.py:835: DeprecationWarning: __init__.py of Products.ExternalEditor has a long deprecated 'methods' attribute. 'methods' will be ignored by install_product in Zope 2.10. Please use registerClass instead. DeprecationWarning) /opt/Plone-2.5/Instance/Products/MemcachedManager/__init__.py:8: DeprecationWarning: The zLOG package is deprecated and will be removed in Zope 2.11. Use the Python logging module instead. LOG('MemcachedManager', log_level, summary, text) /opt/Plone-2.5/Instance/Products/kupu/plone/__init__.py:32: DeprecationWarning: The product_name parameter of ToolInit is deprecated and will be ignored in CMF 2.0: kupu icon="kupu_icon.gif", /opt/Plone-2.5/lib/python/Signals/SignalHandler.py:39: DeprecationWarning: The zLOG package is deprecated and will be removed in Zope 2.11. Use the Python logging module instead. zLOG.LOG('Z2', zLOG.BLATHER, "Installed sighandler for %s" % ( /opt/Plone-2.5/lib/python/OFS/subscribers.py:74: DeprecationWarning: Products.CMFPlone.QuickInstallerTool.QuickInstallerTool.manage_afterAdd is deprecated and will be removed in Zope 2.11, you should use event subscribers instead, and meanwhile mark the class with DeprecationWarning) /opt/Plone-2.5/lib/python/OFS/subscribers.py:74: DeprecationWarning: Products.PluggableAuthService.PluggableAuthService.PluggableAuthService.manage_afterAdd is deprecated and will be removed in Zope 2.11, you should use event subscribers instead, and meanwhile mark the class with DeprecationWarning) /opt/Plone-2.5/Instance/Products/PluggableAuthService/plugins/BasePlugin.py:71: DeprecationWarning: isImplementedBy has been renamed to providedBy return interface.isImplementedBy( self ) /opt/Plone-2.5/lib/python/OFS/subscribers.py:74: DeprecationWarning: Products.PlonePAS.plugins.cookie_handler.ExtendedCookieAuthHelper.manage_afterAdd is deprecated and will be removed in Zope 2.11, you should use event subscribers instead, and meanwhile mark the class with DeprecationWarning) /opt/Plone-2.5/lib/python/OFS/subscribers.py:74: DeprecationWarning: Products.PlonePAS.plugins.role.GroupAwareRoleManager.manage_afterAdd is deprecated and will be removed in Zope 2.11, you should use event subscribers instead, and meanwhile mark the class with DeprecationWarning) /opt/Plone-2.5/lib/python/OFS/subscribers.py:74: DeprecationWarning: Products.kupu.plone.plonelibrarytool.PloneKupuLibraryTool.manage_afterAdd is deprecated and will be removed in Zope 2.11, you should use event subscribers instead, and meanwhile mark the class with DeprecationWarning) /opt/Plone-2.5/Instance/Products/Marshall/config.py:33: DeprecationWarning: The zLOG package is deprecated and will be removed in Zope 2.11. Use the Python logging module instead. LOG('Marshall', level, msg)

So first off, how do we supress these errors that are presumably using the logging module? Secondly how about using the power of the internet and making a web page that describes what to do. I don't know what marking the class actually means, but I'm sure if someone put up a page explaining it, and a link we could figure it out eg:

Products.kupu.plone.plonelibrarytool.PloneKupuLibraryTool.manage_afterAdd is deprecated see http://blogs.nuxeo.com/sections/blogs/florent_guillaume/2005_11_08_events-in-zope-2-9

(which btw the way is a useful blog post on the subject, hiding down there in the Google results) Do I really need to see all this? Can we put this into a unit test that will do something like test --show-deprecations? Deprecation warnings in products I develop are useful, but I fear, I like most or just ignoring these logs. If that's the case we are losing the battle.

The quality of news is not very high, slashdot is proving much better most of the time. Take this article: http://digg.com/apple/Macbook_Outselling_iPod

I noticed that the iPod video [#4 top seller] is being outsold by the MacBook [#2 top seller after the nano]

that ended up on the front page:

1. Wait a second, the top seller is the iPod, its the iPod Nano. The headline is that MacBook outsells the iPod. It may sell more at that one instance than the iPod video, but in the same sentance you are saying the iPod Nano is top. Your headline is totally wrong.
2. Where does it say on the store page that the order implies well anything. Its says: "Top Sellers: iPod nano, MacBook etc". There is nothing to tell you the order, that's implied.
3. How long is this data based on, 10 minutes, 1 day (did someone just order a lot of MacBooks, eg a company).

Just annoying you see something on the front page like this, but its a) wrong and b) questionable data even if it was right. I've got nothing against the person reporting it, just the people who float it up to the top of digg. And this is just one of many items on there.

There's been lots of chatter on Slashdot and digg about Ajax services. Mostly thanks to the questionable linking to COWS which has ganered press for some reason. One of the main responses went along the lines of "I wouldn't trust some external web site for that part of my site, what if it goes out etc...". That's a fair question, but it's focused on the wrong line of thinking... Yes, running your multi-million dollar website with a dependency on my ClearWind service is a mistake. I have no resources to monitor or maintain it and I might change it on a whim to start charging you for all your traffic. However I would say that services from Google, Yahoo, Flickr, Apple, eBay will all likely put more resources into keeping their site up and running than any site I will be rolling out over the next few years. So, pick a trusted server for your service. Bear in mind the Plone AjaxProxy does caching, another advantage of running through a local proxy. If the destination server goes out temporarily the cache might be catching that. The main use case for Ajax services harks back to a scenario I had at Enfold. In a scenario when you have 20 Plone sites with your software installed and you want to roll out a new service to them - you have a choice, install new software on every client or run a service and have all the servers consume it. Yes, the latter does involve installing software on each installed, but it's an easier install. Take the scenario for image manipulation, you normally have a library that can do a series of manipulations on an image. This then means installing PIL, ImageMagick or so on a server. It's also something that occurs rarely, you do a few manipulations and save it. With an Ajax service, you install the client part of the image manipulation, the Ajax onto the Plone site. You install the libraries on the server. That's a much easier install. You can then test and maintain the Image manipulation service on the server. The advantages are the usual one's you'll get from a service oriented architectures: - each service has a clear API and division from the rest of the software - each service is easily testable - each service is easily supportable and maintainable - each service is cross application - software is easier to install on the client The dependence is on your software and services, not on other people.

From Rod Hyde:

There are many programming fads, language fads, framework fads, design fads, architecture fads, management fads and project management fads. Guess what. None of them matter unless they get the job done. So, pay attention to them, because sometimes they have useful aspects, but don't let them take over how you do things. There are no silver bullets.

My MacBook has been getting more and more sluggish. One of the things that has been cropping up is a large amount of time spent by spotlight, seemingly indexing content - in top the mds process has been taking too long. Just so I remember in the comments of this article is how to turn Spotlight off, don't follow the process in the article. Let's see if that and the 2 gig of RAM on it's way help.

As far as I can tell, Flickr is ready to roll as an Ajax service. It accepts requests using REST and then can respond in XML, using SOAP or XML-RPC. Once transformations are ready to roll on the server, we'll be able to insert the Flickr API key in the AjaxProxy as well. Go for it someone!

This just got onto Slashdot, COWS Ajax. Spookily enough this is exactly what I was blogging about last week or so. Sometimes things hit all at once. A quick peek at COWS and it has:

• A server side library written in PHP. I don't know much PHP so I can't comment on that.
• A client side library written in JS that connects to the back end.
• An example client side spell checker

To quote the site:

The trick is that the URL of the script is not pointing to a javascript file, but instead pointing to something like a PHP or other backend.

Looks like an implementation of Ajax Services to me. I would really like to see more implementations of AjaxProxy, so something in PHP would be really nice if anyone knows more than me it would be great to see.

• Stores my data in a relational database
• Provides a nice UI on that relational database
• Allows me to override the bits I want to change
• Is written in Python
• Is written in some XML compliant markup language eg XSLT or TAL or even Kid

I've looked at Django and TurboGears but never of these actually seem to fall down in the following ways:

• Django's markup language isn't to my taste (I can replace with TAL but I haven't got that working yet). It has a thing called generic views, but these don't actually generate HTML just the underlying Python glue.
• TurboGears doesn't seem to generate HTML either every tutorial I've found creates their own HTML as well.

I've rummaged around but haven't spotted anything else. Has anyone else seen anything they'd recommend? At this point I'm tempted to go nuts (client side XSLT and XML), go with framework and ripping out the stuff I don't need (Plone) or building lots from scratch (Zope 3). Bear in mind I've got a completely restricted user base, Firefox only so crazy things like client XSLT can work. Update: using Plone for prototype, I can do it faster and better in Plone than anything else at this point. The UI will be improved a lot for Firefox though.

ClearRSS is an RSS news feed parser written in Ajax, that uses the AjaxProxy to pull RSS feeds from remote sites and display them in your site. This is a quick 0.1 release, I haven't tested it in IE but it worked for me in Firefox. Volunteers to get this to a 1.0 welcomed. RSS feeds are sped up dramatically by local Zope caching of them. This follows on the Ajax Services theme. Note: that the RSS parsing code is actually re-used from another non-Plone project and then customised which is why it only took and hour to write. Download: ClearRSS.0.1.zip