Updated Aspire list editor

It’s been a while since I made a blog post.  Today, however, there is something worth writing about.

XIP-1324 went live today in Talis Aspire.  That code number won’t mean anything to most people but it’s indelibly imprinted into my brain and will be for the foreseeable future.

Today’s release contained an updated version of our reading list editor and is the result of several months work.  I started it around Christmas and took it through many iterations of subtle tweaks and also quite a bit of scope creep from what was originally planned.  Nad dipped in and out as time allowed at the start and BrickRed took the reins a couple of weeks before the end to sort out the data conversions, squash a few bugs and give it a good test as I was diverted onto another project. So, thanks to all who helped out along the way.

The crux of XIP-1324 is giving users the ability to create/edit reading lists in draft and keep working on them until they are happy with the result and then publish them as live.  There may not be much visiblly different in the UI (some of the other features can be found in the associated release blog post) but it involved some complex changes to our caching mechanism and a total internal rework of how the list editor worked under the hood.

I also wrote the first version of the list editor over a year ago and that caused its own problems.  There was a  decision then to use rdfQuery – some of this was based on need – the ability to suck an RDF model out of an XHTML+RDFa marked up page and then save multiple changes to a reading list in one go.  This didn’r scale well however and performance was poor with extracting the model from the page, especially when we ended up sending hte old and new model to the server at save time to update the changes to the list.  Maximum practical list size then was about 100 items and even that was touch and go in IE7 – the most used browser by our end users.

The new version does things pretty differently.  The read-only version of the reading list is still marked up as XHTML+RDFa but when you go into edit mode it drops pretty much all the RDFa on the page but just keeps the subject URIs for the main list elements such as sections and items.  DOM size has been stripped back as much as is practical for now as the previous DOM bloat with RDFa and other superfluous markup was causing some of the speed issues.  The new editor makes small AJAX calls via a queuing mechanism whenever items are added, removed, editor or moved around on the list.  Keeping the updates in a queue allows us to let the user get on with their editing without blocking operations whilst ensuring that the changes are applied to the data for the list in the correct order.  The new version has been tried with a list just under 500 items in size and performs acceptably – even on IE7.  The JavaScript is still using Prototype.js/Scriptaculous behind the scenes – there’s probably still mileage in tweaking that or giving jQuery another go but, at the time, the draggable/droppable functionality in Scriptaculous was much closer to what we were trying to achieve with little customization that jQuery was.

It’s a shame I can’t point at a live runnining copy of the editor that’s open to the general pubiic but if you want to see some of the general reading lists then try out any of the live instances at Plymouth, Sussex, St Andrews or Nottingham Trent Universities.

It’s been a long journey but I am so very glad it is now live.

More IE7 CPU spiking

A while ago I wrote about an issue with CPU spiking in Internet Explorer 7, recently I encountered another instance of this that required a different fix. To recap, we have a rich-client drag-drop interface across two scrolling panes.  You can drag and drop within the main pane on the left (a nested list with 'sections' and 'items') and you can also drag things from a pane on the right onto the pane on the left.  A read-only display of this (without the drag/drop and right-hand pane can he seen here for reference. Last time, we had a CPU spike causing slow juddering drag/drop when there were were only about 7 or 8 items in the left hadn pane.  This got progressively worse as you dragged more items onto the list.  After trying various things from Steve Souders' excellent blog and other Googling it ended up with selectively switching elemts of the screen off until I could narrrow it down to see what was causing the problem.  It turned out to require a few CSS 'min-height's being set on a couple of attributes but I'm not 100% sure why.  I'm guessing it could be some manifestation of the IE 'hasLayout' problem and the min-height could have been the equivilent to the Holly Hack to coax the IE rendering engine into working properly. This time however the same symptoms manifested themselves but a different cause and different solution was found.   More selective disabling of page elements/events and head-scratching later this one turned out to be caused by an absolute positioned button in the top-right of the left-hand scrolling pane.  There was no need for this to be absolutely positioned so I just tweaked the CSS again to float it to the right of the list title and, hey-presto, IE7 CPU spike goes away and performance returns to normal. It's these kind of things that give IE the reputation it has with web developers…. 🙂   Love it or hate it, it certainly gives you some interesting issues to sort out.

Internet Explorer strikes again

Internet Explorer has a way of driving web developers up the wall.  Every version has its own glut of bugs and quirks, some just irritating and some downright bizarre.   It's one of those bizarre ones that I've been looking at over the last few days – specifically in Internet Explorer 7, the major browser of our end users. The screen in question is an editable version (which I can't show due to authentication issues) of this reading list.  When it goes into edit mode a side panel slides in from the right and you get a library of bookmarks that you can drag onto the list in whatever position you want.  You can also drag in a new section or existing items and sections around on the list.  With the complexity of what is going on there were always going to be a few cross-browser issues but one in particular was so severe we had to disable editing in IE until the problem was found and resolved. The symptoms were that when new items or sections were dragged onto a list you'd get a flickering hourglass, CPU usage up to 96% and it would either be unreliable or impossible to drop it in the right place on the list.  If you started off with a new blank list, the first few items would be OK (up to about 4) then the hourglass flicker would start, by the time you got to 7 or 8 it was getting really hard to drop an item and by the time you hit about 10-12 it was impossible to use.  Firefox worked fine and Internet Explorer 8 (RC1, not tried in the newly released full IE8 yet) also worked fine. Several thoughts crossed my mind when first looking into this:  was it my (JavaScript) code, was it one of the third-party libraries being used (Prototype & Scriptaculous) or was it something else?  I certainly didn't imagine at the start that it was just down to the CSS… Once the code and the libraries were ruled out it ended up being a trawl of Google to see what I could find. The first thing uncovered was from a passing comment on Steve Souder's blog post on "Performance Impact of CSS Selectors".  Steve made passing comment to the performance impact on using CSS 'hover' pseudo-selectors.  We were using those on each item on the list to show/hide edit and delete buttons.  So, I removed the hover selectors and there was an improvement but no magic bullet. More digging around ensued and point 41 this list of IE bugs was found.  This was very close to what I was seeing.  A scrolling layer and a problem that maxes the CPU in IE7.   It then became noticeable that in IE7 I was reliably getting the problem at a certain point when dragging items onto the list.  When the number of items dragged over filled the visible space on the scrolling list, the next item dragged would always start to produce the flickering hourglass and CPU peaking.  I also noticed that the vertical scrollbar on the pane was not sizing correctly.  If you scrolled up and down using the scroll-wheel on the mouse the scrollbar would sort itself out and then next item you dragged across would work OK.   Interesting… More Googling later and I found a similar reference on the MSDN.  This described that problem and pointed at a CSS sizing issue causing a problem in IE.  I tidied up the CSS a bit and added a 'min-height' to a few of the selectors and, hey-presto, no flickering hourglass when dragging items across and no strange resizing scrollbar. So, if anyone else gets anything similar to this then I hope this post helps.  It's driven me nuts for a good few days now.   IE always surprises me (by how bad it is), but I never initially thought that some basic CSS could happily cause IE to go into such a bad CPU spike that it could actually crash the browser – amazing! Now that the problem of making it work in IE7 is well on the way to being fixed I now have to improve the performance on large lists (ABF203, above, is a tiny example compared to some we have).  It'll be back to Steve's great work in this field for pointers on this.

What happened to those old projects?

A couple of weeks ago I was out on a lunchtime walk around the Business Park and bumped into someone I used to work with a few years ago at Severn Trent Systems (about 200m from where I work now).    It seems that the project we both used to work on is still alive and well and it made me start wondering about other projects I've worked on; how long they lived for in these fast-changing times and how many still have traces of them being used today. The project in question was an asset inventory system for Severn Trent Water.  It was written in PowerBuilder (I wonder how many people still use that?!) and, horribly, used Tuxedo as a middleware component rather than do traditional 2-tier client/server, which PowerBuilder was brilliant at. So, going from memory, here's a potted history of projects I've worked on.  I wonder what happened to them all… 1989-1991ish – Mainframe COBOL stock systems that had been around for years and years for Kay & Co (Worcester), part of the Great Universal Stores group at the time.  Anyone for Decision Table COBOL ? 1991-1992ish – Model204 stock systems, again for Kay & Co.  I've still never come across anyone from other companies who had ever used this although it was a pretty neat language. 1992-1995 – Various DOS-based systems, again for Kay & Co.  Varying hardware from twin-5.25 floppy Apricot machines to PC's with and without hard drives, then onto more standard Windows PCs.   All written in Clipper / dBase. 1995-1996 – Windows GUI development with PowerBuilder, again for Kay & Co – writing applications for the Catalogue Bargain Shops. 1996-1997 – More PowerBuilder stuff at Fame Computers – what a crap move that was after many good years at Kays… 1997-2003 – A couple of PowerBuilder and then Java J2EE applications for Severn Trent Systems.  The AIMS project was probably written in around 1999/2000 in PowerBuilder and is still apparently being used now. 2003 – A great time at SmartStream Technologies in Bristol working with great people on a great new rich-client project.  Doing AJAX without it being known as AJAX then, it was a voyage of discovery and a great time.   Shame the long commute got too much to bear.  Good times there…. 2004-2006 took me contracting for a few years at about 4 companies and it was a mixture of brilliant new development of AJAX applications just as those technologies were emerging and the usual case of being hired to help out on failing run of the mill J2EE apps…  The companies and projects will remain nameless so the good can't be distinguised from the bad ! 2006 until now brings things up to date at Talis, first as a contractor for 1.5 years, then as a permanent employee. I consider myself pretty lucky over the years with the exposure I've had to different projects, technologies and languages.  It's been a great journey and I hope it continues.

'To Do' lists, LAMP and finding time

I've known for a good while now that I need to make more time out of work hours to look at the things I don't get chance to do at work.  This list gets longer and longer and the time to look at things on it rarely seems to materialise.  I get bits of time here and there to read up on things, dabble and generally do small bit of work but struggle to find enough long chunks of quality time to get a proper project going.  This has to change before the list of things buzzing around my head gets too unmanageable. Number 1 on my list is to do some traditional LAMP development as it's a hole in my skills.   I'm fine with the L, A and P but have never really done much with MySQL in relation to PHP applications.   I've done loads of SQL over the years – mainly Oracle and Sybase with a bit of SQLServer thrown in but only really dabbled with MySQL and never done any traditional RDBMS stuff with PHP.   When work is focussed with what you could class as Linux/Apache/RDF/PHP and Linux/Apache/SemWeb/PHP I need to ensure I keep a broad skill-set and not totally forget (although I have forgotten a lot already) how to be fluent in SQL. So, top home project is LAMP-based and probably using Zend Framework as a starting point as I've only ever used Zend Mail on that and the framework looks pretty comprehensive.  I've blown the cobwebs off my home dev environment tonight, got a stub Zend project checked into my Subversion 1.5 repository (yes, I know I still need to look at the 1.5 merge tracking features as I'd really like to use it at work) and I'm now ready to get on with it. I just hope I get the time to make some progress with it. I'm not quite sure what the project will be yet – it'll probably just start of as trying out loads of the features the framework supports and see where it leads.  A 'ToDo' list manager would be ironic – spending time creating something to list all the things that I don't find time to do…

Frustrating week

It's been a frustrating week so far but thankfully it is nearly over….just one more day.   I've been up to my neck in IE7 performance issues, debugging JavaScript events en masse and generally trying to juggle too many balls in my head with numerous distractions going on around me. There are a million and one things that need doing, and another load that we'd like to do but with the very few people we have it feels, to me anyway, that we're struggling to keep our heads above the water.  My morale is not suffering too much yet – the work is interesting – I just wish I had more time to do some background research on things rather than try and just crank through development stories day after day with no respite. To cap it all off it was planning day today.   The day that always seems to deteriorate as the hours go by with frustrations mounting and comments getting increasingly accusatory.  A thank-you would be nice sometimes  – we're all stressed – we're all doing our best – we all want the best for the project and we're all in it together, so let's all just remember that.  If any day can knock the motivation out of you, it's planning day. Tomorrow is another day.   Tonight it's football on TV, installing a Gentoo VM on the laptop, writing this mini-rant and trying to rest.   Tomorrow it'll be back to the IE7 profiling where I just need to put my headphones on, block out distractions and concentrate so that the mental ball-juggling is a tad easier.

Recruitment agents and LinkedIn

I'm getting a bit tired of lazy recruitment agents and their tactics to try and have an easy ride.   Normally it's just endless email spam about totally irrelevant roles in totally ittelevant locations for a pathetic amount of money.   Most get totally ignored but the exceptionally irrelevant ones get a reply with something along the lines of “What part of my C.V. did you actually think made me relevant for this position?”. Then there are the phone calls…   “Are you looking for work?”   “No, not right now…”   “Do you know anyone who is looking for work?”   “Mind your own business and do the hard work finding them yourselves“. A recent tactic seems to be trying to connect to me on LinkedIn – just so that they can see what other contacts I have and then approach them.   Hmmm….  Nice try but I'm not adding you in, sorry. Unfortunately they are a necessary evil in this line of work.   Over the years I have come across a few good ones – those who know me well, know what I am really looking for, have placed me before and always seem to come up with perfect match roles.  These are the ones to keep in touch with, the ones who take the time to find out what you really want and know when to ring you when something comes up.   Those who cold-call are no better than tele-sales people, using a scatter-gun approach to find someone suitable and annoying the other 99% in the process – they just give the recruiters a bad name. So, to those recruiters who know me well and have always delivered the goods, “thanks”, and keep up the good work.  For those who can't be bothered, I wish you well as it reflects badly on your company and I very much doubt you'll ever place me anywhere. Just to clear things up – for the record, I'm not looking for anything right now – understood?

W3C Case Study: A Linked Open Data Resource List Management Tool for Undergraduate Students

For the last few months at work I've been lucky enough to put my RIA hat back on again after a period of absence doing traditional J2EE  and PHP apps with a strong accessibility bent. Working on these kind of apps is something I have enjoyed immensely since first doing a 'rich client' webapp back in 2003, before the name AJAX was coined and when we were flying by the seat of our pants with cross-browser hell and few libraries out there to help that we are all so used to using these days.  It has highs and lows; a real buzz when it works nicely and sometimes the frustration of getting it to work 'just right' (or sometimes even 'at all' !) Anyway, fast forward to 2008 and working on a successor to the old Talis List product.  There are good cross-browser JavaScript libraries out there, such as prototype, jQuery et al, and nice extension libraries such as Script.aculo.us, but it's still important to maintain an understanding of how it all works behind the scenes. Most of the application is traditional PHP/XHTML/CSS with some progressive enhancement JavaScript thrown in for good measure, but on one area we decided to go down a different route and try out a more WYSIWYG approach with RDFa marked up XHTML. One thing I hadn't used until recently was RDFa.  We've used it on one of the main admin pages in our new product and it's made what was initially quite a complex problem much simpler to implement.  Reference to this can be found on Chris' recent W3C Case Study – "A Linked Open Data Resource List Management Tool for Undergraduate Students", and discussion on it has already started over on Ivan Herman's blog. Currently using Jeni Tennison's RDFQuery library to parse an RDF model out of an XHTML+RDFa page we can mix this with our own code and end up with something that allows complex WYSIWYG editing on a reading list.  We use RDFQuery to parse an initial model out of the page with JavaScript and then the user can start modifying the page in a WYSIWYG style.   They can drag new sections onto the list, drag items from their library of bookmarked resources onto the list and re-order sections and items on the list.  All this is done in the browser with just a few AJAX calls behind the scenes to pull in data for newly added items where required.   At the end of the process, when the Save button is pressed, we can submit the 'before' and 'after' models to our back-end logic which builds a Changeset from before and after models and persists this to a data store on the Talis Platform. Building a Changeset from the two RDF models makes quite a complex problem relatively straightforward.  The complexity now just being in the WYSIWYG interface and the dynamic updating of the RDFa in the page as new items are added or re-arranged. All in all it's been a fascinating exercise over the last few months and it's all coming together nicely.  The read-only views (with RDFa markup) are already live at Plymouth University but editing is only available to someone logged in and with edit priviledges on a list – so no public demo at present. The product has recently got a name, it'll be Talis Aspire.

Subversion 1.5 on Gentoo

A couple of weeks ago I needed to look something up regarding Subversion and headed off to grab the latest copy of “Version Control with Subversion” online.  Whilst looking through it I noticed that Subversion 1.5 had been released without me noticing and that there were a couple of interesting new features.  The main one for me was support for tracking merges up from trunk to a branch.   This is always something I have had mixed success with, normally trying to avoid multiple merge ups from trunk or even trying to avoid merging up from trunk at all because of the inevitable issues when merging the branch back down into trunk. Subversion 1.5 allegedly helps with this so I put it on my list of things to do to investigate. The first part of that was done last night when I set up a new Subversion 1.5 repository on my home Gentoo Linux server.  After initially thinking about setting up a general repository for all users and accessing it through Apache/DAV I soon ditched that option due to lack of time and an annoying HTTP 500 error from the repository when trying to authenticate.  In the end, to just get me up and running, I set up a new repository in my own user home directory and accessed it remotely using the SVN+SSH protocol – job done… So the repository is all now set up, I can access it remotely from my MacBook and I'm ready to go with trying out the new Subversion 1.5 merge tracking feature.   More on this soon….

New year; fresh installs and stuff

Well a new year is here and with it I have some nice fresh dev tools to use.  PDT 2.0 was released on 29th December 2008 and is something I've been looking forward to as a keen PDT user for a while.  PDT 2.0 is much better with dynamic languages now that it's using the DLTK.  The intelligence and code completion on PHP/HTML/Javascript projects is a real leap forward although it can appear a little sluggish re-building the workspace. After switching to PDt 2.0 I also noticed that MacPortts 1.7 had been released.  I've been on MacPorts 1.6 for a while and there's been an on-going issue trying to install a couple of ports; logrotate for one.  It also sometimes had problems doing a PHP5/Apache2 install when you didn't want MySQL in the mix.   Thankfully MacPorts 1.7 fixes this and I decided to completely refresh my PHP5/Apache2 install on MacPorts from scratch.   I've now got a lovely clean PHP/Apache setup with XDebug, PHPYaz & Memcache.   Don't things always feel much better when you've installed them again from scratch?! To top it all off I swapped out the PC I have been running as  a Gentoo Linux server for the last few years.   My parents upgraded their PC after the hard drive died and I inherited a slightly newer, quieter machine to use as a server.   Gentoo has been my distro of since 2003 after previously dabbing with Slackware, Mandrake and RedHat.   I alwasys go through the minimal install to get exactly the setup I want – no universal install CD's here please!  The result is a lean, fast and rock-solid Linux server tailored perfectly to the hardware it's installed on. The final addition to the list is a new router to replace my aging Draytek Vigor 2600We – that didn't work out so well though.   I just knew deep down that I shouldn't have gone to PC World and chosen one.  I just knew that PC World would only sell your typical Belkin, Netgear or Linksys shite but I went and bought one anyway – a Belkin F5D8633-4, 802.11n-draft wireless ADSL modem/router with 4 port switch.  It's going back to PC World this weekend and I'm now back on my trusty Draytek until I decide to invest in a better replacement, probably another Draytek.  More about that in my next post.