While I reported the WordPress 2.0.3 upgrade as soon as I heard about it, I held off on installing the upgrade (mostly because I didn’t have time right away). That worked out to my advantage as it turns out that some ‘buglets’ slipped through quality assurance testing and now they’re going to have to release a 2.0.4 version shortly to fix the bugs that 2.0.3 introduced.

This is not the first time that a WordPress upgrade has been released only to have it introduce new bugs and require another update shortly thereafter to fix the first upgrade’s bugs. See here, here, and here for examples of history repeating.

Given this history of WordPress upgrades sometimes introducing new bugs and headaches, I’d recommend waiting at least a week after an upgrade comes out before installing it. This allows some extra time for the developers and the community to discover any bugs that crop up and provide you with more information as to whether upgrading will be simple and smooth — or a pain in the rear that makes that interferes with your otherwise blissful WordPress experience.

Lorelle (a brilliant woman who provides spectacular WordPress advice) shared her thought process in deciding when to install a WordPress upgrade She writes:

The first thing I always ask myself when considering an upgrade on anything, not just WordPress, is “Are you sure?” I think about the changes, improvements, and new features, and consider if it is worth it. I always upgrade in the end, but I give myself time to debate about it.

This debate time spent on answering the “are you sure” question benefits me in several ways.

1. Waiting helps me decide if the upgrade is worth the trouble by taking time to learn more about it.
2. Waiting to upgrade allows me to schedule the upgrade when it’s convenient to me, and not upgrade in panic mode.
3. Waiting to upgrade forces me to check with the various WordPress Plugins and tools I use to produce my blog to see how this upgrade will impact them. Usually, it doesn’t, but sometimes it does. I’d rather check first than have things bork during the upgrade because a Plugin isn’t working with the new version.
4. Waiting to upgrade usually gives the developers time to fix the upgrade and clean up bugs and problems found in the upgrade before I get to it, as today’s example proves.

I think Lorelle has come up with a wise and well thought out plan that should be adopted by all when deciding when to upgrade Wordpress.

Now, having worked as a software quality assurance engineer for Borland International for some years in the past, I know how ridiculously hard it can be to ensure that your release is genuinely stable before it goes out to the public. It is even harder for the wordpress team as I don’t think they have any specific QA team or test plans (like regression testing to make sure that everything that used to work still works once the latest bugs and enhancements have been added in). And let’s not forget that WordPress is free and that the developers are donating their time to make this awesome software available to you. Not only that, but I can tell you with certainty that retail software, no matter how expensive, always ships with known bugs — there is no such thing as bug-free software.

That said, while the developers are likely more frustrated about the bugs in WordPress 2.0.3 than any of us are, I think it is irresponsible to continue to offer WordPress 2.0.3 for download given the known bugs. Furthermore, I cannot fathom why the 2.0.3 upgrade is still presented in every WordPress dashboard and that the official post for 2.0.3 release has not been updated at all to reflect that users should hold off on installing the upgrade as some bugs slipped through and they are working on fixes at the moment. The bugs in 2.0.3 have been officially announced on Planet WordPress, yet most users wouldn’t know to check there — they see the WordPress 2.0.3 upgrade link in their dashboard and upon clicking it they are taken to the official WordPress development blog and told that the upgrade is, “The latest in the stable 2.0 series” — with no mention of the bugs or that it actually turned out not to be so stable after all. I know the developers are busy working round the clock trying to fix the bugs so that they can release an upgrade that resolves these new issues, but come on, can’t someone take the time to update the dev blog to let users know to hold off on upgrading??

Furthermore, for new users, why hasn’t the WordPress download page been updated (or rolled back) to list 2.0.2 as being the current stable release until the issues with 2.0.3 are worked out? Why are new users being told that the latest stable release of WordPress is 2.0.3 when its not, and that 2.0.3 is the version they should download and install? Why not give new users WordPress 2.0.2 (a truly stable release) to download given that it is now known that 2.0.3 is problematic?

Over at Something Unpredictable, under the post entitled, “What’s Already Broke in 2.0“, there’s a lively discussion about why WordPress 2.0 was released when:

“Many people knew that it was terribly broken. Many people begged on the wp-hackers list, the wp-forums list, the wp-testers list, and at the last IRC meetup to get the release delayed The release candidates were severly broken for a number of people, the rate of bug reporting and committing over the past two weeks is staggering. With all the changes going in, nobody stopped to take the time to test for regressions caused by the changes. Its 1.5 all over again.”

One commenter, Olly, pointed out:

“To be fair to them they have the problems that most commercial developers of popular software find, and that’s that no matter how much beta testing they do, the program will inevitably get hundreds of hours more use on the day of release than they could possibly to in the whole of testing.”

To be clear, these are indeed problems that even commercial developers face. And having worked for several years as a Software Quality Assurance Engineer for a major software company, I can tell you for a fact that expensive commercial software ships with MANY known bugs. The sad truth is that there is no such thing as bug-free software. Introducing new features, and even performing bug fixes, often break existing features (which is why regression testing is so critical). However, in the commercial software world, even when the programmers and the testers are wanting to push back the release date, it’s often the marketing department that controls when the software ‘goes gold’ - unless you found what was known as a ’stop ship bug’, which would only be a bug that would be easily encountered by a regular user AND would be bad enough to crash either the program or their entire system. Beyond that, it was do whatever it takes to get the product out the door on time (even if that means working yourself to death), and sorry ’bout the bugs that still remain.

Nonetheless, even with the idea of being ‘bug-free’ being thrown out as an impossibility, it still stands to reason that users can only tolerate a certain degree of bugginess in a product before the uproar starts. And if many of those bugs turn out to have been known for weeks or months before the release, it does beg the question as to WHY was this product released so early? Given that it is an open-source, community backed FREE piece of software, there is no monsterous marketing team breathing down your back to finish the software that they already SOLD to many customers (and promised them a ship date). There are no numbers that your sales team has to make for any particular quarter, and no shareholders to appease. So far as I can tell, there is no monetary reason to deliver the product before it is truly ready.

Also, I don’t know how open-source projects (and WordPress in particular) work when it comes to Quality Assurance - is there even a QA department, or is everyone associated with the project just expected to do continual bug testing and keep their eyes peeled for problems and anomilies? If it is the latter, that could explain somewhat why there are so many more bugs being found now that the release version has been delivered. There’s more to software testing than just looking for bugs. It involves creating test plans, regression testing, negative testing (wherein you do things with the software that you’re not supposed to and see if it handles the problem gracefully), etc. And different people need to be assigned to different areas of the software so that they are focused and really become experts in their area. It was hard enough to do with a team of well paid developers - I honestly don’t know how you get that done when it’s all volunteer effort (although I’m not saying that the WP team hasn’t incorporated all of these areas of testing as I’m not in a position to know).

But given that it is an opensource project, and apparently reliant on much of its userbase for unearthing the bugs, it would behoove both the WP community and the WordPress team to provide clear and easy to use directions on how to search for a bug in Trac and, if it’s not already listed there, enter it yourself. I’d venture to say that less than 5% of users know about Trac (WordPress’s bug tracking software), nevermind how to submit a bug they’ve found. (I just submitted my first bug: Ticket #2218: Pop-up window for inserting hyperlinks truncated on FireFox 1.5)

On wordpress.com, there is a handy little ‘Feedback’ button that appears on every admin screen designed for sending ‘bugs and hugs’, which I though was really great. I don’t know why that was omitted from WordPress 2.0 - it’s a great way for the WordPress team to interact with those WordPress users who don’t hang out in the support forums, etc.

In sum, any software project of this scope and with this large of a user base is extraordinary challenging to QA, even in the commercial world. I’d imagine it’s that much more difficult to do when everyone is working on a volunteer basis. That said, open source software has a luxury that commercial software doesn’t in that you don’t have to get the product out by a certain date in order to meet your numbers for a certain fiscal period. Any .0 release is a major release, and should have enough new features and bug fixes as well as improved existing functionality to entice existing users to upgrade. As TheBisch has mentioned, I’m not sure the features in 2.0 are compelling enough to get existing users to upgrade, especially when there are so many bugs and broken plugins, not to mention that it is likely that we’ll be seeing 2.0.1 and 2.0.2, if not 2.0.3 coming down the line shortly and have to upgrade again and ugain, all with potential upgrade fiascos (after all, that’s what we experienced with the 1.5 release, and that one seemed more stable than 2.0…) Which leaves me wondering - why was WordPress 2.0 released when there were people purportedly begging to push back the release date until more bugs were resolved??