Saturday, March 08, 2008

Drupal let the quality boat sail by. Hold on, hear me out :) We're a really focused and strong (and almost too commercial) community, and we really have put our focus on features and output, and we've lacked a lot when it's come to making it all work correctly in every possible case.

That's where quality testing comes in. I've been thinking about Drupal's quality assurance process for a couple years now, but in the last few months it's really come to a head for me. My personal "dam-breaking" moment came when the Batch API in Drupal 6 prevented install, and other tasks depending on the API from running automatically, or in text based browsers, or in browsers were people have been privacy conscious and disabled JS and Meta refresh tags.

Without proper quality trails this will continue to happen. And here's another point I'd like to make:

If we expected every patch to have every possible test performed on it by the developer, or even automatically, we're fooling ourselves., even though it's going to help; even though it's going to catch errors; won't ever be able to replace human testing. Simpletest and DrupalCHK framework is not going to be the do-all and end-all of our problems, nor should we try to market it as such. We risk people feeling that if we have unit testing, we can skimp on the other Quality bits. Just like how a newbie computer user might think Norton 1998 is still protecting them in 2008.

Here's what we need in order to make future releases of Drupal truly "quality assured":

  1. We need to have community-created smoketests that members of the community can run on various servers, with various browsers. These are low level things drupal must be able to accomplish in that environment. Each "smoketest", eg, "Create a new node type" would be able to be marked "working", "working with errors", "working with incorrect layout or UI", "failed". We'd log the different environments to find patterns.
  2. We need user interface quality testing
    • information architecture
    • disability-compliance testing (screen readers, color blindness)
    • standards compliance (ECMAscript, CSS, X/HTML)
  3. We need a group to asses changes to the coding standards, and to modify and refine the standards and time goes by.
  4. Quality events (eg, a weekly or monthly online event were everyone can work on bugs, perhaps with prizes)
  5. Quality initiatives (bug spotlight is a good start, but a group that can define more initiatives like this)
  6. documentation, forums, live irc help and mentor for those new to helping with quality testing. Probably best done as, using as a guide. The great thing about quality testing is: you don't need to be a developer, or know how to code!
Hopefully, this will spur the Drupal powers that be to make more of a effort and create a full fledged quality team, and not just the appearance of quality by using a single method of quality assurance. I'll be happy to assist in the development of such a team.

I haven't meant this post to be harsh on Drupal, we've done very well, but this will hopefully nudge people in the "quality" direction.


Jason said...

I hear what you're saying and I agree that what you're asking for would move Drupal to the next level. In fact, as Drupal becomes more widely used the quality aspects you describe become even more important. That said, I'm questioning whether how you're proposing it be done can/should be done in the manner you're proposing.

Admittedly, I haven't been intimately involved with open source projects in the past. Drupal is my first foray into contributing to open source. So really this is more of a question than a commentary. Do other large open source projects have formal regression testing processes and/or minimum requirements or does it happen more organically and ad hoc? Furthermore, do other projects have enough community commitment to sustain such ambitious testing goals?

Again, I'm not arguing against your point, if the Drupal community could make this happen it would be fantastic. I'm just curious if other open source projects have attempted this (or are successfully doing this) today.

Robin Monks said...

I hear you Jason, and I'm saying we need to really have the best quality minds at Drupal sit down and really think this thing out. We really paid too much attention, and put too many resources into Unit Testing without a higher level quality focus.

The current Quality Assurance group at g.d.o is a good proof of that point, most of the posts there are actually from the Unit Testing group.

There are a couple examples of quality processes like I'm proposing:
1) Mozilla, ,
2) Gnome,


catch said...

Well there's been a shortage of patch reviewers for quite some time, with most reviews done by a very small pool of people. Getting more people involved in that would be a good start. We shouldn't have 200 patches in the review queue, ever.

Automated unit testing (and even testing whether patches apply or not) is going to help - if only because it'll free myself, webchick and others from clicking round forms all day to 'see if things break'. Then those of us already doing reviews can concentrate on the stuff that can't be automated.

Additionally, I also drafted a GSoC project outline for a javascript testing framework which hopefully someone will pick up. Again, less clicky clicky for a small group of humans means more human attention where it's actually required.

For me the primary focus will to be get a wider pool of human eyes on patches - until there's enough people looking at them (rather than subscribing, +1 and the rest of activity which has nothing to do with actually reviewing) then anything else seems a long way off.

One thing I'd like to do following the UMN testing is work towards a remote usability lab (using clickheatmap and maybe some other stuff) which people can log into from home. Then try to grab unsuspecting new Drupal users to try out new UI features and the rest. That'd mean a far broader range of people involved in the patch review process but it's not trivial to even spec out.

Robin Monks said...

I agree unit testing will be helpful, and the headmap you suggest may be useful for usability and Information Arch. But, when it gets to the nitty-gritty you really need to have people dedicated to doing smoketests and quality testing.
No one solution can take care of everything; and switching community focus from one solution to another without solving the bigger problem of a more general quality assurance solution, that can adapt and create new tools as necessary, will just wear down those already working on quality, and will hurt the code quality in the end.

Josh McCormack said...

Great post.
My company, InteractiveQA ( does both Drupal development and QA testing. Development and QA seem like a natural fit to me, but many we come across are surprised.

I think automated developer centric testing, like unit testing, are great. As a company we're employing these methods more, too. However, these are very developer centric, even if not done specifically by the developer who wrote the code being tested.

While we also do load testing, automated link testing and scripted functional testing, the main type of testing we do for companies is what we call experiential - it's a largely unscripted, quick turnaround mix of functional and usability testing. It's an easy way for agencies that don't have QA as a part of their process to start using QA. But it also helps to test things in ways that may not have been envisioned by those who made it.

We'd like to get involved in helping Drupal with QA, and definitely noticed the trend lately where unit testing was synonymous with unit testing. Every organization approaches QA a bit differently, and we hope to be able to help out with the unit testing, too. But if there is room for testing by hand in the functional or experiential methods we'd like to get involved and help out.

Josh McCormack
Owner, InteractiveQA
Social Network Development & QA testing

catch said...

Josh, all patches that get applied to Drupal go through a peer review process - this includes manual functional and experiential testing (although often by as little as one or two people). You can get started helping with that right now by reviewing patches in this queue:

Robin Monks said...

catch, I believe Josh is talking about testing HEAD, the complete product, not just each patch. Since multiple patches might come together to form issues not easily found by simple patch review or unit testing.

As I said before, unit testing is not the cure-all and do-all of QA.


catch said...

Ahh now I see. Robin, Josh - as you may know we had a first round of formal usability testing in Minneapolis just before Drupalcon. I and others are very, very keen to repeat this process, both in formal lab testing and more informally within the community. Some of the material from those sessions is available here: - the DrupalCon presentation highlights of the major issues we identified. Although I feel this should be taken as seriously as unit testing in terms of guiding the development process I hadn't connected it with your original article, which perhaps shows how narrow the focus is (perceived or not) at the moment!

hlslaughter said...

Functional testing will be much more relevant if it too is automated (like unit tests).

Human generated testing is just not reliable.

There are OSS versions of stuff like silktest. I saw one the other day, forgot the name, something in Ruby.

It'll be a battle getting something like that adopted though :) Maybe the guy who seems to have gotten simpletest adopted by Drupal could take on that task :)

calypsojaved said...

The current Quality Assurance group at g.d.o is a good proof of that point, most of the posts there are actually from the Unit Testing group.

Automated unit testing (and even testing whether patches apply or not) is going to help - if only because it'll free myself, webchick and others from clicking round forms all day to 'see if things break'
There are a lot of sites out there showing book video. BookVideoTV, BookTelevision and of course CSPAN, but I like how and Reader's Entertainment TV have specific genre channels and original shows. There's just more to see and I can be specific in what genre I'm interested in. Anyone else watch online tv?