When the Customer Won't Wear It

My QA team operates as a horizontal resource across the company. As you'd expect we predominantly provide our services to the dev team and most often that's for production work, although we do research and prototype projects with them too. But other functions, including documentation, marketing, consulting, operations and IT need test input on a regular basis too.

Depending on who's asking, what they're asking for, and the kind of project they're working on, we'll ask for different levels of specificity in the input they give us.  For example:

• a one-time, ad hoc project to gather some performance data from a trial server might require no more than an email confirming the test envelope and environment and an idea of the intent of the experiment
• a three-month R&D project which delivers scripts to a partner might have a brief description of the aims, the high-level requirements and any areas which were not addressed
• a brand new software feature would come with some kind of software requirements specification and we'd expect the QA team to be involved in the development of that before implementation proper began.

Different customers also have different expectations of the way that reports are delivered to them. For example, marketing folk might be comfortable with a Word version of their press release marked up with comments. The dev team might prefer to receive their defect landslide as issues in our bug tracking system.

We try to provide templates/checklists on our wiki for the most common kinds of interactions so that customers learn (or can look up) what we expect as input for a particular piece of work, what kind of coverage we'll provide, and what we'll deliver as output. Sometimes these won't fit and in those cases we offer advice on what might work but have to be prepared to be flexible to what the customer needs.

Like tailors, even though we have the eye, the expertise, the tape measure and the scissors, in the end it's the customer that wants to wear it and so we have to cut our cloth to suit.
Image: http://flic.kr/p/5XT9C under Creative Commons

Schadenfreude++

I'm sure you've worked with people who don't pull their weight. People who deliver sub-standard material when it's too late to do anything other just than use it.  People who find a way to wedge an i into team and make time - theirs, that they could be doing better things with. People whose shoulders slope so steeply that Jackass have arranged to slide down them naked and greased up.

So I'm sure you've enjoyed it (go on, admit it) when eventually they've got it badly wrong. I'd like to call that feeling mustworkhardenfreude.
Image: http://www.flickr.com/photos/dullhunk/6296553264/ under Creative Commons

The Perfect Purview

It's an oft-spoken and oft-heard lament that the scope wasn't well-defined. Usually from project managers towards the end of a piece of work and usually from QA towards the start. There's no way around it: if you don't define the purview of a project you will very likely compromise it in some way. The usual ways are lateness, quality and budget.

As a tester, understanding the feature envelope is important but it's not impossible to test without and you shouldn't be scared or unwilling to try. In fact, even when you do have a definition your first action should be to consider whether it's the right one.

I keep two scoping tools in my testing kitbag: a telescope for the big picture and a microscope for the details. You'll always need both. You may be the only person on a project who recognises that both are necessary and is capable of combining the view the two produce. You may be the only person on the project who really wants to look.
Image:http://jaymontgomery.blogspot.com/2009/12/project-triangle.html

The We in Weltschmerz

As a tester, it's easy to focus on the negative. It's in the nature of what we do. You might not be able to prove the the software is defect-free but you can, and we do, spend much of our time and effort on looking for the problems that we expect will exist in it.

Constantly finding issues can lead you to think that your dev team, your boss, your process, your software, your company, the sandwiches in the canteen, the fashion for colourful jeans and life itself are pointless. You need to find a way to keep those feelings in check. Except for the one about colourful jeans.

But how?

Listen to feedback from your customers. Talk to your sales and post-sales teams and find out what customers are saying about the software, how they are adding value to their business, which features are most prized, what recent additions have saved them time and effort.

Remember that you're in the information generation business. It's your job, amongst other things, to give a balanced view of the status of the software at any given time to your stakeholders. Sure, you have to report the bad and give your analysis of the risk attached to it, but don't forget to give a view of the good too. Oftentimes you'll be able to report that core functionality is solid but the edges and corners are flaky. Oftentimes, this may be enough. Forcing yourself to report both sides can make a difference to your view.

Look back over a couple of major releases and let yourself realise what you've achieved, how much you've improved the product and how little the minor flaws that you know are there have actually impacted on your users.

Dark humour, but only with someone you can trust not to broadcast your doom-laden exaggerations around the company.

Write a blog full of trite obviousnesses and hope it takes the pain away. (It doesn't.)
Image: Salvatore Vuono / FreeDigitalPhotos.net

Watch This

My QA team are observers on all bug and support traffic. They aren't required to respond to any of it or even to read it all, but they are expected to skim it. If nothing else this gives a flavour of where current issues are in the released and in-development builds.

But we find it's more valuable than that. It generates test ideas based on customer scenarios, or even asides or expressions of interest in future features. It means that we can provide a more informed user viewpoint for product decisions, alongside our customer-facing colleagues. It's also not uncommon for QA staff to provide information that helps the support staff identify issues.

With the overview we get, we can join the dots across the product. We reduce the number of dupes that are filed, we spread knowledge of fixes that may affect repro of other bugs, we often find that an issue that someone has seen sporadically and not filed because they can't repro will be related to another ticket and we get a more complete report, more evidence and so on.

Some of this came up in an interesting  forum discussion at the Software Testing club.
Image: Salvatore Vuono / FreeDigitalPhotos.net

Testing is Like Making Love

Testing is like making love... it never lasts as long as you wanted it to.  Testing is like making love... once it's over you're the only one interested in discussing how it went. Testing is like making love... well, I'm sure you get the idea.

But if testing isn't really much like making love, what is it like? Non-testers can think testing software consists of nothing more than trying a couple of examples and then declaring that "it seems to work OK".  Finding a simple analogy for the testing process could help to disabuse this notion.

Which is where word searches come in. Every year, to show what a great guy I am I spend as little time as possible thinking up some test-related game for a team meeting in late December and force the team to play along. (And they love me for it. Or, at least, they know I'd sack the first one to protest.) One year we taste-tested stollen, mince pies and panettone; last year it was proofing a bogus Christmas card from QA to the other teams (complete with the Santa James above) and this year I generated a festive word search supposedly from our Marketing team to our customers for the team to review.

While they were eating panettone and pretending to enjoy the task until I was out of earshot, I went to make some coffee for them (you see, I really do care) and started thinking about the validity of the word search as an analogy for feature testing: the spec is the list of words, it defines everything that must be present in the feature. The rules of word searches are the implicit specs, the conventions and other internal constraints on software such as the UX guide. The finished grid and the list are the delivered software. Testing is a matter of taking what's been delivered and investigating, for example:

• whether the feature contains everything that the spec said it should - are all the words in the list to be found in the grid in legal configurations?
• whether the feature contains anything that the spec didn't say it should - are there other legal and plausible (e.g. for the theme of the word search) words in the grid?
• whether the feature violates any of the implicit rules - are there any non-letters in the grid?
• whether the feature is what the user would want - do the characters that are not part of the list words spell out profanities?

Just like in real life, you can take different testing approaches and they'll have trade-offs in terms of time and effort vs issues found, say:

• bottom-up - starting with each cell and looking in each direction for each of the spec items
• top-down - taking a spec item and scanning for letters that might be in it
• heuristic - looking for common letter patterns and seeing whether they are part of the intended words
• random - just straight eyeballing and hoping to spot something positive or negative
• automated - writing a tool to identify the spec items in the grid and look for other common words too

Again just like real life, while you can convince yourself that all of the spec items are present (or not), it's much harder to convince yourself of the acceptability of the whole. (I like Iain's take on testing against spec in Exploring Uncertainty.)

So testing is like a word search, then? Perhaps, but do you have something better?