Monday, August 29, 2016

Tools: Take Your Pick Part 2


In Part 1, I described my Sunday morning Cleaning the Bathroom problem and how I think about the tools I'm using, the way I use them, and why.  In particular I talked about using a credit card as a scraper for the grotty build up around the sides of the bath and sink. On the particular Sunday that kicked this chain of thoughts off, I noticed myself picking the card up and using a corner of it vertically, rather than its edge horizontally, to remove some guff that was collecting around the base of a tap.

This is something I've been doing regularly for a while now but, before I got the scraper, it was a job I used an old toothbrush for. In Part 1 I recounted a number of conscious decisions around the way I keep the littlest room spic and span, but switching to use the card on the tap wasn't one I could recall making.

Observing myself taking a tool I'd specifically obtained for one purpose and using it for another put me in mind of this old saw:
When all you have is a hammer, everything looks like a nail.
Until I looked it up1 just now I hadn't heard this saying called The Law of the Instrument nor come across the slightly subtler formulation that Wikipedia attributes to Maslow:
I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.
Given the first of those two variants, it's easy to imagine that the universal application of the hammer is a mindless option - and we've all probably seen instances of that - but I think that, generally, tools are used in places where they are inappropriate or sub-optimal for a variety of reasons, and temptations, as Maslow would have it.

There are three explicit variables in play in this space: the problem, the tool, and the person using the tool. Here's one way I explored it, by considering the possible scenarios involving the tool and choice of tool, and trying to understand how the person might have got there:

I recognise the shape of the problem, and I have a tool that I know fits it

  •  ... but I use my favourite tool instead because I'm more familiar with it.
  •  ... but I use something else because of politics in the office, my boss, my colleagues, ...
  •  ... but I use something novel because I want to own this problem and be the expert in it.
  •  ... but I use something else to prevent an increase in our already large tool set.
  •  ... but I won't use it because of ethical or moral issues I have with the tool vendor.
  •  ... but I won't use it because of previous bad experiences with the tool, or others similar to it in some way.
  •  ... but the context changed since I last looked, and I didn't notice, so I'll continue to use the existing tool.
  •  ...

I recognise the shape of the problem, but I don't have a tool that I know fits it

  • ... so I'll use the tool that I have invested heavily in anyway because sunk cost fallacy
  • ... so I'll use the tool I do have that looks closest.
  • ... so I'll use the tool that I have in my hand right now because there's no context-switching cost.
  • ... so I'll continue to use the tool I am using now, despite its flaws, because I believe there is some benefit.
  • ... so I'll use a tool I do have because there's no time/budget/desire to look for others.
  • ... so I'll use something I do have because I'm uncertain of my ability to choose a new tool well.
  • ... so I'll continue to a tool I have because I'm worried about the cost of getting a new tool wrong.
  • ... so I'll use whatever is to hand because I don't really care about doing a good job.
  • ...

I don't recognise the shape of the problem

  • ... so I'll try the tools I've got and see where they get me,
  • ... or make a tool,
  • ... or use no tool,
  • ... or try break the problem down into pieces that can be attacked with tools I do know.
  • ...

The latter class can be particularly galling because it contains two sub-cases:
  •  I don't recognise the shape of the problem, and - even if I did - I don't have a tool that fits it.
  •  I don't recognise the shape of the problem, but - if I did - I would find that I have a tool that fits it.

And much wailing and gnashing of teeth have been caused by the hindsight searchlight focusing on the second of those. Your wailing and gnashing of teeth, right? And the same is true of these scenarios:

I don't, or am not prepared to,  recognise the existence of a problem

  • ... so I make no decisions about tool use at all
  • ... which means that I might stay as I am or unconsciously drift into some other behaviour.
  • ...

I recognise that there is no problem

  • ... but I have an agenda that I am pushing and so force tool use anyway.
  • ... but I just love to try new things so I'll go ahead and use a tool for its own sake.
  • ... but I'm putting off some other work so I'll do needless work here.
  • ... but I haven't got enough to do so I'll try this tool out to look busy.
  • ...

As I enumerate these cases, I begin to think that they apply not just to the person with just the hammer, but to all of us, every time we do or not use any tool for any task.

In using any tool at all you are making a decision - implicitly or explicitly. When you enter three commands into the shell to get something to run you are either accepting that you will not use a script to do it for you, and avoid the typos, being in the wrong directory and so on, or you are missing out on the knowledge that a script could help you, perhaps because you don't care to avoid that time being spent on typing and typos.

In choosing to use the same tool that you always use for editing a file you are missing out on the chance to learn that there is something better out there. But also avoiding paying the costs of learning that new thing. Do you do that consciously?

I started trying to map these kinds of observations back onto my own exploration of ways in which I could satisfy my bathroom mission. As I did this, I came to realise that I have mostly cast the problem recognition and tool choice as something that is done from a position of knowledge of the problem. But my own experience shows me that it's less clear-cut than that.

In this area, I love Weinberg's definition of a problem:
A problem is a difference between things as desired and things as perceived.
I love it not least because it reminds me that there are multiple levers that can be pulled when confronted with a problem. If I am struggling with the shape of the problem I can change it, change my view of it, change my desires about what it should be. Opening out this way in turn reminds me that exploration of the space is an incredibly useful way to begin to understand which of those levers I can and/or should be pulling: perhaps if I can remove the things that look like nails, I can even put down my hammer.
 
Sometimes I find that I can learn more about the shape of the problem by using the tools I have and discovering their weaknesses. Sometimes I can only imagine a possible solution by attempting to resolve the problem the wrong way. If I do that tacitly, deliberately, then I'm here:
I recognise the shape of the problem, but I don't have a tool that I know fits it ... so I will explore the problem space with the tools I have, deliberately experimenting and hoping to learn more about the tools, the space, the problem, myself.
And I might find that I'm then saying "aha, if only I had something which could ..." or "oh, so perhaps I don't really need ..."

But this means deliberately deciding to spend some of whatever budget is available for problem solving on the meta task of understanding the problem. Stated as baldly as this it seems obvious that someone with a problem might consider that, doesn't it? But how many times have you seen something else happen?

How many times have you seen only a tiny fraction of the capacity of some tool being exploited? For anything more complicated than a hammer, it's easy not to know that there are capabilities not being used. The Law of the Instrument can be applied within tools too. If I don't know that Word can do mail merge, I might find myself buying a new tool to do it, for example.

On the other hand, creative reuse can be a good way to get additional value from an existing tool. A hammer can be used for things other than hitting a nail - as a door stop, as a lever, to encourage some seized machinery to become separated, as a counterbalance, as a pendulum weight, as a goal post, and might be a sufficiently good substitute for the "proper" tool in any given situation, at any given time. And, in fact, imagining ways to reuse a tool can itself be a useful way to get creative juices flowing.

But contexts change - the problem might alter, views of it might alter, the available tools might alter. Being open to reconsidering decisions is important in getting a good outcome. Doing nothing, reconsidering nothing - that pretty much guarantees at best standing still or perhaps applying a solution to a problem that no longer exists or applying the wrong solution to the problem that does.

Tool use is inherent in software development and the kinds of choices I've described above are being made all the time for all sorts of reasons, including those that I've given. It was interesting to me, as I enumerated these thoughts, that although in my bathroom cleaning example I have no reason to be anything other than on the level - there are no bathroom politics in our house and the stakes are not high in any dimension - and despite doing my best to be as clear to myself about what I'm thinking and trying at any given time as I can, I still proceeded to make choices unconsciously, to not take account of useful evidence, and to continue with one line of exploration past the point at which its utility was exhausted.

In Part 3 I'll try and recast these thoughts in terms of some practical examples from the world of work rather than bathroom cleaning.
Image: https://flic.kr/p/eFAoHQ

Footnote

1. Given where I come from, and its traditional rivalry with Birmingham, I'm amused that the hammer that's applied to every problem is sometimes called a Birmingham Screwdriver.

Tools: Take Your Pick Part 1


It's early on a Sunday morning and I'm thinking about tools. Yes, that's right: Sunday morning; early; tools.

Sunday morning is bathroom cleaning morning for me1 and, alongside all the scrubbing, squirting, and sluicing I spend time evaluating the way I clean against the results I achieve. My goal is to do a good job (in terms of the cleanliness and appearance of cleanliness of the bathroom) balanced against expense, time and effort.

I've got a set of tools to help with this task. Some are traditional and obvious, including sponge, J-cloth, glass cleaner, bathroom cleaner, toilet brush, Marigolds, ... Some are reasonably well known but not mainstream, including citric acid crystals, squeegee, old toothbrush, ... and some are more niche, including a credit card, flannel, and car windscreen coating, ...

By tool I mean something as basic as this, from Oxford Dictionaries:
 A thing used to help perform a job
And I'm being generous in my interpretation of job too - pretty much anything that it is desired to accomplish is a job for the purposes of this discussion.

Harry Collins, in The Shape of Actions, distinguishes tools from proxies and novelties based on the extent to which they extend our capacity to do the job or can stand in for us in doing the job itself. I find this stimulating but I'm not going to be concerned with it here. (If you're interested, Michael Corum makes an admirable attempt to summarise what is a challenging book in this blog post. My thoughts on it are less comprehensive: here and here.)

I don't think there's any action in my cleaning the bathroom that doesn't employ some kind of tool by the definition I've chosen, so any consideration of how well I'm achieving my goal will implicitly include some element of how well the tool, and my use of the tool, is contributing to it. Which often means that I'm wondering whether I have the right set of tools and am getting the best out of them. Which is where I was on Sunday morning; cleaning the shower screen.

A few years ago, when we had a shower curtain, I didn't need to clean it every week but could wait until the soap and limescale grot had built up and then put it through the washing machine. Although this clearly meant that there was known uncleanliness most weeks it was a decent enough compromise, especially since we bought shower curtains with frosted patterns on them which were less likely to be obviously dirty. (The shower curtain is a tool.)

When we replaced the curtain with a folding glass screen, I simply transferred the shower curtain cleaning cycle to it. And that didn't work well. I found that I was spending a long time every few weeks trying to get stacks of accumulated crud off the thing. Which was very hard work. And the screen, with its clear glass, didn't hide any aspect of the problem either. In fact, it appeared to be more susceptible to splashes leaving a mark than the curtain, and it looked horrible most of the time.

So I experimented with the tools I was using. I explored different cleaning products - amongst them vinegar, newspaper, lemon juice, various glass sprays, and citric acid - and a variety of cloth and sponge applicators. I tried varying my technique, I tried varying the frequency - I now clean it every week - and I introduced the squeegee to remove some of the dirtiness after every shower.

This made an improvement in terms of result - the shower screen looked great on Sundays and decent for most of the week - but I was still putting more effort than I'd really like into maintaining the screen's appearance. And so I started trying to reframe the problem. Could we stop caring about cleanliness? If we could, the problem would simply go away! Yeah! But that suggestion wasn't well received by the bathroom project stakeholders.

So, could we stop the screen getting so dirty in the first place? I considered some options: a water filter, no screen (perhaps back to a curtain), a special screen (I didn't know what was possible), special soap (again, I didn't know what was possible), changing our behaviour (say, baths only), stopping the clag sticking to the screen, ...

The last of these seemed interesting because I could think of a way in which this was similar to a problem that I already understood. I have sometimes used an additive in our car's screenwash that makes water less likely to stick around on the windscreen, and wondered whether I could use it on the bathroom screen.

The cost of researching it - not least because I imagined I'd need to spend time working out what materials my shower was made of and checking for compatibility with any chemicals and so on - and the possible difficulty of application and the likely cost and the fear of getting it wrong and ruining the screen all conspired to put me off looking into it very eagerly. But the lack of desired returns from my other strategies meant that eventually I came back to it.

Making a cup of tea at work shortly afterwards, I was bellyaching to a colleague about the problem, and my idea of putting some additive into my cleaning water. And I got a lucky break! It turned out he had recently bought some stuff for coating his car windscreen which turned out to be suitable also for showers and so had applied it to both, and he was very happy with it.

Accepting his recommendation cut down my potential research effort, making the task more tractable in the short term. So I bought a bottle of the same product he'd used, read the instructions, watched the online videos about how to apply it, checked that I could unapply it, cleaned the screen very thoroughly, and finally applied it (a not insignificant investment in time). And I have been really pleased with the results. Ten or so weeks later, I'm still cleaning once a week but with the squeegee and the coating it's a much lighter clean and the screen looks good for the whole seven days.

Here's another example: initially I used washing up sponges for cleaning the bathroom but I found that they tended to leave tiny grains of their abrasive layer behind, which I then had to clean in some other way than with the sponge itself. So I started using an old washing up sponge (when the one from the kitchen needed replacing I would promote it to bathroom sponge) but that didn't have the scouring power I wanted. So then I wondered whether there were specialist bathroom-cleaning sponges - I know, so naive - and I now use one of them. But then I noticed that there was some accumulated soap stuff that the sponge struggled to remove, stuff that was hard to see against the white enamel of the bath until it had built up into a relatively thick layer.

I found that I could scratch this residue away with my nail and so I could generate a set of properties I might look for in a tool to do the same job better: flexible, strong, thin, easy to handle in confined spaces, non-damaging to enamel.

When confronted by a tool-selection problem with constraints, and without a solution, I find that going and poking around in my toolbox or my shed can be really helpful. I might not be able to imagine what kind of tool fits all of the requirements, but I can probably imagine which of those requirements might be satisfied by the tools, or potential tools that I can see in front of me.

I maintain several boxes of bits and pieces in the shed which are invaluable in fixing stuff and also for daughters' robot building projects:


Rummaging around in one of them, I came across an old credit card, which works perfectly as a scraper. As with the car windscreen coating, it turned out that others have been here before me but that doesn't negate the utility of the tool in my context, even if it does remind me that there were probably faster ways to this solution.

So, shiny bathrooms are great, and the journey to the symbiotic relationship of a suitable tool used in a suitable way to solve the right problem need not be linear or simple. But what here is relevant to work, and testing? Part 2 will begin to think about that.
Image: https://flic.kr/p/au9PCG

Footnotes:

1. I do the washing up every day too! That's right, I'm a well-adjusted modern man, as well as handsome, young and with a full head of my own hair!

Thursday, August 18, 2016

Fail Over

In another happy accident, I ended up with a bunch of podcasts on failure to listen to in the same week. (Success!) Here's a few quotes I particularly enjoyed.

In Failing Gracefully on the BBC World Service, David Mindell from MIT recalls the early days of NASA's Project Apollo:
The engineers said "oh it's going to have two buttons. That's the whole interface. Take Me To The Moon, that's one button, and Take Me Home is the other button" [but] by the time they landed on the moon it was a very rich interactive system ...
The ultimate goal of new technology should not be full automation. Rather, the ultimate goal should be complete cooperation with the human: trusted, transparent, collaboration ... we've learned that [full autonomy] is dangerous, it's failure-prone, it's brittle, it's not going to get us to where we need to go.
And NASA has had some high-profile failures. In another episode in the same series of programmes, Faster, Better, Cheaper, presenter Kevin Fong concludes:
In complex systems, failure is inevitable. It needs to be learned from but more importantly it needs to become a conscious part of everything that you do.
Which fits nicely with Richard Cook's paper, How Complex Systems Fail, from which I'll extract this gem:
... all practitioner actions are actually gambles, that is, acts that take place in the face of uncertain outcomes. The degree of uncertainty may change from moment to moment. That practitioner actions are gambles appears clear after accidents; in general, post hoc analysis regards these gambles as poor ones. But the converse: that successful outcomes are also the result of gambles; is not widely appreciated. 
In the Ted Radio Hour podcast, Failure is an Option, Astro Teller of X, Google's "moonshot factory", takes Fong's suggestion to heart. His approach is to encourage failure, to deliberately seek out the weak points in any idea and abort when they're discovered:
... I've reframed what I think of as real failure. I think of real failure as the point at which you know what you're working on is the wrong thing to be working on or that you're working on it in the wrong way. You can't call the work up to the moment where you figure it out that you're doing the wrong thing failing. That's called learning. 
He elaborates in his full TED talk, When A Project Fails, Should The Workers Get A Bonus?:
If there's an Achilles heel in one of our projects we want to know it right now not way down the road ... Enthusiastic skepticism is not the enemy of boundless optimism. It's optimism's perfect partner.
And that's music to this tester's ears.

Thursday, August 11, 2016

Understanding Testing Understanding

Andrew Morton tweeted at me the other day:
I ran an on-the-spot thought experiment, trying to find a counterexample to the assertion "In order to make a joke about something you have to understand it."

I thought of a few things that I don't pretend to understand, such as special relativity, and tried to make a joke out of one of them. Which I did, and so I think I can safely say this:
Now this isn't a side-splitting, snot shower-inducing, self-suffocating-with-laughter kind of a joke. But it is a joke and the humour comes from the resolution of the cognitive dissonance that it sets up: the idea that special relativity could have anything to do with special relatives. (As such, for anyone who doesn't know that the two things are unrelated, this joke doesn't work.)

And I think that set up is a key point with respect to Andrew's question. If I want to deliberately set up a joke then I need to be aware of the potential for that dissonance:
Reading it back now I'm still comfortable with that initial analysis although I have more thoughts that I intentionally left alone on the Twitter thread. Thoughts like:
  • What do we mean by understand in this context?
  • I don't understand special relativity in depth, but I have an idea about roughly what it is. Does that invalidate my thought experiment?
  • What about the other direction: does understanding something enable you to make a joke about it?
  • What constitutes a joke?
  • Do we mean a joke that makes someone laugh?
  • If so, who?
  • Or is it enough for the author to assert that it's a joke?
  • ...
All things it might be illuminating to pursue at some point. But the thought that I've been coming back to since tweeting that quick reply is this: in my EuroSTAR 2015 talk, Your Testing is a Joke, I made an analogy between joking and testing. So what happens if we recast Andrew's original in terms of testing?
Does being able to test something show that you understand it?
And now the questions start again...
Image: https://flic.kr/p/i6Zqba

Tuesday, August 2, 2016

Know What?


I regularly listen to the Rationally Speaking podcast hosted by Julia Galef. Last week she talked to James Evans about Meta Knowledge and here's a couple of quotes I particularly enjoyed.

When discussing machine learning approaches to discovering structure in data and how that can change what we learn and how we learn it:
James: In some sense, these automated approaches to analysis also allow us to reveal our biases to ourselves and to some degree, overcome them. 
Julia: Interesting. Wouldn't there still be biases built into the way that we set up the algorithms that are mining data? 
James: When you have more data, you can have weaker models
When discussing ambiguity and how it impacts collaboration:
James: I have a recent paper where we explore how ambiguity works across fields ... the more ambiguous the claims ... the more likely it is for people who build on your work to build and engage with others who are also building on your work ...
Really important work often ends up being important because it has many interpretations and fuels debates for generations to come ...  It certainly appears that there is an integrating benefit of some level of ambiguity.
Image: https://flic.kr/p/cXJ31N 

Friday, July 29, 2016

Seven Sees

Here's the column I contributed this month to my company's internal newsletter, Indefinite Articles. (Yeah, you're right, we're a bit geeky and into linguistics. As it happens I wanted to call the thing My Ding-A-Ling but nobody else was having it.) 

When I was asked to write a Seven Things You Didn't Know About ...  article ("any subject would be fine" they said) I didn't know what to write about. As a tester, being in a position of not knowing something is an occupational hazard. In fact, it's pretty much a perpetual state since our work is predominantly about asking questions. And why would we ask questions if we already knew? (Please don't send me answers to this.)

Often, the teams in Linguamatics are asking questions because there's some data we need to obtain. Other times we're asking more open-ended, discovery-generating questions because, say, we're interested in understanding more about why we're doing something, exploring the ramifications of doing something, wondering what might make sense to do next, and you can think of many others I'm sure.

We ask these kinds of questions of others and of ourselves. And plenty of times we will get answers. But I've found that it helps me to remember that the answers - even when delivered in good faith - can be partial, be biased, be flawed, and even be wrong. And, however little I might think it or like it, the same applies to my questions.

We are all subject to any number of biases, susceptible to any number of logical fallacies, influenced by any number of subtle social factors, and are better or worse at expressing the concepts in our heads in ways that the people we're talking to can understand. And so even when you think you know something about something, there's likely to be something you don't know about the something you think you know about that something.

To help with that, here's a list of seven common biases, oversights, logical fallacies and reasoning errors that I've seen and see in action, and have perpetrated myself:

Thursday, July 28, 2016

It's Great When You're Negate... Yeah

I'm testing. I can see a potential problem and I have an investigative approach in mind. (Actually, I generally challenge myself to have more than one.) Before I proceed, I'd like to get some confidence that the direction I'm about to take is plausible. Like this:

I have seen the system under test fail. I look in the logs at about the time of the failure. I see an error message that looks interesting.  I could - I could - regard that error message as significant and pursue a line of investigation that assumes it is implicated in the failure I observed.

Or - or -  I could take a second to grep the logs to see whether the error message is, say, occurring frequently and just happens to have occurred coincident with the problem I'm chasing on this occasion.

And that's what I'll do, I think.

James Lyndsay's excellent paper, A Positive View of Negative Testing, describes one of the aims of negative testing as the "prompt exposure of significant faults". That's what I'm after here. If my assumption is clearly wrong, I want to find out quickly and cheaply.

Checking myself and checking my ideas has saved me much time and grief over the years. Which is not to say I always remember to do it. But I feel great when I do, yeah.
Image: Black Grape (Wikipedia)