Exploratory Testing with the team

I use Exploratory Testing (will call it ET from now on) every day at work. I like to use it, because it requires focus, creativity, and deep thinking skills.

Now, even though my team is doing Scrum, we are now close to the beta-release of our product and the business starts to panic a little (I know, scope and time shouldn’t be fixed in agile, that’s a whole different story). We have waaaaaaay more points in the Sprint than we can ever commit to normally, and a sudden explosion of bugs happened. This left me feeling not so confident about our product and I also noted that the developers didn’t test that often. I then (finally!) formed the idea of getting everybody in the team together and do a couple of ET sessions.

I told the PO and Scrummaster about the plan and they liked it. (a small voice in my head said: ‘why didn’t we do this before?’). The original plan was to do an ET session on Thursday, with a time box of 2 hours. Not 2 hours of testing, but 15 minutes explanation and 2 time boxes of 30 minutes of testing. In between a little break for ppl to run around, get coffee etc.

The plan changed when the PO wanted to do it earlier! So suddenly on Tuesday I had 30 minutes to prepare myself….ok here goes. I quickly made a presentation about ET in a nutshell. With only a couple minutes of explanation at hand I thought about the essentials of ET. I thought the most important thing was to explain what ET is NOT. Many ‘n00bs’ think it’s somewhat monkey-testing. “You just go around in the application, and you’ll find something”. No, ET is about focussing on the charter you’ve just made. So, I had to explain about charters (the mission of the ET session). Also, documenting is very important during the session. Developers, who mostly write automated tests, are maybe not used to remembering what they’ve done. And you don’t want to come across a defect and then not remember how you’ve produced it. So, document what you’re doing. I also had to explain that I don’t need their whole documented journey, we just want to find nasty defects and that’s the only outcome of the session: defects and steps to reproduce.

In those 30 minutes prep, I also created some charters. I thought it would take too much time if we would create those with everybody involved. Time was sparse. The charters involved testing risky area’s in the product. We are making a tablet app, so one of the charters said “Explore rotating in the app with an Android tablet, to discover unexpected behaviour”.

For this first session, we got together with a part of the team (the iOS developers and the back-end developers). I wanted to let the iOS developers test the Android app and vice versa. I explained the bare basics of ET and we divided the charters among those involved, only focussing on Android this time. Some people paired up, but some preferred to test alone. And off we went! We only had time for one time box of testing, but after 30 minutes we found a lot already. During the debrief I wrote down 14 defects. The Android developers came to me and we discussed the defects. Some would be solved (like a crash we found after rotating) and some were dismissed as ‘won’t fix’.

But wait a second, 30 minutes of testing with 6 people uncovered 14 defects. Wow. When I test alone, I cannot find that many. Plus, the input of other people is invaluable. To turn around the idea that ‘testing is done by the tester’ into ‘testing is done by the team’ was the best outcome. The developers liked the testing (I guess because it was only a short amount of time :P) and also that it was focussed on a clear goal. Of course, they weren’t expert testers at once. I noticed that they were distracted so easily! I wonder if they really did ET or that it was a bit of monkey testing after all. But for the first time, I don’t really care.

On Thursday we had another ET session, with more people. Again, I explained the idea of ET, and with more people on board we could do more charters. We had so much fun again this time. Many nasty defects were found and I saw a lot of ‘testing potential’ in some developers. One of the iOS developers was very good at it; very methodical in his documenting skills and approach and able to focus very well.

Observations this time included:

  • Pairs of people are able to focus better, even more so when they are alone in a room.
  • Some developers are very easily distracted (again). They find a bug and start debugging (I had to remind them of the mission, focus and timebox). 

And then, when the time box was nearing its end….the test environment died. Frustration all over the place. It was down for the next 2 hours so we couldn’t do another session. But in the end, with designers, PO, testers (even one from another team) and developers participating (!) we found twenty something defects. More importantly, there was good enegery, everybody liked the approach and talked about doing this every sprint. This made me very happy. I really hope this was a turning point for the team and we will see (‘manual’) testing more as a team activity instead of something that ‘only testers do’.

And now a voice in my head screamed: “Why didn’t we do this before!!????”.

And: Exploratory Testing approach all came from “Explore It!”, by Elisabeth Hendrickson.

Next time I’m going to try and introduce more ET stuff like the different test techniques of ET and the Test Heuristics Cheat Sheet. Hopefully, we can progress the ET skills of our whole team.

2 thoughts on “Exploratory Testing with the team

  1. Hi Maaike,

    Thanks for writing this experience report! I like the way how you started solving a testing challenge you faced. Too many times I’ve seen testers just sunk in and complain they can’t solve their problem.

    I have a few comments for improving your article/thinking – from my point of view. Don’t take my words for granted; arguing my points and asking questions about them is more than welcome.

    Firstly, at the beginning of the article you state you “use” exploratory testing. ET is an approach to testing, it’s not something one uses. Since exploration is essential to testing, some people dislike using the term “exploratory” and they prefer to just talk about testing.

    Then you mention ET requiring things that it doesn’t quite require. When ET is done good it might require those (well, “focus” depends a bit on how you see it, since I think good ET requires also de-focus, but then again, to notice the need to de-focus usually comes from focusing…), but in itself those are not needed. Of course we could make the poin that focus and creativity are needed, but then again, in the same way they are needed for example for writing a sentence.

    On a side passage you note “developers didn’t test that often”. I prefer to talk about programmers, but that’s another topic. However, I’d like to remind they test continuously. Programmers are fantastic testers. If you don’t believe me, sit next to them while they code and observe what all is happening. Their testing is most likely very different from yours, but they do test a lot; and many do a fantastic job in it.

    Later on you write “ET is about focussing on the charter you’ve just made.” This makes me believe you are talking about session-based test management. ET doesn’t need a charter. ET, as a term, has been used to describe the work is not based on scripts. (Some say “testing” already includes that idea and I tend to agree with them.) Have you seen http://www.huibschoots.nl/wordpress/wp-content/uploads/2013/01/ETscale-300×223.png?

    That leads us to “Many ‘n00bs’ think it’s somewhat monkey-testing.” Now this is a difficult one. I’ll make a guess what you mean with the term, but if I am wrong, please do tell me so I can correct myself. So my guess is that you use this term to describe doing something without having a specific test in mind. I think that is one example of exploratory approach to testing. Quite often I start wondering around an application, because something made me think of an alternative path in my testing. Quite often I test an application before knowing much about it, and even less having specific tests in mind.

    In fact, if you are an observer of someone’s testing (let’s say you can’t communicate with the person, only observe his actions), it would be very difficult to say if the tester is doing valuable work or not. It would look like pushing buttons and clicking links. Sometimes the tester might look at a Youtube video OR use those few minutes to think what to do next. As an observer of the *actions* of that person, it would be hard to say what is happening.

    Having all that written, I enjoyed reading your post. I’m trying to help you clarify your writing and thinking. The things you wrote about are wonderful. You seem to be a bright person and I like talking with smart people.

    Best regards,

    PS. The sudden explosion of bugs didn’t happen because having too many “points” in the sprint. It’s a consequence of trying to work faster than what is sustainable. That makes the overall progress a lot slower. Please tell your PM/SM/PO they are delaying the project by rushing the programmers. If you need help/support with this, I’m available.

    1. Hi Jari!

      First of all, thanks for the elaborate comment. And sorry for taking so long to respond, I’ve pretty much ignored my blog for 1,5 month :/

      Thanks for giving me tips about ET. I consider myself just a beginner, so any tips for articles, books etc are much appreciated.

      My approach is always a pragmatic one. I don’t really care if I’m not doing/using something entirely the way it was intended, as long as it yields results. I do acknowledge that developers test, but in the article I mean ‘functional’ testing without diving into the technical details (observing the application one lvl higher than the code, so to say). I also noticed that the devs in my team feel lost if you ask them to test that way. That’s why I put so much focus on the Charters. The charters gave the devs a simple way to know what to test. And by pairing a dev with a tester we made golden combination.

      The biggest gain was that the devs got to learn/understand more about what a tester does, and they liked it! I guess the whole positive energy we got from it was the best result we could get.

      And I do agree with what you’re saying on the explosion of bugs being a result of too much pressure from the business. Sadly, we told them many times that we couldn’t deliver quality software, but it was still a given that they put too many points in sprint…:/

      Overall, I realise by the points you bring how annoying language is. I might want to say something and in my head it makes sense, but putting it into words that can mean something entirely else to somebody else is a problem. It also doesn’t help that English isn’t my first language. I do agree with almost all of your points and with some I’m like…we’re trying to say the same thing but apparently I wasn’t clear enough in my blog post :D.

Leave a Reply

Your email address will not be published. Required fields are marked *