The TestOpsy

Back in January I hosted James Bach and Michael Bolton for an AST webinar on the concept of a TestOpsy or a way to learn about the testing you do by dissecting it. Below you’ll find not only a description and the webinar video but a transcription for what I hope is easier reading.

By looking very carefully at what you actually do, identifying your own heuristics, and putting that process into descriptive, evocative words, you can discover surprising depths in each act of testing you perform. In a testopsy, you build your skills of observation, narration, and test framing. You may even discover a technique no one yet has written about. And if you do it with a colleague, it stimulates discussion on test design.

James Bach & Michael Bolton

What do we mean by a Testopsy

We are talking about an autopsy for testing. We are talking about taking a very close look at a session of testing and you can do a testopsy based on just a few minutes of testing.

How to do a Testopsy

I have done as little as 7 minutes of testing where I’ve broken it down into 2-second increments and analyzed every single thing that happened. I’ve also done a testopsy of a whole day of testing, but not to that level of detail. The basic idea is you don’t have to do recording for this, but I think the best way that I’ve found that it works is I record a session of testing on video or on the screen capture. And then we go through the recording, so we know everything that we did. We try to describe that with specific words. Sometimes that means that we invent terms for things. We’ve invented lots of things for RST by watching ourselves test.

Also, this one was particularly fruitful. We’ve done a live testopsy, we haven’t been on a video testopsy. This is the first time we’ve tried to do a video testopsy with each other. So the advantage of this one is that the participants can step back and observe their own work in a live testopsy. The typical thing is for somebody to be testing and for the commentator and the observer to be separate from that process. So the person doing the testing and the person analyzing the testing are to some degree distant. In this one, both of us could do a deep dive on this and that was super handy.

So we’ve done this on stage. In fact, the original idea for this presentation is that one of us was going to test and the other one was going to comment on the testing in real time. But then we decided to do something a little more controlled. And what we did is we found a video that we made of ourselves back in July that we didn’t make for the purposes of a testopsy. We just were recording it because I want to make sure that I captured all the information because I might need it later. We were doing testing in preparation for the rapid software testing applied online course.

So we were doing real testing that we needed to do ahead of a class where we were going to teach using this tool called Meet-O-Matic. So you’re going to see us test Meet-O-Matic. Since we had that recording, we had it all ready to go. We decided to take that recording and just do a testopsy of that. In doing so, we discovered new bugs that we didn’t see the first time in real time. We discovered some new insights about testing and we’re eager to share them with you.

Session Report

So let’s first look at our session report. Now, what a session report is in this case is this is a report of what we did in that test session. The charter was to analyze at Meet-O-Matic in preparation for deep testing. This was probably the third day of the rapids software testing applied class as we teach online and we had already done some observation and some interaction with this product, but we wanted, among other things, to create a product coverage outline that could allow us to select a target for deep analysis. We wanted to look at some aspects of the product deeply, and one of the insights we’ve had through the course of developing this testopsy report is that deep testing is a testing that is intended to maximize the probability of finding every elusive bug that matters.

The goal of this is to prepare for that kind of testing. But on the way, we found a bunch of things about the product that surprised us. And of course, since we were testing in order to learn how to test the product deeply, we found a lot of bugs.

This is the written report that I prepared immediately after the session, which we show in class to say this is an example of a session test report. Now, I prepared this without closely looking at the video that we then looked at just yesterday and the day before. So this is just mostly this report you see here is not just on memory, actually, since it was also assisted by the mind map.

Video Analysis

This is the mind map that we were creating during that test session. So normally in a non-testopsy situation, this is all you’d have. You’d have the session report if you’re used to doing session reports and then you’d have whatever else you created during this. And during that session, Michael created this mind map of all these features. So this was an analysis session to create a mind map of the elements of the products so that we could systematically test those elements. He’s got information about testing here, about our process. And then down here is stuff about the products. So normally this would be all you’d see. You just see the result. But now we get into the actual testopsy. So what I’ve done is I have prepared a set of clips of different things we did. This is near the beginning. This is in fast motion here. I’ve sped this up 16 times. This is the beginning where we were getting ready to take notes. So you see that we got an idea in mind of how we want to organize these notes.

So what you see in the center where it says Test session: Meet-O-Matic, what I’m intending to do is to create a coverage outline based around that central node. Up in the upper right there, you’ll see something. It’s a little bit small, but what I’ve got, it’s separate from the product coverage outline in another tree on the same map. I’ve got my notes about the testing itself that includes metadata about the testing session. So these days what I’m putting in my listing like that, preparing for a session report, bugs that we found, issues (something that threatens the value of the testing effort), risks that are associated with the product, test ideas, tools and general notes. Now, I set that out as a kind of template that is parallel to the information that’s going to go into the session sheet later on. But what I found out is that it’s kind of cool to have that immediately at hand visible in the same visual space as my product coverage outline, but they’re not the same thing.

Here’s a real quick look at in-fast motion. I decided I would drive. Michael was the scribe. This is Michael’s screen. He is typing out the different features of the product as I’m walking through the product. I’m walking through Meet-O-Matic and I am calling out what I see. And Michael’s just taking notes as fast as he can. So Michael’s in charge of creating this thing. He’s creating it. I’m not actually seeing his screen. I’m looking at the product. And that’s a way I usually like to do it. I usually am the one taking the notes. And when I’m pairing with people, I like when Michael takes the notes, when I’m working with him. But usually I’m so fussy about the notes that I don’t like anybody else doing them. I’m glad that Michael’s fussier than I am.

One thing that you don’t see is I’ve got the product on another screen. So I’ve got a screen running and I’ve got the notes on the other side. And James, I’ve got on yet in yet another place, I can see what James is up to. So I’m often trying to corroborate stuff that he’s doing in real time.
Now, here is a moment that we found that we thought was particularly interesting. I’m going to play this clip and I’m going to play it with some audio if the audio plays. Let’s see if it will.

Now, just used my copy all links extension to copy all links on the front page. And there’s also a terms and conditions link. Just copy all links to an extension called copy links. It turns all links into text and lets you copy and paste them into a thing.

So what’s interesting there is first of all, I’m using a tool to help me explore. We constantly use tools to help us explore. This is why we do not use the term automated testing. We think it’s a ridiculous term. We don’t automate our testing. We use various tools to help us test but we’re always in charge of our testing. If we’re going to call it testing, then we’re going to say we’re in charge of our testing. And just like a programmer is in charge of the programing, that’s not automated programing. We’re in charge of our testing, but we absolutely use tools to help us do various things. And in this case, this tool has pulled all these links out. And then for the next few minutes, we go over those links to make sure that we’ve added those things to the product coverage outline.

So an example of how in a test session, frequently you’re pricing each other. If you’re working with more than one person of different tools that you know about, it’s a great way to find out about things. It’s amazing. And I found in organizations how much of that data is assumed to be known and is not. People tend to sometimes be a little bit circumspect about the tools that they’re hoarding, not intentionally, but they don’t talk about them as much as they might.

Now, let me fast forward to this little moment, which only took us a few seconds. But then during the testopsy, we discussed this for like 20 minutes.

But this is interesting, though, because right here we’ve got a map of features. How long that took? That was just a few seconds, but we ended up talking about that quite a long time because we realized that while many times the product gives you a list of its own pictures. But the thing is, you can’t just trust that. So we talked about assumptions validated versus unvalidated assumptions and how that works in testing. We must have talked about that for 20 minutes or something.

Discovery Tree

I’ve made a map of the discoveries that we made during the testopsy. So what you’ve seen so far is we started the testopsy. We discuss the tools we might want to use. Well our basic method was we played the video at high speed and slowed it down whenever we saw anything interesting on the screen. And then we would listen to ourselves and then we would talk. We noticed the note taking process. We noticed the tool used. We noticed acknowledgement of the product map. This then led to a discussion about validated and unvalidated assumptions.

Bug Cascade

Then we noticed the Bug Cascade. There was a 20 minute period in this 75 minutes session where we got overwhelmed with bug reports and we thought that was really an interesting phenomenon. And we decided we would focus on the rest of our testopsy just on analyzing what happened. Because we saw that we had multiple bugs, multiple overlapping investigations that interrupted each other. So we kept going into some kind of confusion. We saw that the process of reporting bugs interrupted the process of analyzing the product. We realized later that we didn’t know enough about the product to sort out what we’re bugs, and that was hindering our bug investigations. And we saw later that in all of this confusion, we missed several bugs that were staring at us right on the screen. Other things we saw we didn’t see until quite a bit after they showed up on the screen.

It was really amazing to review this because in the thick of it, almost by definition, you can’t keep track of all the stuff that’s coming at you in this blizzard. Something goes onto the stack. It’s exactly like this, it looks like a money booth. You know those cash grab booths. That’s what it felt like. It was in a storm of stuff that was tornado-ing around us. And when you’re trying to grab the stuff as it’s blowing around like that, you’d grab onto something and then you drop it as you grab onto something else.

Mutually Disruptive Process

That led to this that we coined the term “mutually disruptive processes” to talk about this interesting idea that we hadn’t noticed and named as such before. So as we believe we’ve coined this as a term mutually disruptive, probably Googled it lightly and didn’t see anyone else talking about this kind of thing. But we bet that this is all through testing. We bet you’ve experienced this, too, and didn’t maybe didn’t have a word for it. So this notion of mutually disruptive processes, when the processes are all inside of you, create self-disruption where there’s turbulence inside your own processes, inside your own work that it might feel that it’s wrong or that it’s bad. But we think it might be essential. It might be just fundamental to the nature of the fact that you’re trying to do different things at once in these processes get in each other’s way. And that creates the money booth effect.

Money Booth Effect

The condition of productivity collapse that results from too much self-disruption. And so what we would like you to do, my friends out there, is to begin by forgiving yourself for this because it’s not your fault. Yes, you can practice. And yes, you can mitigate some of the effects of this. But we’ve been practicing this for years and we don’t think that we can get rid of this. And that was a second thing.

Necessary Confusion

Susan says that this is also known as ADHD. And I would like to issue a caution on that. It’s not a D. It’s in fact, it’s neither D. It’s not a deficit of attention and it’s not a disorder. I mean, it might feel disordered, but it’s more like the natural course of things. I would say disorder, if you feel disordered, if you feel like it’s a problem, then it’s a problem. But I have long ago reconciled myself to the fact that my mind flits from one thing to another. And I’ve just arranged my life so that’s a strength, that that’s a feature and not a bug. I’ve taken on a kind of job where it’s good that I’m distractible.

Anyway, for me it’s kind of like saying that to G. Waterfalls are noisy and turbulent and G wouldn’t it be nicer if they could all come in a nice straight pipe? Well, they could, but nature forced that way, you know. Well, let’s watch the big pipe fall. You know, it’s not so interesting. Niagara pipe. Listen how silent it is and how they know it is not wet. It’s an elevation change in a pipe. Okay. So this led us to talk about something that we call necessary confusion. Necessary confusion is also related to something that my pair and I call the Bootstrap Conjecture.

Bootstrap Conjecture

The Bootstrap Conjecture can be summarized as you’d never do anything right the first time. If it’s not trivial. If it’s anything that’s hard to do is not going to be done right the first time. You’re going to have to try it and iterate. And in fact, all of Agile is basically based on the Bootstrap Conjecture and what that means to us. One of the corollary of that is that certain kinds of confusion and certain amount of confusion is absolutely necessary. You can’t be blaming yourself for it. You have to embrace it and work through it.

The Power of Video Review

It really came through when we were doing video review, how we were seeing bugs that we hadn’t seen the first time through. And I already knew about this, but it got reinforced for me. I can tell you that whenever I’m on a really, really important test project and usually for me, these are court cases, I video all the testing I do and then I watch the video to get every last drop of value out of every bit of testing. Now, this isn’t feasible to do in an ordinary commercial project. It’s only feasible if you have a situation where you have a limited time with a project, with a product, and you have to get everything you possibly can out of that limited time but it’s still useful.

By the same talking, I observed that it’s probably pretty cheap to record your testing sessions. And if you never look at them, big deal. You know, the storage space these days is incredibly cheap. But the value of having this was amazing. Just having them, even if you don’t look back, even if you once in a while look back at them. I just have a video camera for every professional project that I do now. And then we also use screen recording. So all this green stuff you see here is stuff that we got out of the testopsy. And we’re not even done with that yet. But I want to go back to the video. So I want to show you some examples now, because everything you just saw us talk about there is substantiated by what actually happened in the test process.

So let’s see an example of what we’re talking about here. This is immediately before the bug cascade hit. So what we got to do is characterize the features on this and the features on this include the default times are displayed. So that’s a time selection pop up underneath, meeting creation. It’s in the time block mode. So for time block mode, you’ve got to select times and just like the AM and PM. And under time selection is timezone. Which I think timezone doesn’t appear. It only appears in time block mode. So timezone selection is in time block mode.

So a lot of the test session is just this, it just goes on like this where we’re just trying to make sense of what we see. And then we started running into bugs with this particular dialog box.

Now, I bet you you don’t know what’s going on there. Dear viewers, I had to watch this video clip about 25 times to try to figure out what the heck was I talking about? I don’t know what am I to this day? I do not know what I meant when I said, that’s interesting. I don’t see what was interesting. Something was going on in my head and I was trying to figure out what it was. And Michael didn’t know. Michael even said what’s interesting and I was trying to put it into words. And while I was putting into words, I was getting interrupted by the fact that there’s actually a bug here, because when I tried to click add time, the time disappeared. And I was trying to make sense of that. I didn’t know whether I clicked the right button or the wrong button. But watch what happens after that.

So we’re already talking about selection, but this interrupted process where I was trying to figure out about the add time thing. So I’m trying to do this time select a window and then we get distracted by the selection. We certainly get this money booth thing. Here’s another element from just a moment later.

What I was trying to do is remove the time. So I wanted to add a time. This clear day not mean what we think it means? So now we’re talking about clear day, so I’m distracted again.  So I’m in the middle of doing something else my pair says what about clear day? And then immediately throws me off track. Now, I like being thrown off track, by the way. I like that about pairing. I think it’s more useful in general to have the value of somebody else noticing things and wanting you to do something than the value of, you know, staying on the same thing. But I have to say that I like to alternate. I like to do paired testing where I get confused. And we have a lot of energy. And then I like to alternate and just test by myself for a while so I can focus and then go back and test with another person because I get more energy when I’m testing with other people and I get different ideas.

It is to try adding a time. Now, is it clear day? And you notice that the purple thing stays there. Now, it’s good and I think that purple thing is supposed to show that you’re on that day. It’s some sort of formatting bug, which we can reproduce. So I just refreshed the screen see if we can reproduce this. So if I just say done, just do it and say done.

So what you see here is that I’ve gotten onto the thread of an inquiry and pursuing a specific inquiry. And now my ideas in my own head have gotten sorted out so that I can systematically create an experiment. That’s what you see there. It’s the beginning of systematic experimentation instead of blundering around and wondering what is going on. Now we’re more in sync in this part of the video. This led us to a more general concern, which is right here about the non-modality of this dialogue.

Let me just jump to the end of this cascade of problems almost to the end. Here’s one more thing I want to show you and then I’ll jump to the end. This is the fun part. So we’re trying to set up a meeting now with all the times put in. Make it a meaningful title on this.

500 error, server crash

After all that we submitted our data, and it turned out there was a back-end bug and we somehow inadvertently found ourselves able to force it to take bad data. And then it crashed. Then the next thing we did is we went to the back-end logging system and verify that it was the server that received duplicate data or something in that request. And that was what caused the problem.

Bug Reporting Sieve

Out of all of that, here is something that we discovered. We conceived of a new thing that we’re going to be talking about in our classes. We call the Bug Reporting Sieve. And what we basically did is we noticed that there is a set of stages or gates, you might say that we have to go through. The normal situation is that in order to report a bug, the bug has to exist. It has to be producible. By producible which means we’ve got to be in a circumstance where the bug can actually manifest. For instance, if there’s a bug in the product’s handling of stuff under Firefox and we’re running on Chrome, the bug might exist, but it’s not producible. Or if it’s a fault in code that never gets executed, it’s dead code. It’s not producible, even though you’ll find it in a code review. So it’s got to be producible.

Notice that producible is not the same thing as reproducible, that’s different things. But of course, it’s related. You have to notice it. You have to comprehend it after you notice it. You have to authenticate it. You have to decide that it’s actually a bug is what I mean when I write authenticate. Yeah, we’re trying to figure out a really good word for saying it’s about bug, it’s definitely a bug. But I mean, I know what’s going on. I noticed something. I know it’s going on and what’s going on is wrong and then you report it.

Now, this is the sieve. At all these gates have to be passed through or you can’t report a bug. But the insight that we came to from watching that twenty minutes section of us stumbling over ourselves is we investigate several bugs. And then there are several of them that we saw that we didn’t report because we lost track of them. There are some things we didn’t see. What we realized is that while sometimes a bug exists and is reproducible, observable, you notice it, but you don’t comprehend it. Sometimes you comprehend it, but then you don’t realize it’s a bug. Sometimes you realize it’s a bug and forget to report it. Other times you see something that’s not a bug, it’s producible, observable and then you realize when it comes time to authenticate it’s not really a bug.

So you see that lots of confusing things can happen. And here’s the thing that happened. It’s particularly interesting in this case. I saw something I was trying to comprehend it, and while it was doing that, I saw something else that knocked the first bug off so I end up reporting the second one and forgetting about the first one. So you have like 3 or 4 times a traffic jam of bugs coming in like this.

A question came in, by producible bug, do you mean that it has to be seen once? No, not that I mean that it’s seeable. Producible means that it is capable of occurring. It’s capable of manifesting. So, for instance, that a bug has to exist. So there has to be something wrong. For it to be producible, you’ve got to be able to have a set of conditions in which it can happen. So one of the better examples of something that exists but is not producible is there’s a bug in code that never, ever, ever, ever gets called, dead code. So that’s a bug that exists, but it’s not producible, certainly not in the circumstance where that code is not altered such that the bug can never come up.

In IEEE Technical language, we’re talking about the distinction between a fault and a failure. The failure is the product that’s something wrong, a fault with something wrong in the code or in the design, a defect in construction that could be a latent failure. It could be a failure, but it hasn’t failed yet. So. So in testing, we can only find problems that are producible. A problem isn’t producible, testing’s not going to find it. Review might find it. Static analysis may find it, but not testing. So anyway, that’s what it means there. So the point is, is that by doing the testopsy, we came up with a set of insights.

Progress Sim

Now, one more thing I want to show you guys is something we call progress sim. It’s part of an idea we call the secret life of testers. What I did is I analyzed everything we did in that 20 minute period during the bug cascade. And in 30 second increments, I’ve characterized what we actually did. And what you see is that we spent most of our time on bug investigation with only a little bit of actual progress toward the thing that we were trying to do in our test session. So our goal in the test session was to create a product coverage outline based on an understanding of the features of the product. We got one quarter of the way through that in the 20 minutes we’re analyzing right here. And the rest of the time, we were distracted by bugs.

This kind of analysis where you break the amount of time you spend into three kinds of time bug investigation and reporting productive bug finding work, which is the green stuff and yellow which is setup and admin time can be very useful if you are trying to get a sense and be able to explain to your managers and your coworkers why it is that you spend hours testing, but you don’t feel like you get much done. When you are able to do this kind of analysis, you get really good evidence. You can walk them through it. In fact, we did this with a whole week once this is made up data. This isn’t based on a real testopsy, but this is an entire week of what a week might look like for a tester where you are black as you’re spending time in meetings and the colors or when you’re trying to test. And this shows that in an entire week of testing, you might only get one day of testing worth of actual work done.

You get actual coverage in one week of testing. You’re doing testing work all the way through the week. Distinguishing between performing tests and doing testing work. That’s important. Our clients are constantly thinking that every moment of our time is spent performing tests. But actually, a small part of our time is spent performing tests because we get interrupted by bugs. We get interrupted by setup activities.

Each block being 30 seconds. 20 minutes of testing. Ostensibly, we could cover 40 featurettes little tiny aspects of a feature and test those things, but all that time, so much of that time is being interrupted by bug investigation and reporting and a little tiny bit of time being interrupted by something we call setup work. Now, this isn’t our whole session. This is just 20 minutes of that 75 minutes. That’s important because the rest of it we actually got a lot done. This is just an interesting part of our session that turned into a bug cascade.

So one of the questions is like, how did you collect this data? Like if you were trying to take this information and present it to your managers, how would you actually go about doing that? Does it require you to having recorded your information?

Over the course of a week? What we would suggest people do and you don’t want to do this all the time. It would be super, super irritating. But there’s a couple of ways you can do it. One is for a day or a week, set up something to alert every 10 minutes and take a literal 5-second note on some kind of list or in some kind of notebook that you’ve got, you know, your Moleskine or a computer based notebook or something on your phone. But every ten minutes, take a one word or so note on what you were doing over the previous ten minutes. Super annoying to do that for even a whole day but you don’t have to do it forever all the time. Another approach is to get an intern or a sociologist and anthropologist to follow people around.

The important thing to understand about this is we’re not talking about doing this all the time. What we’re talking about is when you decide I want to make another step toward learning how my own test process really works. Then you can set aside half a day to watch yourself very carefully, to video yourself. You can also get together with other people and look very carefully at the work that you do. How you actually collect the data is, I would suggest, video as the first step, but you just sit with a notebook and you can also in real time watch somebody else test and I’ve done that and then write down what they do. I have found that I miss a lot when I do that. So that’s why I prefer video so that I can look at it very carefully. But the important thing is that it takes a lot of time to do a testopsy properly. In took us probably eight hours to analyze this seventy five minutes of testing, at least.

We don’t do this all the time. What we are trying to point out is that we get a lot out of doing this once in a while. And by doing that, you can just by setting an alarm and taking notes for the course of two days so that you just have two days of every 15 minutes, you’ve got a log of what you were doing. Just that, will allow you to see patterns that you can talk about for weeks and weeks.

Here’s an example. When I was at a client, there was a I was assigned to observe testers doing testing ostensibly and over two of the sessions that I looked at, one of them was a little less than half of it was actual test coverage and the rest of it was setup and bug investigation. In the other session, it was a mass of yellow. It was all setup work. And yet management was curious as to why testers weren’t finding bugs. And when we analyzed why this stuff was yellow, the answer was that the conditions that management had set up made it really, really difficult for testers to get testing work done. This was a tricky product to set up. The server lab was far enough away from the tester that she had to get up and walk to it to make an adjustment to things in the server room. She did that 11 times over the course of a four hour afternoon. And in those 11 walks, each walk was three minutes round trip. Never mind what she did when she was actually there. So, you know, it put the tester closer to the server room. In cases like that, was it an insight that I was able to provide to that company right away?

A formal log that I prepared for a project where I was developing a process for Microsoft a long time ago, in 1999, which goes to show you that you don’t have to do this a whole lot because what I learned by doing this one day of intensively analyzing a several hours of testing. I then applied to my class. I’ve created a lot of class materials based on what I learned from doing this. And I used an alarm that every 15 minutes automatically reminded me to say what I was doing and took a snapshot of my notes. And then I have the snapshots of all the notes. So I collected this data just one time for about four hours and I learned so much about how exploratory testing works that I’ve been teaching based on that for a long time. So I’ve done different kinds of different levels of these kinds of testopies and a little bit goes an amazingly long way. We probably have a half hour’s worth of new material for our class or for conference talks. In fact, expanding on the Bug Reporting Sieve or expanding unnecessary confusion. Each one of those is an hour’s worth of discussion and examination.

So often a testopsy and results we get from it turn into a point of departure for even deeper study of a particular thing. More notes from years ago where my pair watching me in real time and he was writing down everything that I was doing. So he was trying to characterize my entire test process that I did live. So this is a live testopsy. So this is what I mean, when I say my pair and I have been doing bits and pieces and different kinds of stuff like this for years. We’ve done it in different ways. And if you want to know if there’s a bottom line, if you think my pair and I have any insights to give about testing, this is why this is the main reason why. Because we have learned to watch our testing very closely and we see all kinds of amazing things there that most people aren’t writing about. Most people aren’t talking about. And I think the secret reason for that is that people who write about testing don’t do testopsies. They should be watching their testing very carefully. But I think they generally don’t.

So, there’s other questions here: is there any particular recording software you recommend? – I’ve used BB TestAssistant years ago, it was my go to thing, but I don’t know if it still exists. It costs a couple hundred dollars and I’ve been reluctant to upgrade to the latest one. These days I’ve been using Snagit to do videos or my pair and I were using Zoom since we were telecommuting with each other. We just used Zoom recording and that worked for us. But I’ll tell you, when I do this really professionally, I set up a video camera because video cameras have never lost my data. But every screen recorder tool has lost my data. And to add on to that.

So if you can use Zoom, then you can use other free ones like Google Hangouts and that kind of thing as well. Not the best quality, but you can. I recommend do not use a GoPro camera. I tried that once. The problem is GoPro doesn’t have an analog zoom, so you end up having to zoom into the screen and then it gets all pixelated. But with a regular video camera you can set it up some distance away to an analog zoom right into the screen and it’s great.

So another one of the questions is regarding the video recording: How long do you record your sessions for? How long do you keep the data, that kind of stuff? – Well, this stuff, we never throw away the data. Basically, I don’t throw away the data. This is that we just did a testopsy based on something we did in July that we just found on our disk drive like, here’s something, let’s do that. I’m very glad I kept this because this was wonderful doing this. This testopsy you can keep about 100 days of testing video because it runs. I find it runs about 15GB a day. You can keep like 80 days of testing video on terabyte drive and terabyte drives are cheap. So I would just keep the data. Now, sometimes you work for companies that won’t let you do this. I’ve had problems in banks where they didn’t want me to do video and I understand. But in most cases, that’s not really super sensitive.

One thing that happened, I found a bug once because I forgot to turn the video off. I left the video on during lunch and I came back and the thing had been staring at the screen all while I was gone. And for fun, I just ran the video back in fast motion just to see if anything happened while I was gone. And they found out that a temperature control system that was supposed to hold the temperature of a water bath at exactly a certain temperature was slowly warming up over the course of the hour. You couldn’t notice it unless you played it in fast motion. And then even for this testopsy, even for the video that we just saw. One of the things that we noticed in that Bug Cascade is we noticed bug after bug after bug that we had not recorded because at some point they fell through the sieve.

So a few more questions: So do you think that observing your testing changes, your testing also known as the Hawthorne effect, and maybe you talk a little bit about that? – Definitely. It’s when you do testing in a pair, you are definitely testing differently than when you’re alone because you feel the need to explain yourself. You feel the need not to waste the other person’s time. So there’s already a kind of a Hawthorne effect just by working with another person. There’s less of that when you just do a video and you don’t think about, it’s easy to forget that anyone’s going to look at this or that you’re going to look at this later and you just you just start testing and you just get into it. It’s definitely different when you’re testing on stage and someone’s watching you, which I’ve done a number of times while being, you know, commented at by my pair or my brother.

It definitely changes what you’re doing. And one big thing to think about is that and this, by the way, this is nothing to testopsies. This can have to do with just having the wrong kind of manager. But if you’re doing something, let’s say, with automation and you’re having trouble making the automation work, a lot of people working with automation will pretend that it didn’t happen. They’ll work all night to get the automation to work, and they’ll pretend that they watched movies all night because they don’t want management to think that they’re incompetent. They’re worried that if they reveal to management just how much work it is to get this automation to work, then they’ll say, you must not be good at it. Because all those all those advertisements said that it’s so easy to automate using X, Y, Z tool. So if you’re having problems, that must be your problem.

This is a hugely important point because I think almost everybody is lying about how hard this stuff actually is. That’s why we call the Secret Life of testers. So, for instance, there’s a programmer that I respect a lot named Gary Bernhardt. He has had some wonderful educational series on really deep fundamentals of computer programming. And in order to prepare those things, he has to do a lot of video recording and audio recording. And he’s a pretty thorough checker of his own work. And he tweets about how the tools that he’s using to the system with this are driving him crazy. And it’s really wonderful to see at least one person saying, hey, this stuff is really hard and irritating. And a lot of the stuff that’s supposed to make things really, really easy doesn’t work. And a lot of the new stuff that comes along to make it easier still introduces new problems of its own.

We spent four hours trying to get cucumber to work, Michael. And then we had to reinstall Ruby and it messed up and we never did get it working. And a funny thing is when people who really like Cucumber say, you’re just not competent. I’ve been a programmer, a production developer. I worked at Apple Computer. I was on the Borland C++ team. And you’re saying I’m an incompetent programmer? Well, there must be lots of incompetent programmers in the world then. If I can’t get this thing running. So there’s a lot of denial in our industry about how fiddly this stuff is and how hard it is to get it to work. What we think we need to do is display to people, to management, to our clients what’s really going on so that we can start understanding the true cost of these processes and the true value of these processes. We want to get credit for the real good testing that we’re doing and we want to not be dinged for things that aren’t our fault.

Somebody said that the basically we need to write a book that’s called The Secret Life of Testers someday. That that would be a great title for a book.