BBST Domain Testing – An Experience Report

In late January of 2014, after the Workshop on Teaching Software Testing (WTST) at Florida Institute of Technology, Dr. Cem Kaner and Dr. Rebecca Fiedler put together a 5-day pilot course to beta test a new Black Box Software Testing (BBST) course called Domain Testing. I was one of ten participants to try it out.

Two series of BBST

• The first BBST series (Foundations, Bug Advocacy and Test Design) came from research funded by the National Science Foundation and Dr. Kaner. Part of the agreement with the NSF was to make the materials open source while also creating a way to teach the materials online with high standards. The Association for Software Testing (AST) was the lab for the initial classes and they continue to teach them to this day.
• BBST Domain Testing represents the next step in a second BBST series focusing on specific test design techniques (Domain, Scenario and Specification based testing). Unlike the first series, the NSF didn’t fund the development of these classes so the materials won’t be open sourced. They also won’t be taught through AST, instead they will come directly from Dr. Cem Kaner’s corporate training firm Kaner Fiedler & Associates (or KFA for short).

What is domain testing?

Although domain testing sounds like it relates to domain knowledge, it is an umbrella term for equivalence class analysis (partitioning) and boundary testing. Domain testing is a sampling strategy where possible values of a variable (anything that can change) are divided into a subset of values that are in some way equivalent (equivalence class analysis). Then tests are designed to only use one or two values from each subset, along with boundary and extreme values that increase the likelihood of exposing bugs.

Why use domain testing?

Designing tests is about making choices. Choices such as:

• How many tests do we want to run (from an impossibly large set of potential tests)?
• How powerful are the tests we have chosen?
• What do we hope to learn from the tests we’ll run?

When used appropriately domain testing can help us increase our efficiency by helping us run less redundant tests and increase our effectiveness by helping us find bugs thanks to powerful tests.

The Workbook

The Domain Testing Workbook (available on Amazon or directly from Context Driven Press) was published for those interested in self-study but also with the intention of using it in a class. The book contains among other things, a schema (list of ideas or step by step sequence of questions to answer) and examples ranging from the easy, classic problems, to increasingly more difficult examples. During class I used the book frequently for background context, explanations of concepts and examples, details of the Schema and for the many ideas sitting in the books multiple test catalogs.

The Class

Prior to the class, my experience applying domain testing consisted of trying to overflow input and output fields. The problem with the way I was doing things, besides being ad hoc, was I didn’t utilize the power of domain testing as a sampling technique. That’s what inspired me to take the class.

The pilot class was in-person for five full days at Florida Institute of Technology’s campus. Dr. Kaner delivered lectures in the morning and afternoon followed by exercises after the lectures and concluded the day with homework for additional practice. For example, on our first day, we learned how to characterize variables and worked through classic examples that originally appeared in the book Testing Computer Software.

Our first assignment was to demonstrate we were comfortable doing a variable tour (variables the basis for domain testing) of our sample application, FM Starting Point (a FileMaker Pro template), using Xmind. After the tour it was time to classify a smaller set of variables (a dozen) based on their data types; determine their primary dimensions and what benefit, if any, would come from doing domain testing. Finally that information went into a classical table.

On our second day we pushed further. We continued to classify variables and build classical tables only with more complexity than before. Continuing with the Schema we turned to creating a risk equivalence table. Risk equivalence tables are essentially risk-based versions of the classical table but you explicitly talk about the risks (or failures) your tests hope to expose and therefore describe why your test designs are powerful. The challenging and time consuming aspects of creating risk or classical tables is coming up with failure modes for tests – luckily the workbook has catalogs you can use for inspiration.

Our third day continued our practice of variable analysis; creating classical and risk based equivalence tables but added in pair testing of independent variables using the ACTs tool (Microsoft’s PIC tool that we used in Test Design was also an option). Our final days in the class focused on putting together the individual things we learned from the Schema, culminating in a capstone project.

The capstone project allowed us to showcase the skill and understanding we gained from the class into action by choosing a piece of an application and working through each step of the eighteen step schema. It was a difficult assignment but certainly the most fun part of the class. I made a lot of mistakes as I went through the steps without the assignments as a guide but the capstone allowed me to figure out what I like and don’t like, what order to address aspects of the schema (I didn’t like going in logical order) and going forward I’ll be more efficient and effective.

Future Classes

The pilot was a bit different from how BBST Domain Testing courses will be handled in the future. Besides the obvious in-person versus online transition, there will likely be a wide range of example applications to practice domain testing on.

At the time of the pilot (January of 2014) the class had two finished sets of real world example videos – QuickBooks (account software) and Electric Quilt 7 (design software). There were plans to add more real world examples including a Sewing machine (not your grandmothers sewing machine but a modern one to serve as an example of an embedded device), a video game (for a glassbox approach), an investing application and a database application.

Taking the class changed the way I look at testing

I barely remembered the introduction of domain testing in BBST Test Design after the class was over – I just didn’t get it. Things went by so fast, at such a high level that made it difficult to understand how to address things properly. After the class was over I couldn’t apply it to my work. This time things are different. After BBST Domain Testing, using the Schema and workbook I get domain testing.

When I look at an application today, I have a strong sense of where the strategy can be applied and where it shouldn’t. That makes me more confident in my abilities (not over confident, I’m still a novice), gives me lots of new and interesting ideas and a place to start practicing. More importantly I can actually see myself applying the technique to the software I test on a regular basis.

When Becky and Cem asked what I thought of the class I said:

it wasn’t as easy as I thought, but it was fun!

References

1. Kaner, Padmanabhan & Hoffman. The Domain Testing Workbook. Context-Driven Press, 2013.
2. Kaner, Cem. Teaching Domain Testing: A Status Report.

This article was originally published in the February 2014 edition of Testing Circus Magazine. This was such a great learning opportunity I wanted to highlight it by slightly updating and reposting it.