My Testing Philosophy

As I study software development, especially testing and quality, I've begun to develop a general philosophy.

Now when I say philosophy I mean:

[T]he critical study of the basic principles and concepts of a particular branch of knowledge, especially with a view to improving or reconstituting them.

Or more simply stated:

[A] foundation for studying and improving my ideas on software testing.

I’ve generally been aware of this development (others have highlighted it during discussions) but when I saw Andy Tinkham go through the exercise of writing out elements of his philosophy, I wondered if I could do something similar? I borrowed heavily from Andy’s list and think this is a good start. It will and has evolved over time.

Here’s my foundational understanding of software testing and quality:

• Software is eating more of the world. Thus, quality is becoming more complex and important.
• Quality is subjective. What one person values another might not.
  • For this reason, customers are the only ones capable of judging and evaluating the quality of our product.
• Quality is everyone’s responsibility (the whole team).
• Software needs to solve the customer’s problems or it is useless.
• Testing is a challenging intellectual process.
  • Thus, any testing activity that discourages thinking or questioning while performing it is potentially harmful. Test execution off of pre-written scripts is often an example of this.
• There are no best practices, just good practices for a particular context. Blindly applying a practice to a situation because it worked somewhere else can cause more harm than help. Understanding the context of any practice will be used in is crucial.
• Complete (or exhaustive) testing is impossible and therefore involves tradeoffs.
• Testing is about providing information to the individual, team and stakeholders. If testing is failing to provide the needed information, it is wasting time and resources.
• To provide that information, test reporting needs to clearly tell 3 stories:
  • The story of the product quality
  • The story of the testing done and not done, and
  • The story of the testing quality (why what was done was or wasn’t sufficient).
• A test is an experiment designed to reveal information (or answer a specific question) about a product or service.
• To be successful, the tester needs to approach a SUT trying to dis-prove the existing understanding (for example, that it has bugs) rather than proving some functionality is correct.
  • Otherwise, we run the risk of falling victim to confirmation bias.
  • It’s all about empirical evidence.
• Tests are only ever as good as their oracles.
• Testers are not the gatekeepers of the quality. They should have a voice, but not the sole voice, in release decisions.
• Developers can and should learn to test as well as testers. They just might not have as much time.
  • Testers can help coach and improve the developer’s ability
• The value of any action has 3 components: The benefit gained by completing the action, the costs incurred by performing the action, and the hidden costs of not getting the benefit from performing other actions which can no longer be performed. Choosing the right action at any point in time means striking a balance among these 3 elements.
• Writing out detailed, explicitly defined tests is rarely the best use of time and in many cases it is improbable (to fully detail them).
  • Letting the tester executing tests make decisions that don’t impact the test’s goal leads to better testing.
  • Writing down every expected result can be a huge amount of work on one testing task and often doesn’t consider what other testing tasks remain.
• Writing tests at the start of a cycle of testing effort means we are creating those tests at the point where we have the least knowledge about what we’re testing that we’ll have during the cycle.
• Communication is essential.
• Variety in tests, techniques and approaches is essential.
• Conventional software testing metrics are often misleading and can lead to undesired behavior. However, they’re what we have to work with at the moment, so until something better comes along, we need to use them to approximate the information we need. We just need to ensure we use them wisely and be conscious of the impacts and risks in doing so.
• When necessary, Bug Reports should be written in a clear, concise and persuasive way that convinces the reader to take some action. Writers of reports never know who will eventually read them.

Thanks to Andy for the inspiration and I highly recommend testers try this exercise. Think this was cool and looking for another exercise? Try thinking about your commitments to your programmers or your team and write them out like I have.

Think you can help? Feel free to challenge one or more of these elements, you’ll only be helping!

(Last updated on 08/17/2022)