Exploring Exploratory Testing

Exploratory testing (ET) is both conceptually familiar and technically foreign: The idea is widely used – in some form or another – yet the discipline isn’t widely studied. ET can summarily be described as concurrent learning, test design and test execution and while that, to some, may insinuate sloppiness, immeasurability and ambiguity; ET is as much of an intellectual skill as mathematics or chess. This is to say, your familiarity with the principles of ET are a fundamental indicator as to how successfully you can employ it. Further, if done correctly, you can execute organized, measurable and precise tests.

James Bach, author of ET Explained, best illustrated ET by likening the process to solving a jigsaw puzzle and I think this analogy serves us well. When considering the solution to a jigsaw puzzle, there are testing behaviors for solving the puzzle that are more common than others. For instance, you might agree that an uncommon behavior would be brute force: taking 1 puzzle piece at a time and comparing it against all other pieces, at all different angles (Exhaustive Testing). In the same vein, you wouldn’t write up a step-by-step instruction guide on how to complete the puzzle before trying to solve it; just as you wouldn’t follow a pre-written step-by-step instruction guide (Scripted Testing). Instead, more commonly, you attempt to find the next best possible action given the current state of the puzzle and that action heavily depends on previously gained knowledge and actions (Exploratory Testing). In this way, ET can be described as extremely situational. For a more in-depth analysis and description of ET, you can reference Bach’s article. Continuing, I’d like to switch gears and introduce a common way to effectively utilize exploratory testing.

Effectively utilizing exploratory testing

In the past, critics of ET pointed to its lack of preparation and review, which they believed resulted in sloppy and ambiguous tests. However, the introduction of session-based-testing has enabled ET to become a much more powerful and agile testing methodology. Session based testing (SBT) involves a bit more in-the-moment and post-assessment documentation as opposed to pre-approved and pre-scripted test plans and documentation; this allows the tester the freedom to make calculated decisions while still capturing some key metrics. Specifically, most session-based testers record things like

• number of sessions completed
• number of problems found
• functional areas covered
• percentage of session time spent setting up for testing
• percentage of session time spent testing
• percentage of session time spent investigating problems.

While one doesn’t have to strictly adhere to these metrics, they’re a good foundation. A common SBT pattern then calls for a manager and tester to meet and discuss the findings of the session. It’s important to note that while exploratory testing is a testing methodology, session based testing is a complimentary structure to enhance ET, and isn’t necessarily a testing methodology in itself – the two aren’t mutually exclusive, but rather complimentary.

Testing at Constant Contact

At Constant Contact we have many diverse teams who use at least as many diverse technologies and testing methodologies, but our testing bread-and-butter remains a mix of automated and scripted testing complemented by sessions of exploratory testing. For instance, on the team where I am positioned, we run a robust suite of automated regression testing each night against a production-like environment. We bolster those results with an automated smoke test run in all major browsers, to give peace of mind when releasing small product features or changes. In tandem with the automated suites, Quality Engineers not only validate the existing automation via overlap, but expand upon it with ET; testing scenarios that aren’t easily captured by scripting and automation, but instead by calculating the next best possible action based on a history of previous interactions.

Share your thoughts and experiences about Exploratory Testing in the comments section.