Mutation Testing
This is the third in a series of Visual Connections blogs focused on exploring continuous testing.
Our goal is to share industry best practices, help you reduce costs and testing efforts, and increase your return on investment (ROI) potential.
Mutation Testing
By:
Fred Deese | Visual Connections President & CEO
Bas Dijkstra | Visual Connections Testing Engineer & Test Automation Expert
In response to the increasing demand for delivering high-quality software at speed, many organizations have adopted -or are in the process of adopting- a Continuous Delivery strategy. A vital part of this strategy is the effective leverage of test automation to have insight into current application quality at all times, a process known as Continuous Testing.
Mainly, these Continuous Testing efforts are targeted towards writing and executing functional automated tests, anywhere from the unit test level all the way towards full-stack integration and end-to-end tests. However, to get a complete overview of the current state of application quality, only performing functional test automation is not enough.
This three-part series introduces three of the many other techniques that you can leverage as part of your Continuous Testing approach to get a better and more complete indication of application quality at all times during the software development lifecycle.
Mutation Testing
The final technique discussed in this three-part series does not concern testing your application under test, or at least not directly.
For all the time and effort that teams and organizations spend on building and maintaining automated tests that should safeguard the quick and successful delivery of an application into a production environment, there is often still a shocking disregard for the quality and the value of the automated tests themselves. Teams are so busy creating ever more automated tests to keep up with application development and increase their test coverage that they spend little to no time on the periodical assessment of the quality and the usefulness of their test automation suite.
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.
Over time, this might lead to bloated automation suites that take a long time to run and maintain, as well as a large portion of tests that have lost their defect detection power and therefore are of little value in safeguarding application quality.
One of the techniques that can help gain insight into the value and effectiveness of your tests is mutation testing. This technique involves making small changes to your application code (creating ‘mutants,’ hence the name) and then running your tests against those mutants to see if they can detect the changes (‘kill the mutants’). Here, failing tests are actually a good thing: this means that they have detected a change in the application under test.
Sometimes, after running a mutation testing program, there are mutants left that have not been killed. This may be for one or more of these reasons:
The code has not been executed by at least one test, but the existing test suite is deemed to be of sufficient coverage. This implies that the code might be considered ‘dead code.’
The code has not been executed by at least one test, but this code is not ‘dead code.’ This is an indication of insufficient test coverage.
The code has been executed by one or more tests, but they have not picked up the change that has been put in there by the mutation program. This is an indication of an ineffective test.
This means that mutation testing can tell you not only which parts of your code are not yet under test (similar to what code coverage tools do), but also whether or not your tests are potentially weak.
There are various types of mutations that mutation tools can perform, including database, XML, and design and specification mutations, but the most common form is program mutation, where the source code of applications is subject to mutation. Typically, mutation testing is also applied using only unit tests, since creating and deploying a large number of mutants and then running integration or end-to-end tests against them quickly becomes computationally inefficient.
There are several sophisticated mutation testing tools available nowadays, most notably Stryker (https://stryker-mutator.io/stryker/) for JavaScript, TypeScript, React, Angular, VueJS and NodeJS, Stryker.NET (https://stryker-mutator.io/stryker-net/) for C# and PIT (http://pitest.org/) for Java.
Blog 3 in this Series
See other blogs in this Series
Consumer Driven Contract Testing
Service Virtualization
Fred Deese
After nearly a decade of providing advanced web and application development services to both the private and public sector, Fred Deese co-founded Visual Connections, an information technology consulting firm, in 2007. As Chief Executive Officer of the company, Fred is focused on new business development, overall company positioning, and client relationships.
Since the company's inception, he has held project management positions on many prominent and complex commercial and government IT projects, personally providing the services that have made Visual Connections widely recognized by its customers for its innovative technologies and quality work. See Full Bio Here
Bas Dijkstra
I help teams and organizations improve their testing efforts through smart application of tools.
In my 13+ years in the testing profession I have successfully designed and implemented test automation solutions for a wide variety of clients. I’m currently mostly active as a trainer, teaching people and teams how to make their first or their next move in the world of test automation.
I love to share and expand my knowledge of the test automation field by delivering talks, workshops and training courses, both at conferences as well as on-site with my clients.
You can contact me on LinkedIn via https://www.linkedin.com/in/basdijkstra or through my blog OnTestAutomation.