Mutation Operators
Mutation Operators
A systematic mutation testing approach requires that a set of mutation operators be chosen. Mutants are then constructed from the source program by systematically applying the operators to each qualifying instance in the PUT (program under test).
FORTRAN: The MOTHRA Operators
The earliest systematic tools for mutation testing were developed for the FORTRAN language. The following 22 mutation operators were defined.
- AAR - array reference for array reference replacement
- ABS - absolute value insertion
- ACR - array reference for constant replacement
- AOR - arithmetic operator replacement
- ASR - array reference for scalar variable replacement
- CAR - constant for array reference replacement
- CNR - comparable array name replacement
- CRP - constant replacement
- CSR - constant for scalar variable replacement
- DER - DO statement end replacement
- DSA - DATA statement alterations
- GLR - GOTO label replacement
- LCR - logical connector replacement
- ROR - relational operator replacement
- RSR - RETURN statement replacement
- SAN - statement analysis
- SAR - scalar variable for array reference replacement
- SCR - scalar for constant replacement
- SDL - statement deletion
- SRC - source constant replacement
- SVR - scalar variable replacement
- UOI - unary operator insertion
Language-Based Mutation Operators
As illustrated by the FORTRAN case, mutation operators are often defined with respect to the syntax features of a particular programming language.
Java
Ada
Too Many Mutants
One of the key challenges of mutation testing is that too many mutants may be produced.
- Cost of compiling each mutant and executing the full test suite against it.
- Cost of analyzing unkilled mutants for equivalence.
Reducing The Number of Mutants
- Mutant sampling: choose only a sample (e.g., a random sample) of the mutants for evaluation: even a sample size of 10% can be effective.
- Selective-mutation: apply only the critical subset of operators.
- 5-selective mutation using only the 5 expression modication operators ABS, UOI, LCR, AOR, and ROR has found to be very effective.
- Note that these mutation operators are relatively language independent.
Strong vs. Weak Mutation
- Strong mutation: the mutant program must produce different output that fails to pass the test suite in order for it to be considered killed.
- Weak mutation: a mutant may be considered killed if the program state immediately after execution of the mutated operation is detectably different than in the PUT.
- Weak mutation may be computationally less expensive and also lead to a higher kill ratio.
Equivalent Mutants
The problem of equivalent mutants is a critical issue for mutation testing.
Recall that the effectiveness of a test suite is determined by TS= K/(M - E), where K is the number of killed mutants and M-E is the number of non-equivalent mutants.
- Determination of whether mutants are equivalent or not often involves a human in the loop as the test oracle, i.e., as the one who makes the decision.
- Human decisions on mutant equivalence are slow and also potentially error-prone.
- But there are automated techniques that can help.
- Trivial compiler equivalence: use the highest optimization modes of a compiler. In this case, many equivalent mutants produce exactly the same object code as the PUT, and are therefore known to be equivalent.