Mutation Adequacy Study

Your first assignment will be to perform a mutation adequacy study of an existing test suite for an open-source software project.

You will work in groups of two to four students. The requirements will be dependent on n, where n is group size.

The following requirements apply to the open-source project.

• It must be a currently active open-source project with a significant organization and development community supporting it. For example, organizations sponsoring Google Summer of Code 2016 projects generally qualify as both significant and currently active (but don't necessarily confine yourself to the specific summer project).
• The code base must be at least 20,000 lines of code (20 KLoC).
• A substantial test suite must exist for the project.

Proposal

Your group must submit a proposal identifying the project and the test suite. Your proposal should include the following elements.

• Identification of the open-source project.
• Identification of the supporting organization.
• Size of the code base.
• Proposed evaluation platform (OS, language).
• Build time to compile and link an executable from source code.
• Execution time for the test suite.

This proposal is due Tuesday September 13 2016.

Assignment

Each group member should choose one source code file from the open source project and generate mutations to test the routines within it. Choose a file with complex routines that are likely to have errors.

Generate mutants by applying a small number of mutation operators to the code. You may apply the mutations manually, with a script or with a tool. Document your mutation operators. Each mutant should be identical to the original code except for the application of one mutation operator at one specific location. Each team member should generate at least 100 mutants for testing.

The source code for each routine tested should be submitted, together with a "diff" file showing a 3-line unified diff for each mutant created. This should be a single file with just a few lines for each mutant, not 100 copies!!

Apply the test suite to all mutants. Document the results: how many mutants are killed for each of the chosen routines. For every mutant that was not killed, try to determine either (a) that it is an equivalent mutant that should not be killed, or (b) how to add a test to kill it.

Calculate the effectiveness of the test suite. Effectiveness should be calculated separately for each routine tested. Overall effectiveness for each file should also be calculated using the ratio of total mutants killed to the total number of non-equivalent mutants created. Discuss your results and your methods, including how you might automate the generation of mutants and/or the application of tests. Focus on interesting observations and lessons learned.

The assignment is due Tuesday September 27 2016, 13:00. It is worth 10 per cent towards the overall course mark.

Results are to be presented in class on the following Thursday. The group should have one slide summarizing the project and its test suite. Each team member should present one graph or chart summarizing their individual testing results.