Regression testing is the use of existing test suites to reassess software systems as they evolve, with a goal of ensuring that existing functionality is maintained.

Most testing is regression testing.

"Also as a consequence of the introduction of new bugs, program maintenance requires far more system testing per statement written than any other programming. Theoretically, after each fix one must run the entire batch of test cases previously run against the system, to ensure that it has not been damaged in an obscure way. In practice, such regression testing must indeed approximate this theoretical idea, and it is very costly." — Fred Brooks, The Mythical Man Month

Automated regression testing is the central focus of continuous integration systems.

Long-Term Testing

• Tests may be initially designed to validate specific functionality.
• Once created, test cases are used over and over again.
  • 10 check-ins/hour, 12 hours/day, 300 days/year for 5 years = 180,000 potential executions per test case.
• The reusability and maintainability of test cases is far more important than the initial convenience for validation.

Too Many Tests?

• Test suites grow in size, and test configurations grow in complexity.
• Google is finding that, even with all of its resources, the growth in test cases may be unmanageable.
• How can regression testing be managed?

Change-Centric Testing

• Selectively apply test cases that are relevant to a new check-in.
• Coverage-based:
  • Record statement coverage per test case.
  • Identify the change set of modified statements for a check-in.
  • Run only those tests that cover the statements in question.

Test Failure

• Eventually tests fail.
• What causes the failure?
• Changes to input-output requirements: replace/modify test.
• No change in requirements:
  • New check-in may have a bug to fix.
  • Or the test itself may be need to be repaired.
    • The test is valid, but the harness does not account for a new configuration.
    • Test data is stale, or new data is being used.
    • The test oracle for this case has failed: modified software has a benign change to output format.

Test Robustness

• Is it possible to make tests robust?
  • Consider making checks for test output flexible enough to accommodate changes to output format.
  • Oracle can use pattern matching rather than exact match.