Fuzz Testing

Fuzz testing is the practice of testing software by bombarding it with randomly generated inputs.

Security Goal

• Systematically test software for security vulnerabilities.
• Identify cases where software crashes or operates incorrectly for incorrect or out-of-bounds inputs.
  • Make sure that the software does not suffer security failures:
    • Confidentiality - inappropriate access to data.
    • Integrity - corruption of data
    • Availability - crashes leading to denial of service
  • These failures may be triggered by many kinds of defects.
    • buffer overflows
    • division by zero
    • null pointer dereference
    • failure to terminate (infinite recursion)
    • uninitialized memory

Creating Fuzz

• Entirely random input to programs may catch vulnerabilities that lead to crashes.
• But restricted inputs may be more helpful.

Model-Based Generation

• In this case, inputs are based on some form of model of the input domain.
• For example, grammars can be used to generate random strings of the language described by the grammar.
• Frameworks may provide simple tools to construct models.
  • Simple Models with Google FuzzTest

Mutation-Based Fuzzing

• Another way of generating inputs for fuzzing is to mutate inputs already used in a test suite.
  • Small random changes are made to inputs, e.g. bit flipping, appending data, deleting data.

Coverage-Guided Fuzzing

• A variant of mutation-based fuzzing is to include a white-box source code coverage tools.
• Mutated inputs are then investigated to identify new code paths and potentially locate faults for poorly tested paths.
• AFL++
• LibFuzzer

Oracles

• Random inputs are very useful for finding security vulnerabilities due to crashes.
• But if normal looking output is produced, how do you check for correctness?
• That is, how do you created oracles for the randomly generated test cases.

Equivalent Functions

• One way to create oracles for randomly generate input is to rely on equivalent implementations.
  • For example, a high-performance implementation of a given function could be accompanied with a reference implementation that computes the same result.
  • Different language-based implementations may be used, for example a Python implementation may be used as an oracle for a C++ function.

Round-Trip Functions

• Some systems have complementary functions that are inverses of each other.
• For example: encoders and decoders for compression.
• Checking the encoder output of a randomly generated output may be done by confirming that the decoder can take this input and regenerate the random input.

Property-Based Systems

• In this case, we may define oracles that just check that certain observed properties of the output are correct.
• For example, one property could be the assertion that the output is a simple numeric value.
• For security vulnerabilities, another property is that the program does not crash.
• For denial-of-service vulnerabilities, ensuring that the output is produced within a given time period may suffice.

• Hypthosis Property-Based Testing for Python

Sanitizers

• Sanitizers are code instrumentation tools that help identify faults due to resource management issues in software.
• Sanitizers are often provided as options to compilers such as Clang.
• Address Sanitizer (Clang )
• Thread Sanitizer (Clang )
• Memory Sanitizer (Clang)

Sanitizers with Fuzzing

• It may be particularly useful to incorporate sanitizers when performing fuzz testing.
• Once an input that crashes a program has been identified, a sanitizer may help pin down the exact bug.

Hardening

• Control Flow Integrity
• SafeStack