SecurityFocal
Fuzzing or Fuzz testing is increasingly getting popular and is used and advocated by some of the most reputed organizations, including Apple, Microsoft, Google, Cisco, and IBM. In this post, we will do a deep dive around what is Fuzzing, its advantages, disadvantages, and the tools used for Fuzzing.
Industry bodies & regulatory agencies requiring or recommending fuzz testing include NIST, FDA, IEEE, IETF, DoD, and ETSI.
Fuzzing is widely assumed as the next big thing in Cybersecurity. It is used to test software and systems and was developed in 1989 at the University of Wisconsin-Madison by Professor Barton Miller and his students. You can refer to their continued work here.
Fuzz testing is best used in conjunction with other testing mechanisms, including Beta Testing, Black Box Testing, and debugging. Fuzzing is also known as Dynamic Analysis or Negative testing.
What is Fuzzing?
Fuzzing is a software testing mechanism in which a software tester or an attacker intentionally bombards a software or system with invalid data to cause it to misbehave or crash. The data input is called Fuzz. The output is then analyzed to identify the root cause of the behavior at the programming level.
Fuzzing is the degree to which a system or component can function correctly in the presence of invalid inputs or stressful environmental conditions.
What are the types of Fuzzing?
Fuzzing can be divided into three different types based on the methodology used, which are as follows:
1. Black Box Fuzzing
Black box Fuzzing does not consider the internal architecture of the application and treats it as a Black Box. An example of a black box Fuzzer will be a Random Testing Tool that generates inputs at random. While highly efficient, the Black Box fuzzing in most scenarios can only scratch the surface and expose shallow bugs.
Due to this limitation, the developers are focussing on developing self-learning algorithms that can analyze the application’s output and improvise its inputs.
2. White Box Fuzzing
White Box Fuzzing employs program analysis to systematically increase the code coverage and test critical parts of the application coding. With the program specifications available, a While Box Fuzzer can use model-based testing to generate specific inputs to target the vulnerabilities and bugs hidden deeper in the program.
While Box Fuzzing, due to its methodology, is often considered slow compared to Black Box Fuzzing which is far more efficient. That makes developers focus on developing a mechanism to combine the efficiency of Black Box Fuzzing with the efficacy of White Box testing.
3. Grey Box Fuzzing
Grey box Fuzzing leverages instrumentation (e.g., coverage feedback) to reach deeper into the application. This is not Blackbox since it uses some program analysis and not Whitebox since it does not use the program analysis. While it adds a reasonable performance overhead, it makes Grey Box Fuzzing an efficient tool for vulnerability detection.
What is a Fuzzer?
A Fuzzer is a software tool used by testers to employ Fuzzing for Software Testing. There are many different types of Fuzzers available, including Mutation based or Generation based Fuzzers.
What are the Types of Fuzzers?
While a Fuzzer can be categorized in many different ways, we can broadly categorize Fuzzers in the following types.
1. Generation Based Fuzzer
A Generation Based Fuzzer starts from a product specification, typically the file format or network protocol, and builds test cases using the specifications. The Fuzzer needs to make each use case differ from valid data to cause a problem in the application, but not too invalid as the application will otherwise discard the input.
Generation Based Fuzzer requires a good amount of up-front work for testers to go through the specifications and manually create test cases. Often manually created test cases turn out to be too similar to the specification rendering the process limitedly effective.
2. Mutation Based Fuzzer
A Mutation Based Fuzzer collects known good data (Files, Network Traffic, etc.) and modifies it to create the input (Fuzz). These modifications can be random or heuristic-based. Heuristic mutations include changing small strings to long strings or changing the length values to too small or too big.
While Mutation Based Fuzzing is easier and assumed to be less effective than Generation Based Fuzzing, there have not been any attempts to establish it otherwise.
3. Evolutionary Fuzzer
Evolutionary Fuzzer uses feedbacks from each test case to self-learn the format of the input over time. For example, by measuring the code coverage of each test case, the fuzzer can learn which properties of the input exercise a given area of code. The fuzzer can then gradually evolve a set of inputs to cover the majority of the program code.
Evolutionary Fuzzer often relies on different techniques similar to genetic algorithms and typically requires binary instrumentation to work in the desired way.
4. Dumb Fuzzer
Dumb Fuzzer is a Fuzzer that generates completely random inputs; as it does not have any built-in intelligence regarding the program, it’s fuzzing. It is more efficient and cost and time-effective compared to a Smart Fuzzer.
Testers can usually start with a Dumb Fuzzer and increase its intelligence as the code quality improves. If an application gets several errors and crashes with a Dumb Fuzzer, there is no point in using a Smart Fuzzer unless the code quality improves beyond a certain level.
5. Smart Fuzzer
Smart Fuzzer is programmed with the knowledge of the input format, for example, a protocol definition or the rules for a file format. A Smart Fuzzer can then generate mostly valid input and fuzz the input parts within that format.
How does Fuzzing work?
Following are the basic steps required to be performed by a tester for Fuzzing:
- Identify the target application or system
- Identify the tools and type of Fuzzer to use
- Identify the types of inputs to be generated (application, protocol, or file format)
- Perform the test using invalid data (Fuzz)
- Observe application behavior
- Analyze the logs
Open source Fuzz Testing Tools
Following are some of the leading open-source Fuzz Testing tools:
1. Radames
Radames is a general-purpose Fuzzer for robustness testing. It is used to test how well an application withstands malformed and potentially malicious inputs. It can read sample files of valid data inputs and generate interestingly different outputs using them. Radames has already found several bugs in leading programs and applications. It is easy to configure, scriptable, and set it up.
Radames is more of a “Black-Box” Fuzzer since it does not require information about the program or the data format. However, you can optionally pair it with coverage analysis for testing to improve the quality of the sample set. Testers can start with easy tests and gradually refine the technique as required. You can find the Github link for Radamsa here.
2. Honggfuzz
Honggfuzz is a multi-process and multi-threaded Fuzzer. This eliminates the requirement to run multiple copies of the fuzzer since Honggfuzz can potentially use all your available computing power with a single running instance. The file corpus is shared automatically and improved among all fuzzed processes.
It is blazingly fast and has a solid track record of unveiling security bugs. Honggfuzz discovered the only critical vulnerability (to the date) in OpenSSL. You can find the Github link for Honggfuzz here.
3. American Fuzzy Lop
American Fuzzy Lop (AFL) is a free brute-force Fuzzer coupled with a rock-solid yet straightforward instrumentation-guided genetic algorithm. AFL uses a modified form of edge coverage to seamlessly pick up subtle, local-scale modifications to program control flow.
AFL is thoroughly tested to deliver out-of-the-box performance much superior compared to blind Fuzzing or coverage-only fuzz tools. You can access AFL from this Github link.
4. LibFuzzer
LibFuzzer is an in-process, coverage-guided, and evolutionary fuzzing engine. It is linked with the library under test and provides fuzzed inputs to the library using a specific fuzzing entry point. The fuzzer can then track which code areas are reached and create mutations on the corpus of input data to maximize the code coverage.
LibFuzzer can be accessed at this link.
5. Sulley
Sulley is a Fuzzing engine and a Fuzz testing framework that consists of multiple extensible components. Sulley (IMHO) is known to have exceeded the capabilities of most previously published fuzzing technologies, including in the commercial and public domain. The goal of the Sulley framework is to simplify the data representation, data transmission, and instrumentation.
Sulley is affectionately named after the fuzzy creature from Monsters Inc. movie because he is fuzzy. You can access the GitHub link for Sully here.
6. OSS-Fuzz
The OSS-Fuzz tool was primarily designed to work with open-source software to make them more secure and stable. OSS-Fuzz supports open-source programs written in C, C++, Go, Rust, Python, and Java/JCM. OSS-Fuzz may also work with other languages supported by LLVM (https://llvm.org/). OSS-Fuzz supports fuzzing x86_64 and i386 builds.
You can access OSS-Fuzz at Github using this link.
7. APIFuzzer
APIFuzzer does not require coding. It reads through your API description and can step by step fuzz the fields to test if your application can stand the fuzzed parameters. APIFuzzer supports Fuzzing of request body, path parameter, query-string, and request header.
You can use this link to access APIFuzzer.
8. Jazzer
Jazzer, developed by Code Intelligence, is a coverage-guided, in-process fuzzer for the JVM platform. It is based on libFuzzer and extends many of its instrumentation-powered mutation functionalities to the JVM.
The JVM bytecode is executed within the fuzzer process, ensuring fast execution speed and seamless Fuzzing of native libraries. Jazzer supports x64 Linux and experimentally supports macOS 10.15. You can access Jazzer on this Github link.
9. Boofuzz
Boofuzz is a fork and the successor of the Sulley fuzzing framework. While Sulley has been the leading open-source fuzzer for some time, it has fallen out of maintenance. Boofuzz includes numerous bug fixes and aims for extensibility with the goal to Fuzz everything.
You can access the Boofuzz Github link here.
Following are the advantages and disadvantages of Fuzz Testing:
Advantages of Fuzzing
1. Fuzzing is extremely efficient
Once a fuzzer is set up and running, it can work without any intervention for days or even months to identify bugs in an automated way. It can significantly reduce manual efforts.
2. Fuzzing helps to Identify high severity bugs
Fuzzing often helps find the most severe bugs that hackers can exploit, including memory leaks, crashes, and unhandled exceptions.
3. Fuzzing reveal bugs missed in manual review
Fuzzing often reveals bugs missed in a manual audit and the bugs missed by other testing methods due to the limitation of time and resources.
4. Reveals a high-level picture
Fuzzing provides an overall view of the robustness of the application tested.
5. Zero False Positives
Fuzzing does not produce any false positives. If the Fuzzer found an error, it’s a problem that needs to be addressed.
6. Fuzzing is scalable
Several tools can test the application simultaneously, which makes Fuzzing highly scalable.
Disadvantages of Fuzzing
1. Fuzzing is time consuming
Fuzz testing may take a long time to yield effective results.
2. Fuzzing is ineffective against malware
Fuzzing focuses on the threats which may cause a program to crash or error. However, it’s ineffective against threats like Trojans, Viruses, and other Malware. For more details around these threats, please refer to our earlier blog post.
3. Testers need to still invest in other testing methods
Fuzzing is best used in conjunction with other testing methods like Black Box Testing, Manual Testing, etc.
4. Result Interpretation is difficult
Fuzzers typically provide high-level information which describes the issue and usually does not provide a clear path to remediation.
In Software Engineering, Fuzz testing highlights the bugs present in an application. Fuzz testing cannot guarantee detection of all the bugs in an application. But by employing Fuzzing, you can make sure that the application is robust and secure. Fuzzing effectively helps to unveil most of the common vulnerabilities.
The methodology of Fuzzing relies on the foundation that there are bugs within every program waiting to be uncovered. Hence, an organized, systematic approach will find them sooner or later.
Fuzzing can also add another dimension to classical software testing techniques, including manual code review and debugging since it’s a non-human and automated approach. While it does not replace them, it most certainly complements them nicely.