Comparison of the combined fuzzing algorithm based on mutation analysis with the fuzzing algorithm based on code coverage

DOI: 10.31673/2412-9070.2023.025459

Abstract

This article presents a comparative analysis between the μ2 algorithm and the Zest algorithm using the same amount of time rather than inputs and comparison by metrics other than mutation analysis. The Zest algorithm, embedded in the JQF framework, prioritizes comprehensive code coverage by generating inputs that satisfy both structural and semantic requirements. In contrast, μ2 is an approach that expands the scope of coverage-guided fuzzing through the integration of mutation analysis, which, in turn, enables the generation of higher-quality test cases. The obtained conclusions confirm that mutation-based fuzzing requires more time for generating input data that ensures complete code coverage. The research has revealed that under equal conditions, the combined algorithm based on mutation analysis provides less overall code coverage compared to the code coverage-based algorithm. Furthermore, it is noteworthy that the approach of fuzzing guided by mutation analysis, such as the one exemplified by the μ2 algorithm, is relatively less explored in the existing body of research. In reality, the shortage of publications dedicated to the topic of fuzzing based on mutation analysis is evident from the limited references in fuzzing review papers. This gap in awareness might contribute to the restrained popularity of mutation analysis within the security research community and in software testing. Additionally, this article aims to enhance the visibility of mutation analysis among fuzzing researchers.

Keywords: fuzz testing; mutation testing; mutation analysis; test generation.

References
1. Gopinath R., Görz Ph., Groce A. Mutation analysis: Answering the fuzzing challenge. arXiv preprint // arXiv:2201.11303, 2022.
2. Systematic Assessment of Fuzzers using Mutation Analysis / Ph. Goerz, B. Mathis, K. Hassler [et al.] // 2023. Usenix Security.
3. Guiding Greybox Fuzzing with Mutation Testing / V. Vikram, I. Laybourn, Ao Li [et al.] // ISSTA 2023248.
4. Padhye R., Lemieux C., Sen K. JQF: Coverageguided property-based testing in Java // Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 2019. p. 398–401.
5. Semantic fuzzing with zest / R. Padhye [et al.] // Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 2019. p. 329–340.
6. Laybourn I. μ2: using mutation analysis to guide mutation-based fuzzing // Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. 2022. p. 331–333.
7. Reachable Coverage: Estimating Saturation in Fuzzing / D. Liyanage [et al.] // Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE’23), 17-19 May 2023, Australia. 2023.
8. Investigating Coverage Guided Fuzzing with Mutation Testing / R. Qian [et al.] // Proceedings of the 13th Asia-Pacific Symposium on Internetware. 2022. p. 272–281.
9. Guidelines for Coverage-Based Comparisons of Non-Adequate Test Suites / M. Gligoric [et al.] // Space. 2014, 6.1,350: 1,142.
10. Hemmati H. How effective are code coverage criteria?: 2015 IEEE International Conference on Software Quality, Reliability and Security // IEEE, 2015. p. 151–156.
11. Can this fault be detected: A study on fault detection via automated test generation / P. Ma [et al.] // Journal of Systems and Software. 2020. 170: 110769.