A Framework for Fuzz Target Generation and Evaluation
This framework generates fuzz targets for real-world C/C++/Java/Python projects with
various Large Language Models (LLM) and benchmarks them via the
OSS-Fuzz platform.
Overall, this framework manages to successfully leverage LLMs to generate valid fuzz targets (which generate non-zero coverage increase)
for 160 C/C++ projects. The maximum line coverage increase is 29% from the existing human-written targets.
Note that these reports are not public as they may contain undisclosed vulnerabilities.
Usage
Check our detailed usage guide for instructions on how to run this framework and generate reports based on the results.
Independent Agent Execution and Evaluation
You can also execute or evaluate individual agents without running full experiments, using the integrated agent execution framework.
See the framework's documentation for detailed instructions on how to run individual agents or sequence of agents.
Collaborations
Interested in research or open-source community collaborations?
Please feel free to create an issue or email us: [email protected].
Bugs Discovered
So far, we have reported 30 new bugs/vulnerabilities found by automatically generated targets built
by this framework:
These bugs could only have been discovered with newly generated targets. They were not reachable with existing OSS-Fuzz targets.
Current top coverage improvements by project
Project
Total coverage gain
Total relative gain
OSS-Fuzz-gen total covered lines
OSS-Fuzz-gen new covered lines
Existing covered lines
Total project lines
phmap
98.42%
205.75%
1601
1181
574
1120
usbguard
97.62%
26.04%
24550
5463
20979
3564
onednn
96.67%
7057.14%
5434
5434
77
210
avahi
82.06%
155.90%
3358
2814
1805
3046
pugixml
72.98%
194.95%
9015
6646
3409
7662
librdkafka
66.88%
845.57%
5019
4490
531
1169
casync
66.75%
903.23%
1171
1120
124
1678
tomlplusplus
61.06%
331.10%
4755
3652
1103
5981
astc-encoder
59.35%
177.88%
2726
1745
981
2940
mruby
48.56%
0.00%
34493
34493
0
71038
arduinojson
42.10%
85.80%
3344
1800
2098
4276
json
41.13%
66.51%
5051
3339
5020
8119
double-conversion
40.40%
88.12%
1663
779
884
1928
tinyobjloader
38.26%
77.01%
1157
717
931
1874
glog
38.18%
58.69%
895
331
564
867
cppitertools
35.78%
45.07%
253
151
335
422
eigen
35.38%
190.70%
2643
1947
1021
* "Total project lines" measures the source code of the project-under-test compiled and linked by the preexisting human-written fuzz targets from OSS-Fuzz.
* "Total coverage gain" is calculated using a denominator of the "Total project lines". "Total relative gain" is the increase in coverage compared to the old number of covered lines.
* Additional code from the project-under-test maybe included when compiling the new fuzz targets and result in high percentage gains.
Citing This Work
Please click on the 'Cite this repository' button located on the right-hand side of this GitHub page for citation details.
google/oss-fuzz-gen
