Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fuzz: Adds fuzz test #386

Merged
merged 1 commit into from
Feb 5, 2023
Merged

Conversation

nathaniel-brough
Copy link
Contributor

@nathaniel-brough nathaniel-brough commented Jan 28, 2023

Adds a fuzz test harness. I run the fuzzer for about 12hrs on 8 cores and didn't find any bugs, which is quite impressive :) So you've done well with your library. I thought I'd open a pull request with the hopes that the fuzzer can be integrated into the CI.


This change is Reviewable

@jarro2783
Copy link
Owner

Is this generating random input to the options parser and then checking that it doesn't segfault? Would you also use fuzzing to somehow check that a range of input is processed correctly (like integers being parsed correctly) or is it more often just to check that the program doesn't crash on weird input?

Can you integrate this with the CMake build as well?

@nathaniel-brough
Copy link
Contributor Author

Is this generating random input to the options parser and then checking that it doesn't segfault? Would you also use fuzzing to somehow check that a range of input is processed correctly (like integers being parsed correctly) or is it more often just to check that the program doesn't crash on weird input?

Yeah, that's pretty much it, although there are some heuristics to guide the process. For example, if a random input achieves good code coverage, instead of trying again with new random data, it'll randomly mutate the original data. There is also the option to combine it with various sanitizers. If you aren't familiar with sanitizers you can think of these as runtime "bug detectors". Here's a non-exhaustive list of sanitizers that are supported;

  • Address sanitizer: Detects buffer overflows, amongst other bugs
  • Undefined behaviour sanitiser: Among other things detects integer overflows/divide by zero's.
  • etc. (Search for sanitizer here for more details).

Can you integrate this with the CMake build as well?

Yeah sure, I rarely use Cmake so that might take me a bit to work out what I need to do :)

@nathaniel-brough
Copy link
Contributor Author

RE:

range of input is processed correctly (like integers being parsed correctly)

This is very much supported. Though the reason I didn't include it in this fuzzing harness is that most of that seems to happen using the >> operator which is either provided by the standard library or user provided. So I may be wrong here, but the assumption that I made was that the parsing of string->variable, wasn't really covered by this library. More so you provide a convenient interface to implement this on your own.

@nathaniel-brough nathaniel-brough force-pushed the master branch 10 times, most recently from b4f2f6a to 2b9dc41 Compare February 3, 2023 00:43
@nathaniel-brough
Copy link
Contributor Author

Some notes for recent changes:

  • I removed ubuntu 18.04 from the CI as it was causing problems and is depracated
  • I enabled the fuzz tests to run on linux+clang as it's not supported with gcc and I couldn't get it working out of the box with macos. Libfuzzer is not supported anywhere else.

Sorry for the CI spam :)

@nathaniel-brough
Copy link
Contributor Author

As requested I also added the Cmake support

@jarro2783
Copy link
Owner

This looks good, thanks.

@jarro2783 jarro2783 merged commit 58daccc into jarro2783:master Feb 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants