from Hacker News

Fuzzing vs. Property Testing

by chwolfe on 6/25/19, 10:03 PM with 7 comments

• by kcsrk on 6/26/19, 4:28 AM

  Of interest is Crowbar[0,1], a testing tool for OCaml that combines Quickcheck like property testing with AFL fuzz.

  [0] https://github.com/stedolan/crowbar

  [1] https://ocaml.org/meetings/ocaml/2017/extended-abstract__201...

• by stevekemp on 6/26/19, 5:18 AM

  One of the things that I like about go is that the standard-library really encourages the use of writing test-cases. Most of the bigger projects have good coverage, and I try hard to get my own.

  But nothing beats the sheer tenacity of running your code through a fuzz-tester. I've written a simple virtual machine, which interprets bytecode, and a BASIC interpreter amongst other things recently. Both of these projects benefited hugely from fuzz-testing, despite having high coverage via manually-written test-cases.

  Fuzz testing is cheap and largely automatic, so if it takes a few hours or a few days to find an interesting result that's not a problem. I remember the first time I tested my interpreter when I had ~90% coverage of the code with my test-cases and it crashed via fuzzing within seconds! Magic!

• by ehsankia on 6/26/19, 3:13 AM

  Related, I've been having a lot of success with Hypothesis [0], the Python property-based testing library.

  [0] https://hypothesis.readthedocs.io/en/latest/

• by dnautics on 6/26/19, 6:29 AM

  A good property testing framework has two properties, 1) inputs increase in complexity over generations, and 2) properties can shrink, enabling you to find "minimum error conditions", so property tests can help you identify the source of the error too.

• by carlmr on 6/26/19, 6:31 AM

  Is there something like a good proptest in C++?