from Hacker News

Effective Property-Based Testing

by mullr on 2/3/21, 6:39 PM with 34 comments

• by spockz on 2/3/21, 9:09 PM

  I would like to make a shout out to quickcheck (Haskell, https://hackage.haskell.org/package/QuickCheck) as it was my first introduction to property based testing and has since let me to use property based testing everywhere in the last 10 years. Then there is scalacheck (https://www.scalacheck.org/). Both let you write your own generators and are quite good at shrinking down to minimal use cases.

  The suggestion elsewhere in this thread to decrease the number of iterations during normal testing and crank it up during nightlies is also good.

  The only thing I’m still missing from the libraries is a convenient mechanism of remembering previously failing generated inputs and use them as a static list of testcases next to the at runtime generated ones like a regression test of sorts.

  Edit: typos

• by mristin on 2/3/21, 8:00 PM

  If you use Python and want to infer test strategies from contracts, you might want to check out this library of mine: [1].

  There are also plugins for IDEs (Pycharm, VS Code and vim), which can be quite helpful during the development.

  [1]: http://github.com/mristin/icontract-hypothesis

• by rck on 2/3/21, 7:46 PM

  This is a great set of ideas for using property-based testing. I've found it useful to think of code in terms of invariants and contracts, and property-based testing lets me express those very directly in code. No other testing method comes close.

• by Smaug123 on 2/3/21, 8:49 PM

  Brilliant article.

  If I were to add just one thing to the list: metatest. Write a test that asserts that your generated test cases are "sufficiently comprehensive", for whatever value of "sufficiently" you need. In an impure language, this is as easy as having the generator contain a mutable counter for "number of test cases meeting X condition" for whatever conditions you're interested in. For example, say your property is "A iff B". You might want to fail the test if fewer than 10% of the generated cases actually had A or B hold. (And then, of course, make sure your generators are such that - say - A and B hold 50% of the time; you want an astronomically small chance of random metatest failure.)

  (I did a brief intro to this in the Metatesting section of a talk I did two years ago: https://github.com/Smaug123/talks/blob/master/DogeConf2019/D... . On rereading it now, I see there's a typo on the "bounded even integers" slide, where the final `someInts` should read `evenIntegers`.)

• by mathw on 2/4/21, 8:04 AM

  I've been trying to push property-based testing at work but there's not a load of enthusiasm showing. I guess it's in part because we work in C# and the best option appears to be FsCheck, which is very inconsistently documented between its F#-native and C#-accessible APIs.

  I think there's a strong argument with FsCheck to write all your proptest code in F# just to take advantage of the vastly better generator syntax, but that's a hard sell for a team who mostly don't know F# and aren't convinced proptests are much better anyway. Writing the generators in C# seemed really incredibly tedious. I did start to get the hang of identifying properties to test though. Once you're past the mechanics of "how does this work" that can become much easier.

  A load road to travel here, but I kind of gave myself a remit to improve software quality and I do think we need to be looking at this kind of testing to help.

  Where do people who are using it find that it offers the most value? I keep feeling that we could really solidify some of our bespoke parsing and serialisation code using this kind of tech.

• by hinkley on 2/4/21, 12:41 AM

  Hypothesis:

  Property Based Testing is Monte Carlo simulation for model checking.

• by lowercase1 on 2/3/21, 10:21 PM

  This is a good overview of property based testing but the mentions of Cucumber threw me off.

  In my job Cucumber seems to add little more than just commenting and sequencing functions, tasks that are better suited to your programming language of choice, while adding overhead and complexity.

  What am I missing?

• by the_gastropod on 2/3/21, 7:46 PM

  Property testing is awesome, but it does significantly slow down test suites. Are there any standard practices for striking a balance of getting the added confidence/safety of using property tests without introducing large delays into your CI pipeline?

• by spion on 2/3/21, 10:12 PM

  For JS and TypeScript, the best property testing library I've encountered so far is fast-check https://github.com/dubzzz/fast-check

• by delightful on 2/3/21, 8:34 PM

  Does anyone have any thoughts on the graphic showing the spectrum of testing options listed in the article (or additional resources that cover at a high-level this topic of understanding range of testing options)?

  link to the graphic for ease of reference:

  https://blog.auxon.io/images/posts/effective-property-based-...

• by mavelikara on 2/3/21, 11:30 PM

  When you first start on PBT, the first hurdle is finding good properties of your system to test. I found this article [1] to be a great overview to get started.

  [1]: https://fsharpforfunandprofit.com/posts/property-based-testi...

• by leeuw01 on 2/3/21, 7:42 PM

  Auxon also looks like an interesting company.

• by pfdietz on 2/3/21, 11:48 PM

  Turning off shrinking makes little sense to me. If tests are passing there's nothing to shrink. If you are bombarded with so many failures you can't afford to shrink them then more testing is pointless.

• by mekatronix on 2/3/21, 7:33 PM

  These are great tips. Thanks.