January 12, 2020

Simulation testing

Speaker

Michael Nygard, consultant at Cognitect and writer of Release it!

Thoughts

The idea of generating tests is very cool. I will try to write something like that for one platform that I have online at the company.

This will be interesting.

Quotes

weaknesses of example-based testing: severely limited coverage, fragility, poor scalability, non-composability

property-based testing: programmer models the domain and invariants. A program generates many individual tests

simulation testing: extend property-based testing to whole systems. Programmer models the domain, Program generates a repeatable script of inputs, system tested in-situ.

Recommendations

  1. Have each property test being about only one concept under testing

  2. What is testing? attempts to prove the system is wrong!

  3. Code coverage is misleading!

  4. making actions repeatable

  5. target system setup

  6. test duration and intensity (and the problem of lag)

  7. simulation testing involves more upfront investment

  8. lifecycle cost is lower

  9. remember: searching the space to find errors

Tags: testing