FuzzDotNet
FuzzDotNet is a tool for .NET which enables fuzzing to be used alongside unit tests. It will randomly generate unit test cases in order to greatly increase the coverage of a test suite. The tool promotes testing by contract, by using a formal methods-like approach to unit testing. Because it uses randomly generated data, it finds counterexamples without the need to write proofs of programs. The library is available on NuGet.org for MSTest and xunit. You can read more about it on the project page.
The distribution of values chosen may be customized, but by default the library will choose “hard” values, like null, -1, strings containing emoji, etc. The library provides reflection-based generators for several types of classes, as well as facilities to format and notify developers about counterexamples which were discovered. It is fully user-customizable.
Here’s an example which tests that secrets are never returned from an API call.
[FuzzTestMethod]
public void TestSecretObfuscated(ModelId modelId, Model model)
{
// Test that reads always obfuscate secrets.
api.Create(modelId, model);
var read = api.Read(modelId);
Assert.SecretsObfuscated(read);
}
FuzzDotNet will execute this test case many times with randomly generated, worst-case data. This allows it to execute more test cases than a human could write. It is likely that cases which the human would not have come up with are tested as well.
In formal logic, this test could be written like so:
\[\forall modelId, db: \text{SecretsObfuscated}(\text{Read}(db, modelId))\]This formulation, while precise, is difficult to parse for the average programmer. FuzzDotNet allows programmers to write tests in plain old C#, but get the benefits of this logical formulation and increased test coverage.
While developing this library, I have used fuzz tests to verify it’s correctness and the tests have immediately found edge-case bugs which I initially overlooked.