In this multipart blog post I want to introduce you in the world of property-based testing and how to do this in C#.
In the first part, I gave you an introduction on what property-based testing is, why it useful and how it can help you write better code. In the second post, I showed you a concrete example on how to start writing property-based tests in C# using FsCheck.
Today I show you how I use property-based tests to find edge cases and can help to understand a codebase. Along the way I’ll share some of the other features that FsCheck has to offer.
And to give you a more realistic example, I will use an open source library created by a colleague(thanks Willy for allowing me to use your library as an example); https://github.com/WilvanBil/NationalRegisterNumber.
National Register Number is a package that can generate and validate Belgian national register numbers. The logic is based on Official Documentation by the Belgian Government
The library is small and offers 2 API’s:
- A
NationalRegisterNumberGenerator.IsValid()method that allows you to check if the provided string is a national register number. - A
Generate()that will return a random valid national register number.
When I opened the codebase, I immediately noticed the tests:
You see that these are typical examples(no pun intended) of example based tests. Some magical values where used as input to validate the test, but are we sure that we covered all edge cases? Let’s find out…
Willy used NUnit in his codebase. Luckily FsCheck has support for NUnit, so before we write our first property-based test let’s add the FsCheck.Nunit NuGet package:
dotnet add package FsCheck.NUnit
Let us first focus on the IsValid() method. When using FsCheck with NUnit, we don’t use the [Test] attribute but the [FsCheck.NUnit.Property] attribute:
This tests succeeds which is a good started but not unexpected. It would be quite a coincidence if we provide a pseudo-random string that is also a valid national registration number. Here as well you can use the Verbose property on the attribute to see the values used.
You can write the same test also like this:
As we now that the national register number is in fact a number, let’s give it another try with a numeric value:
Still succeeds! Let’s try some other values by specifying an Arbitrary but therefore I need to first explain what it is.
What are Arbitraries?
FsCheck uses a combination of generators and shrinkers to produce test data. Generators make a random choice of a value from an interval, with a uniform distribution.
Shrinkers come into the picture when FsCheck finds a set of values that falsify a given property. In that case it will try to make that value smaller than the original (random) value by getting the shrinks for the value and trying each one in turn to check that the property is still false. If it is, the smaller value becomes the new counter example and the shrinking process continues with that value.
Arbitraries brings a generator and shrinker together to be used in properties. FsCheck defines default arbitraries for some often used types. Some examples:
- NegativeInt
- NonNegativeInt
- NonEmptyString
- IntWithMinMax
- NonEmptyArray
- …
Check the code on Github for the full list.
To use a specific arbitrary we can pass it to the Prop.ForAll method:
In the example above we used an Arbitrary that includes the Int.MinValue and Int.MaxValue in our test data.
So far, so good. Nothing unexpected with the IsValid() method. Let’s now switch our focus to the Generate() method.
The Generate() method has multiple overloads:
Generate()
Generate(DateTime birthDate)
Generate(BiologicalSex sex)
Generate(DateTime birthDate, BiologicalSex sex)
Generate(DateTime minDate, DateTime maxDate)
Generate(DateTime minDate, DateTime maxDate, BiologicalSex sex)
Generate(DateTime birthDate, int followNumber)Let us first focus on the one that allows us to specify a DateTime.
Our first test fails because the Generate method expects DateTime values before today.
We'll fix our property-based test to filter out these DateTime values in the future:
However this time when we run our tests, it still fails:
If we have a look at the full test output, we see the concept of shrinking in action:
You can see that we start with a full datetime including date and time and the shrinker will set first the seconds, minutes and then hours to 0.
Mmmh. This seems to be a bug. I can use the NationalRegisterNumberGenerator to generate an invalid number. Just to be sure I created an example based test that reproduces the problem I found:
Time to contact Willy to investigate the root cause of the bug!
This post is already quite long, so I’ll show you how to build your own generators in a next post.