Sign in
Log inSign up
How many unit tests are good enough | Property-Based testing

How many unit tests are good enough | Property-Based testing

Miguel Bernard's photo
Miguel Bernard
·Apr 13, 2020

As I was mentioning in my previous post, the way we are doing unit testing today has many issues. One of the biggest problems is to know for sure when we have enough tests to be confident to release in production. Why is that? Well, we mostly base our tests on examples, so it's tough to know for sure if we covered all the edge cases and that we have all the required samples.

Tale of a developer

Let's use a story to demonstrate what I mean.

Notes:

Here, the gray circle will be defining what the client wants, the Xs will visually represent the tests, and the red lines will represent the actual output of the algorithm.

Once upon a time

Client: Could you create an algorithm that draws a circle given a radius? Something like that.

CircleReq

Developer: Of course! Easy peasy.

After some time

Developer: Alright I'm done, it's working. Client: How do you know if it's working properly? Did you test it? Developer: Well... I made it run on my machine. If you want, I can add a unit test to prove it. Client: That would be great. Developer: Here you go. I added a unit test that makes sure that the point (1,0) is part of the circle.

Circle_1_0

Client: How do you know that you are not only drawing a horizontal line that passes through that point? Developer: Ok, here's another test that makes sure that the point (0,1) is part of the circle.

Circle_0_1

Client: What about coordinates that are not int? Developer: Sure. I added another test for this.

Circle_05_05

Client: I don't want just a quarter of a circle. Can you make sure other quadrants are also supported? Developer: Arrrrg... Sure. I added another test for this. I'm pretty sure we covered everything. We should ship it.

Circle_-078_023

Client: Alright, after all, you are the expert. Let's ship it.

Once in production

Circle-in-prod

Client: What the heck. It looks nothing like the circle I wanted. Developer: I don't understand. All my tests are passing, which means this should work just fine according to your requirements.

Circle_prod_tests

Developer: Maybe if I add another test, I'll find the issue.

The truth is

In fact, with this kind of problem, you'll never be done adding examples. It's because a circle is composed of an infinity of points, which makes it almost impossible to validate with example-based tests.

What do we do then

At this point, I'm pretty sure we all agree that we need a better strategy to test this. That's precisely the kind of problem where Property-Based testing shines. Instead of validating points along the circle, let's try to understand and model the relationship between the input (the radius) and the output (the circle). It turns out that this relationship is well known and defined in the math world.

A circle can be defined as the locus of all points that satisfy the equation x^2 + y^2 = r^2

It's an excellent property for a test. With only one test, we can validate any point of a circle given a radius. Voilà!

Conclusion

Stay tuned for my next blog post, where we'll solve a real problem with C# and property-based testing.