Sunday, February 14, 2016

A different kind of test metric - test depth

A better metric on test quality

How do you measure the quality of your tests? Test coverage is one of the more widely used metrics. This shows the percentage of code lines that are executed by unit tests.

This metric does not tell anything about the quality of the tests, though. It has a purpose as an alarm bell if coverage is low, but a high coverage does not guarantee that the tests are good.

One problem is; consider a scenario where we only have high-level tests. Let's say that we have an application which is calculating the weight of a 3D printed model. The application has three layers:

Say that we have one single test which is testing the "calculate" button in the User interface layer:

public void CalculateButton_Clicked_ShowsCalculationProgress()
  ... verify some GUI state here

Even if we intend to test only a GUI feature, the rest of the layers may be covered by this test. We are not even verifying the calculation results, yet the Calculation layer may have 100% code coverage. Obviously, test coverage is not a good metric here.

I have some thoughts on a better metric.

Test depth

In the example above, the Calculation layer is tested only through the User interface layer. In other words, the Calculation layer is multiple levels away from the test code in the call stack. We will name this distance between test and tested code as "test depth". A high test depth (for this particular test) means that the test is not actually testing the tested code very well.

This gives us an improved metric; by weighting the test coverage with the test depth, we will get a new depth-weighted coverage. In traditional test coverage, each line is either covered or not covered:

C(line) = 0% or 100%, depending on whether line is executed by a test or not

Whereas the coverage for an entire class is the average

C(class) = Coverage_sum / line_count.

A depth-weighted coverage would then define the coverage per line (if it is covered)

C(line) = 100% / (lowest_depth_of_test - 1), where depth_of_test is the distance in the call stack.

If a line is covered directly by a test, C(line) will be 100%. If it is one level deeper, it will be 50%. Two levels deeper will yield a C(line) of 33%.

Is this the perfect metric?

Is depth-weighted metric a perfect test metric? No, it still doesn't actually verify that you are asserting on the right things. It is not a silver bullet, but it does however prevent the deep-test scenario above.

How do I calculate this?

I haven't found any tools that actually makes this kind of metric. Perhaps I need to make one myself... or are there any volunteers out there? ;-)

Saturday, December 19, 2015

Event testing with the constraint model

In my previous post about the NUnit constraint model, I briefly touched into the new constraint model which is used to write more elegant asserts. As I mentioned, the constraint model is more than just syntactical sugar. I have had a lot of fun with extending the constraint model to allow for easy event testing, and I'd like to share it with you. This is my Christmas gift to you!

Testing events - traditional method

In order to test that an event has been fired from the tested object, one would typically do something like this:

// Arrange
var testee = new Testee();
bool eventWasFired = false;
testee.MyEvent += (s,e) => eventWasFired = true;

// Act

// Assert
Assert.That(eventWasFired, Is.True);

In order to test a PropertyChangedEvent, the syntax becomes a bit more complex and harder to read:

// Arrange
var testee = new Testee();
bool eventWasFired = false;
model.PropertyChanged += (s, e) =>
                              if (e.PropertyName == "SomeProperty")
                                eventWasFired = true;

// Act

// Assert
Assert.That(eventWasFired, Is.True);

This can be tweaked as needed to assert that an event was NOT fired or to assert that the event was fired a given number of times. However, it is not as obvious and readable as one would like. Keep in mind the axe murderer who is going to read your tests!

Having so much more fun with the constraint model

One of the advantages of the NUnit constraint model is the extensibility. My event asserts allows you to test for events like this:

// Arrange
var testee = new Testee();

// Act

// Assert
Assert.That(testee, Did.Fire.TheEvent("SomeEvent"));

I dare to say that this is slightly more readable and less complex than the traditional method! Specifying the event with a string is not typesafe, which is a bit sad. You will not get a compile error if the event does not exist. However, the assert will throw an IllegalArgumentException if the event does not exist. Hence, you will not get false positives or negatives because the event was misspelled.

The difference is even more obvious when testing for a PropertyChangedEvent (compare with the second traditional example):

// Arrange
var testee = new Testee();

// Act

// Assert
Assert.That(testee, Did.Fire.ThePropertyEvent("SomeProperty"));

The other variants can be tested with

Assert.That(testee, Did.Not.Fire.TheEvent("SomeEvent"));
Assert.That(testee, Did.Fire.TheEvent("SomeEvent").Once);
Assert.That(testee, Did.Fire.ThePropertyEvent("SomeProperty").Times(3));

I have uploaded the source code with the required constraints here. Simply add them to your test project or a shared test helper project, and you are good to go. Code samples in the form of tests (of course) are included in the DidTest class. The constraint class is doing a lot of safeguarding to prevent the developer from using a wrong syntax.

Have fun!

Tuesday, November 24, 2015

NUnit assertions - constraint model and classic model

NUnit assertions - constraint model and classic model

If you are using NUnit for your unit tests, there are two models for writing assertions. The "old skool" way of writing assertions is like

Assert.IsTrue(a == 5);
Assert.AreEqual(expectedValue, result);

This way of writing assertions is well known and is used in most unit testing frameworks. Some years ago, NUnit started introducing a new assertion model.

Constraint model

The equivalent assertions in the new constraint model are

Assert.That(a, Is.EqualTo(5));
Assert.That(result, Is.EqualTo(expectedResult);

When the constraint model was introduced, it seemed like nothing more than syntactical sugar. I didn't see any real benefit to it and continued using the classic model.

The constraint model has however been refined and extended over the past years. New and clever ways of writing assertions have been introduced to the constraint model, while the classic model has not evolved.

Rather than writing complex asserts, there is now a number of useful asserts on collections like

Assert.That( iarray, Is.All.GreaterThan(0) );
Assert.That( myArray, Is.SubsetOf( largerArray ) );
Assert.That( sarray, Is.Ordered.Descending );
Assert.That( sarray, Has.No.Member("x") );
Assert.That( iarray, Has.Some.GreaterThan(2) );
Assert.That( iarray, Is.All.GreaterThan(0) );

The meaning of these asserts are pretty self-explaining because of the new syntax.

The constraints typically have additional modifiers. For example, for comparing two doubles, one would often provide a tolerance in the classic model in order to cope with floating-point round-off errors:

Assert.AreEqual(expectedValue, result, 0.000001)

Where the tolerance often was somewhat arbitrary and dependent on the expected value.

With the new constraint model, this can be made much more robust by specifying it like

Assert.That(result, Is.EqualTo(expectedResult).Within(0.01).Percent
Assert.That(result, Is.EqualTo(expectedResult).Within(2).Ulps

The "Ulps" tolerance is quite interesting. "Ulp" means "Units in the Last Place" and is more robust for allowing numerical inaccuracies that are introduced than using a fixed tolerance.

The list of constraints is too long to replicate here. The full set of constraints is listed in the NUnit documentation here:

Custom constraints

If you happen to write your own classes (!), you may need to be able to compare them using a less strict method than the Equals(). For example, you may need to compare mathematical vectors within a tolerance.

Rather than using three asserts for a 3-dimensional vector, you can write your own constraints with a given tolerance.

The classic model is disappearing

Until now, NUnit has been fully backwards compatible. The developer has had the choice to use the classic model if desired. Starting in version 3.0, however, some of the classic assertions have disappeared. For example, the Assert.IsNullOrEmpty() has been removed.

I don't know if the classic assertion model will eventually be completely retired. Who cares anyway, when the constraint model is so much better? ;-)

Wednesday, August 26, 2015

NDepend - an impressing tool for code analysis

If you are a fan of test-driven development AND code analysis, it might be worth having a look on a tool called NDepend: http:///

NDepend analyses your codebase and creates all kind of metrics and charts to assess the readability, maintainability and robustness of your code. You can either read the metrics as-is or create custom Linq-like queries to find the data of interest.

For architects, this tool seems to be very useful for monitoring how the codebase evolves. It might also be useful as a one-shot analysis in e.g. a due diligence process where the overall code quality in a project needs to be assessed.

I am more interested in the test-driven aspects of it though. NDepend can gather code coverage data and use them conjointly with the other NDepend metrics to get a picture on how well complex parts of the code are covered by tests.

Is this useful for a TDD developer? Personally, I am not using code analysis tools very often and find the amount of information a bit overwhelming. For architects or others that like to view their code from a more analytical point of view, however, NDepend is a gold mine of information.

Friday, May 15, 2015

Should we test private methods?

Should we test private methods?

One question which very often pops up when it comes to test-driven development is: "should we test private methods?" There are some (controversial) ways to test private methods in C#. For example, you can declare the test assembly a friend class of the testee assembly. That doesn't smell good though.

So should we test private methods?

This blog post will not try to answer that... because the question is wrong!

It's a sign

If you have read my previous blog post Help, my class is turning into an untestable monster, you may recall that untestable code is a sign of a poor design. It's time to do some refactoring.

Testing private methods is a related topic; if you feel the need to test private methods in order to test the behavior of a class, then it's a sign that you should do some refactoring. Your class is probably trying to do too much. It is violating the Single responsibility principle.


Let's have a look on an example. Consider a class Logger which updates a log in a database when certain events occur. The Logger class has a method LogEvent(event) which generates a message with a timestamp based on the event and adds it to the database. It might look like this:

public class Logger
  public void LogEvent(AppEvent event)
    string message = CreateMessage(event);

  private string CreateMessage(AppEvent event)
    string message = "";

    foreach (var something in event.AffectedObjects()
    { stuff here, add to string etc etc

    ...add nicely formatted timestamp here

    return message;

  private void AddMessageToDatabase(string message)
  { all kinds of database magic here, add to correct table, avoid duplicates, etc

Although a fairly trivial example, the Logger.LogEvent() method is trying to do two things: create a nicely formatted message AND add it to the database. Both of those tasks can have a number of different scenarios and failure conditions, so surely we feel the urge to test both the private CreateMessage() and AddMessageToDatabase() methods!


Having realized that this is a sign, we decide to refactor it:

public class MessageGenerator
  public string CreateMessage(AppEvent event)
    ...create nicely formatted string

    return message;

public class DatabaseLogger
  public void AddMessageToDatabase(string message)
  { all kinds of database magic here, add to correct table, avoid duplicates, etc

Now we have two smaller classes with one single responsibility each. The CreateMessage and AddMessageToDatabase methods are public because they are a part of the public contract and expressed responsibility of these classes.

Did we answer the question about whether or not to test the private methods of the Logger class? No, because the question was wrong!

Wednesday, October 15, 2014

Doing it wrong

Test-driven development is great and all that... if you do it right. My blog posts so far have been focusing on how to do it right. However, it's important to be aware of the pitfalls so that they can be avoided.

Many teams that try to TDD fail at some point. What are the most common mistakes when trying to introduce test-driven development?

Not realizing the paradigm shift

Doing test-driven development is not just about "writing unit tests". It's a whole new way to do software development. Therefore, it's important that the team realizes that it takes some time to get up to speed on TDD.

This is like walking across a ravine from one hilltop to the next. Your team is probably well-established and has reached a productivity peak. In order to reach the next peak, you need to realize that your productivity will suffer for a limited period while you adapt to a new development paradigm:

Crossing the ravine
If the team, their manager or other stakeholders don't realize this, you/they may get impatient and decide to abandon the whole TDD idea. What a misery!

No buy-in from management

This goes hand in hand with the previous section. Switching to TDD is an investment. Like any other investment, it comes with a cost. The management needs to realize that there will be an initial cost in terms of training and temporarily reduced productivity while crossing the ravine.

The management also needs to realize that this is an investment that pays off. After the initial cost, the benefit is obvious: software with fewer bugs means higher quality, a shorter beta testing phase and happier customers. It's not always easy to quantify this benefit into a language that managers and shareholders understand (the $/£/€/kr language), though.

Developers decide up front that TDD is a bad idea

This is perhaps one of the toughest obstacles. If the developer team is reluctant to do TDD in the first place, it's hard to enforce it. As a team lead, one of the most important tasks will be to motivate the team to do TDD. A TDD introductory course is highly recommended, as it's hard to learn TDD on your own without any mentoring.

It's important that a critical mass within the team does proper TDD. If several developers completely ignore the fact that there are tests that need to be maintained, they will quickly ruin the entire TDD process for the others.

No focus on maintainability

I mentioned maintainability in my blog post "Pillars of unit tests". You should treat your test code as well as your production code. Your test code is not a second class citizen. Test code should be reviewed and refactored as often and as carefully as the production code.

If the test code turns into spaghetti, maintainability will suffer. As you add or change functionality in the production code, it will become harder and harder to do the required changes to the test code. The team may give up testing new use cases, tests will fail for the wrong reason and the team gives up doing TDD.

Too hard to run tests or tests are not trusted

It should be easy to run tests. Ideally, the team should use a test runner like NCrunch so that the developers don't need to run the tests manually.

If it's hard to run tests, or the tests run very slowly, developers tend to skip it. They may check in broken code into the source repository because they don't realize that the tests are failing. All hope is lost.

Also, it doesn't help to run the tests if you don't trust them. If the team does not have a good habit of rejecting code that causes tests to fail, the team will stop trusting the tests. If the tests are not trustworthy, much of the benefit with TDD is lost.

Thursday, August 7, 2014

Productivity tips: live templates in Visual Studio and ReSharper

In my blog post Two readability tips for naming and organizing tests, I shared some tips for arranging tests. The tests are typically named and organized like this:

public void ItemUnderTest_Scenario_ExpectedBehaviour()
  // Arrange
  Arrange the test here...

  // Act
  Do the action which is being tested here...

  // Assert
  Do assertions here...

While the naming and arrangement scheme is simple enough, it quickly becomes boring to write this stub every time you write a new test. If you are fortunate enough to use Visual Studio with the ReSharper plug-in, however, there is a nice feature called "live templates" which can do it for you!

Live templates allow you to type a keyword, select one of the live template entries from the popup menu which automatically pops up, and ReSharper will automatically insert the template for you. In this case, the template for a test is named "test":

After the template is pasted into the test class, there is no need to navigate the cursor around in order to edit the details in the test name. Red rectangles represent the fields that you will typically want to edit (ItemUnderTest, Scenario and ExpectedBehaviour). Enter the contents of those fields, press enter after editing each field, and then you're ready to implement the test!

Create and edit the live templates in ReSharper->Templates Explorer. As you can see, there are already many useful built-in templates:

Notice how you define the fields that the user typically edit after inserting the template: $ItemUnderTest$, $Scenario$ and $ExpectedBehaviour$. After the user has entered the contents of those fields, the cursor is placed at the optional $END$ keyword.