Combinatorial Tests and Splitting Context From Specification

Chris Bilson — Tue, 15 Jul 2008 12:05:19 GMT

Yesterday on twitter I had the start of an interesting conversation with Steve Harmann and Scott Bellware, but then I had to do work, so I had to stop just when it was getting interesting.

One of the things I have found useful in writing unit tests for an application I work on is mbunit’s CombinatorialTest. If you are familiar with RowTests in mbunit, CombinatorialTests are similar, but allow you to use a factory method to generate the parameters for the test.

By way of example, below is a dumbed down version of a real test I have. Note that this was developed before I really knew anything about BDD.

The system under test needs to find the next best move as a number. There are two candidate moves. If the candidates are on both sides of the number we’re on now (mixed signals), we don’t move. If they are both on the same side, we move that direction, but to the number with the smallest change from our current number. I received these specs from our product owner in the form of an Excel spreadsheet with a formula, mixed with a crap-load of test data.

I have found that it’s useful to define a class for the parameter to the test, and that it’s really useful to have that class have lambdas for the actual test, and that it’s really, really useful to use Expression<> for the lambdas, so you can print the result. I am going to call this class a “Specification Class” for lack of a better term. Here’s the one for the stories described above:

public class PositionSpec

    public decimal Candidate1
{ get; set; }

    public decimal Candidate2
{ get; set; }

    public decimal Current
{ get; set; }

    public Expression<Predicate<ModelPosition,decimal>>
Test { get; set; }

    public override string ToString()

        return String.Format("c1={0:p}
^ c2={1:p} ^ current={2:p} -> {3}",

            Candidate1, Candidate2, Current, Test.Body.ToString());

Notice that the Test property is an Expression. If you are not familiar, this is like a lambda - Expression<Predicate<ModelPosition, decimal>> can do everything Predicate<ModelPosition, decimal> can do – but you can examine the body of the expression. I think this is really cool and a gateway to functional+dynamic programming. In this case though, I just want to use that so I can print the expression in ToString – mbunit will call this when printing the name of the test in the results, so we can see which expression we are testing.

Here’s the factory function that produces instances for tests (specifications):

public IEnumerable<PositionSpec>
TestPositions()

    yield return new PositionSpec()
{ Candidate1 = .15M, Candidate2 = .20M, Current = .17M,

        Test = (pos, w) => w == .17M };

    yield return new PositionSpec()
{ Candidate1 = .15M, Candidate2 = .20M, Current = .10M,

        Test = (pos, w) => w == .15M };

    yield return new PositionSpec()
{ Candidate1 = .15M, Candidate2 = .10M, Current = .20M,

        Test = (pos, w) => w == .15M };

    yield return new PositionSpec()
{ Candidate1 = .15M, Candidate2 = .13M, Current = .10M,

        Test = (pos, w) => w == .13M };

    yield return new PositionSpec()
{ Candidate1 = .15M, Candidate2 = .17M, Current = .20M,

        Test = (pos, w) => w == .17M };

    yield return new PositionSpec()
{ Candidate1 = .15M, Candidate2 = .20M, Current = .17M,

        Test = (pos, w) => pos.IsNoActionRequired == true };

    yield return new PositionSpec()
{ Candidate1 = .15M, Candidate2 = 0M, Current = .14M,

        Test = (pos, w) => pos.IsNoActionRequired == true };

    yield return new PositionSpec()
{ Candidate1 = .15M, Candidate2 = 0M, Current = .14M,

        Test = (pos, w) => w == .14M};

    yield return new PositionSpec()
{ Candidate1 = .15M, Candidate2 = 0M, Current = .20M,

        Test = (pos, w) => w == .15M};

    yield return new PositionSpec()
{ Candidate1 = 0M, Candidate2 = .15M, Current = .14M,

        Test = (pos, w) => pos.IsNoActionRequired == true };

    yield return new PositionSpec()
{ Candidate1 = 0M, Candidate2 = .15M, Current = .14M,

        Test = (pos, w) => w == .14M};

    yield return new PositionSpec()
{ Candidate1 = 0M, Candidate2 = .15M, Current = .20M,

        Test = (pos, w) => w == .15M};

    yield return new PositionSpec()
{ Candidate1 = 0M, Candidate2 = 0M, Current = .20M,

        Test = (pos, w) => w == 0M};

And finally the test that consumes them:

[CombinatorialTest]

public void should_result_in_smallest_delta_from_current_with_netting(

    [UsingFactories("TestPositions")]
PositionSpec spec)

    Mocks.ReplayAll();

    ModelPosition candidate1 = new ModelPosition(){
Weight = spec.Candidate1 };

    ModelPosition candidate2 = new ModelPosition()

        Weight = spec.Candidate2,

        HeldPositionWeight = spec.Current

};

    decimal result
= _methodology.GetNetPositionWeight(candidate2, candidate1);

    Assert.IsTrue(spec.Test.Compile().Invoke(candidate2, result));

The result has the expression bodies printed out, so looks like:

------ Test started: Assembly: XXX.Tests.dll ------
 
Starting the MbUnit Test Execution
Exploring XXX.Tests, Version=1.0.0.2, Culture=neutral, PublicKeyToken=null
MbUnit 2.41.232.0 Addin
Found 13 tests
[assembly-setup] success
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=20.00 % ^ current=17.00 % -> (w = 0.17)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=20.00 % ^ current=10.00 % -> (w = 0.15)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=10.00 % ^ current=20.00 % -> (w = 0.15)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=13.00 % ^ current=10.00 % -> (w = 0.13)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=17.00 % ^ current=20.00 % -> (w = 0.17)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=20.00 % ^ current=17.00 % -> (pos.IsNoActionRequired = True)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=0.00 % ^ current=14.00 % -> (pos.IsNoActionRequired = True)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=0.00 % ^ current=14.00 % -> (w = 0.14)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=0.00 % ^ current=20.00 % -> (w = 0.15)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=0.00 % ^ c2=15.00 % ^ current=14.00 % -> (pos.IsNoActionRequired = True)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=0.00 % ^ c2=15.00 % ^ current=14.00 % -> (w = 0.14)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=0.00 % ^ c2=15.00 % ^ current=20.00 % -> (w = 0.15)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=0.00 % ^ c2=0.00 % ^ current=20.00 % -> (w = 0)))
[reports] generating HTML report
TestResults: file:///C:/Users/cbilson/AppData/Roaming/MbUnit/Reports/XXX.Tests.Tests.html
>
 
13 passed, 0 failed, 0 skipped, took 3.30 seconds.
 

So that’s one example. I had a lot of little tiny contexts. I wanted to apply the same basic proto-spec to each one. I could have done this with a base context class and a bunch of subclasses, which seems to be the BDD way to do this, but using combinatorial tests seems more concise and readable to me.

There’s one other really powerful thing you can do with combinatorial tests. Since you can use these to separate context from spec, you could apply a set of specs to a set of contexts. Your actual tests would be the cross product of the two, without having to specify each product.

For example, say you had a really complicated data entry screen. Your user stories indicated that everyone needed to be able to edit data in some subset of the fields on the screen, only admins another subset, and only under certain conditions everyone could edit all fields.

There are lots of ways to tackle this, but one way would be to define the two different subsets using two factories. Then your specifications could be really simple and readable, like “everyone_should_be_able_to_edit_the_everyone_fields”, “only_admins_can_edit_the_admin_fields”, and “everyone_can_edit_the_admin_fields_in_some_special_case".”

For admin vs. normal user fields, it would probably be better to just tag the fields somehow, but what if there were way more conditionals, like “for gold card members with the super-duper price-protection plan, normal customer service reps should be able to edit the discount field.” I have a couple of cases like this where the user interface changes a lot based on a couple of different special conditions and I have found combinatorial tests really good for this.

Now that I am getting into BDD, I think I could do a better job of separating the context from the specification. It seems like the “specification” class I had above should really be the “context” and I should remove the Test property, since that’s the core of the specification. I could then use this technique when I have a specification that applies to a large number of contexts.

Chris's Weblog

Combinatorial Tests and Splitting Context From Specification