Yesterday on twitter I had the start of an interesting conversation with Steve Harmann and Scott Bellware, but then I had to do work, so I had to stop just when it was getting interesting.

One of the things I have found useful in writing unit tests for an application I work on is mbunit’s CombinatorialTest. If you are familiar with RowTests in mbunit, CombinatorialTests are similar, but allow you to use a factory method to generate the parameters for the test.

By way of example, below is a dumbed down version of a real test I have. Note that this was developed before I really knew anything about BDD.

The system under test needs to find the next best move as a number. There are two candidate moves. If the candidates are on both sides of the number we’re on now (mixed signals), we don’t move. If they are both on the same side, we move that direction, but to the number with the smallest change from our current number. I received these specs from our product owner in the form of an Excel spreadsheet with a formula, mixed with a crap-load of test data.

I have found that it’s useful to define a class for the parameter to the test, and that it’s really useful to have that class have lambdas for the actual test, and that it’s really, really useful to use Expression<> for the lambdas, so you can print the result. I am going to call this class a “Specification Class” for lack of a better term. Here’s the one for the stories described above:

public class PositionSpec
{
    public decimal Candidate1 { get; set; }
    public decimal Candidate2 { get; set; }
    public decimal Current { get; set; }
    public Expression<Predicate<ModelPosition,decimal>> Test { get; set; }
    public override string ToString()
    {
        return String.Format("c1={0:p} ^ c2={1:p} ^ current={2:p} -> {3}", 
            Candidate1, Candidate2, Current, Test.Body.ToString());
    }
}

Notice that the Test property is an Expression. If you are not familiar, this is like a lambda - Expression<Predicate<ModelPosition, decimal>> can do everything Predicate<ModelPosition, decimal> can do – but you can examine the body of the expression. I think this is really cool and a gateway to functional+dynamic programming. In this case though, I just want to use that so I can print the expression in ToString – mbunit will call this when printing the name of the test in the results, so we can see which expression we are testing.

Here’s the factory function that produces instances for tests (specifications):

public IEnumerable<PositionSpec> TestPositions()
{
    yield return new PositionSpec() { Candidate1 = .15M, Candidate2 = .20M, Current = .17M, 
        Test = (pos, w) => w == .17M };
    yield return new PositionSpec() { Candidate1 = .15M, Candidate2 = .20M, Current = .10M, 
        Test = (pos, w) => w == .15M };
    yield return new PositionSpec() { Candidate1 = .15M, Candidate2 = .10M, Current = .20M, 
        Test = (pos, w) => w == .15M };
    yield return new PositionSpec() { Candidate1 = .15M, Candidate2 = .13M, Current = .10M, 
        Test = (pos, w) => w == .13M };
    yield return new PositionSpec() { Candidate1 = .15M, Candidate2 = .17M, Current = .20M, 
        Test = (pos, w) => w == .17M };
    yield return new PositionSpec() { Candidate1 = .15M, Candidate2 = .20M, Current = .17M,
        Test = (pos, w) => pos.IsNoActionRequired == true };
    yield return new PositionSpec() { Candidate1 = .15M, Candidate2 = 0M, Current = .14M,
        Test = (pos, w) => pos.IsNoActionRequired == true };
    yield return new PositionSpec() { Candidate1 = .15M, Candidate2 = 0M, Current = .14M,
        Test = (pos, w) => w == .14M};
    yield return new PositionSpec() { Candidate1 = .15M, Candidate2 = 0M, Current = .20M,
        Test = (pos, w) => w == .15M};
    yield return new PositionSpec() { Candidate1 = 0M, Candidate2 = .15M, Current = .14M,
        Test = (pos, w) => pos.IsNoActionRequired == true };
    yield return new PositionSpec() { Candidate1 = 0M, Candidate2 = .15M, Current = .14M,
        Test = (pos, w) => w == .14M};
    yield return new PositionSpec() { Candidate1 = 0M, Candidate2 = .15M, Current = .20M,
        Test = (pos, w) => w == .15M};
    yield return new PositionSpec() { Candidate1 = 0M, Candidate2 = 0M, Current = .20M,
        Test = (pos, w) => w == 0M};
}

And finally the test that consumes them:

[CombinatorialTest]
public void should_result_in_smallest_delta_from_current_with_netting(
    [UsingFactories("TestPositions")] PositionSpec spec)
{
    Mocks.ReplayAll();
 
    ModelPosition candidate1 = new ModelPosition(){ Weight = spec.Candidate1 };
    ModelPosition candidate2 = new ModelPosition()
    {
        Weight = spec.Candidate2,
        HeldPositionWeight = spec.Current
    };
 
    decimal result = _methodology.GetNetPositionWeight(candidate2, candidate1);
    Assert.IsTrue(spec.Test.Compile().Invoke(candidate2, result));
}

The result has the expression bodies printed out, so looks like:

------ Test started: Assembly: XXX.Tests.dll ------
 
Starting the MbUnit Test Execution
Exploring XXX.Tests, Version=1.0.0.2, Culture=neutral, PublicKeyToken=null
MbUnit 2.41.232.0 Addin
Found 13 tests
[assembly-setup] success
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=20.00 % ^ current=17.00 % -> (w = 0.17)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=20.00 % ^ current=10.00 % -> (w = 0.15)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=10.00 % ^ current=20.00 % -> (w = 0.15)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=13.00 % ^ current=10.00 % -> (w = 0.13)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=17.00 % ^ current=20.00 % -> (w = 0.17)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=20.00 % ^ current=17.00 % -> (pos.IsNoActionRequired = True)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=0.00 % ^ current=14.00 % -> (pos.IsNoActionRequired = True)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=0.00 % ^ current=14.00 % -> (w = 0.14)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=15.00 % ^ c2=0.00 % ^ current=20.00 % -> (w = 0.15)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=0.00 % ^ c2=15.00 % ^ current=14.00 % -> (pos.IsNoActionRequired = True)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=0.00 % ^ c2=15.00 % ^ current=14.00 % -> (w = 0.14)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=0.00 % ^ c2=15.00 % ^ current=20.00 % -> (w = 0.15)))
[success] NettingMethodologyTests.BaseSetup.should_result_in_smallest_delta_from_current_with_netting(TestPositions(c1=0.00 % ^ c2=0.00 % ^ current=20.00 % -> (w = 0)))
[reports] generating HTML report
TestResults: file:///C:/Users/cbilson/AppData/Roaming/MbUnit/Reports/XXX.Tests.Tests.html
 
13 passed, 0 failed, 0 skipped, took 3.30 seconds.
 

So that’s one example. I had a lot of little tiny contexts. I wanted to apply the same basic proto-spec to each one. I could have done this with a base context class and a bunch of subclasses, which seems to be the BDD way to do this, but using combinatorial tests seems more concise and readable to me.

There’s one other really powerful thing you can do with combinatorial tests. Since you can use these to separate context from spec, you could apply a set of specs to a set of contexts. Your actual tests would be the cross product of the two, without having to specify each product.

For example, say you had a really complicated data entry screen. Your user stories indicated that everyone needed to be able to edit data in some subset of the fields on the screen, only admins another subset, and only under certain conditions everyone could edit all fields.

There are lots of ways to tackle this, but one way would be to define the two different subsets using two factories. Then your specifications could be really simple and readable, like “everyone_should_be_able_to_edit_the_everyone_fields”, “only_admins_can_edit_the_admin_fields”, and “everyone_can_edit_the_admin_fields_in_some_special_case".”

For admin vs. normal user fields, it would probably be better to just tag the fields somehow, but what if there were way more conditionals, like “for gold card members with the super-duper price-protection plan, normal customer service reps should be able to edit the discount field.” I have a couple of cases like this where the user interface changes a lot based on a couple of different special conditions and I have found combinatorial tests really good for this.

Now that I am getting into BDD, I think I could do a better job of separating the context from the specification. It seems like the “specification” class I had above should really be the “context” and I should remove the Test property, since that’s the core of the specification. I could then use this technique when I have a specification that applies to a large number of contexts.


 
Categories: bdd | tdd

So let’s say I have a user story like:

As a Portfolio Analyst

I need to be able to change an account’s market value, overriding what was reported by the custodian

So I can reflect cash flows I know are going to hit the account.

So we talk with them and know that we are going to make the Market Value field on some screen editable. When they don’t do anything, we will just use the market value reported by the custodian (which is what we already do now) but when they enter a manual plug, we will use that value instead.

So the developers talk about it an decide we first want to introduce properties ReportedMarketValue and EstimatedMarketValue. Here is the original Account class

public class Account
{
  public virtual string AccountNumber { get; set; }
  public virtual decimal MarketValue { get; set; }
}

Which we change to:

public class Account
{
  public virtual string AccountNumber { get; set; }
  public virtual decimal MarketValue { get; set; }
  public virtual decimal ReportedMarketValue { get; set; }
  public virtual decimal? EstimatedMarketValue { get; set; }
}

Next we want to make MarketValue read-only, have it either be EstimatedMarketValue if there is one, or ReportedMarketValue. All the places that don’t compile anymore (where someone said “account.MarketValue = “) will be changed to use the ReportedMarketValue mutator.

…but before that, let’s get our specs in place.

MSpec makes heavy use of lambdas, fluent interface, and a custom nunit test runner to take away a lot of the curly braces. Again, read Aaron’s blog for the introduction, but basically, we are going to declare a base class class to establish all the actors in a series of specs around this story:

public abstract class account_spec
{
  protected static Account Account { get; private set; }
 
  Establish context = () => {
    Account = new Account();
    Account.ReportedMarketValue = 100M;
  };
}

Establish is just a delegate and we’re setting it to an anonymous method to just create a new Account. Notice that the Account property is static? Yeah, that sucks. You can’t reference non static members from inside an initializer for a delegate:

image

I think the driving design choice is to limit the amount of alien noise in the spec. Here the tradeoff is adding “static” vs. doing the initialization inside of a constructor or something. Tough call. Whatever, this works, so let’ just go with it.

Next we want to specify what happens when there is no plugged market value:

[Concerning(typeof(Account), "Market Value")]
public class with_no_plugged_market_value : account_spec
{
  Because the_account_has_no_plugged_market_value = () => Account.EstimatedMarketValue = null;
  It should_give_the_reported_market_value_as_the_market_value = () => 
    Account.MarketValue.ShouldEqual(Account.ReportedMarketValue);
}

There is more decoration here. Let’s go over that:

  • The Concerning Attribute: Is used to format the output of the test run and to introduce a “test fixture”.
  • Because: is just another anonymous delegate. I think in one of my experiments I tried having two because’s (I was thinking it be like “because of this and because of that”) but only one got executed. This makes sense. Just put all your because stuff in the same block.
  • It: is…yep, just another anonymous delegate. You can have multiple It’s.
  • ShouldEqual: is a nice extension method that comes with mspec. Thanks!

When run, this spec fails:

------ Test started: Assembly: Specs.dll ------ 
 
Account Market Value, with no plugged market value
!!» should give the reported market value as the market value !! 
 
TestCase 'should give the reported market value as the market value' failed: 
    NUnit.Framework.AssertionException:   Expected: 100m
      But was:  0m
    at NUnit.Framework.Assert.That(Object actual, Constraint constraint, String message, Object[] args)
    at NUnit.Framework.Assert.AreEqual(Object expected, Object actual)
    C:\projects\machine\Source\Specifications\Machine.Specifications\ExtensionMethods.cs(25,0): at Machine.Specifications.ShouldExtensionMethods.ShouldEqual(Object actual, Object expected)
    C:\Users\cbilson\Documents\Visual Studio 2008\Projects\ClassLibrary2\Specs\Class1.cs(24,0): at Specs.with_no_plugged_market_value.<.ctor>b__1() 
 
0 passed, 1 failed, 0 skipped, took 0.54 seconds.

So we implement it (Spec First Development!):

public virtual decimal MarketValue { 
  get { return ReportedMarketValue; }
  set { throw new NotImplementedException(); }
}

(don’t worry, we’ll get rid of set later in another refactoring – or forget about it and find a zillion other tests broke.)

Now we pass:

------ Test started: Assembly: Specs.dll ------
 
Account Market Value, with no plugged market value
  » should give the reported market value as the market value
 
 
1 passed, 0 failed, 0 skipped, took 0.46 seconds.
 
 

Note the nice Product-Owner-friendly output. This was one of my bullets from Part 1. Check.

Remember MTEGOTAG? It turns out that this increases my MTEGOTAG number to about 5 minutes. After that the product owner’s eyes glaze over and their “thinking about leaving early for a round of golf” program pegs their CPU. They can still search this output though if they have a question about something. But from a mean of around 13 seconds to 5 minutes is an amazing feat!

The other spec:

[Concerning(typeof(Account), "Market Value")]
public class with_plugged_market_value : account_spec
{
  Because the_account_has_plugged_market_value_of_200 = () => Account.EstimatedMarketValue = 200M;
  It should_give_the_estimated_market_value_as_the_market_value = () =>
    Account.MarketValue.ShouldEqual(Account.EstimatedMarketValue);
}

Which fails until we implement it:

public virtual decimal MarketValue { 
  get { return EstimatedMarketValue ?? ReportedMarketValue; }
  set { throw new NotImplementedException(); }
}

And all our specs run together look like:

------ Test started: Assembly: Specs.dll ------

Account Market Value, with no plugged market value
» should give the reported market value as the market value

Account Market Value, with plugged market value
» should give the estimated market value as the market value


2 passed, 0 failed, 0 skipped, took 0.46 seconds.

This can almost go straight into our trac wiki. Maybe someone I work with will read this and fix our build scripts to update trac.

One quick note: When you run the spec project in TD.NET, it doesn’t show the output of all the specs, just how many failed/succeeded. I could probably fix this and submit a patch. Hopefully I will.

Final things to note: There are a lot of conventions in play here – ex.: the concern’s name and the spec’s name and how they get formatted in the end. I like conventions because it means there’s less to explain to get going. I can see new developers being able to easily learn write specs like this, but I bet they are going to run into a lot of bizarre compiler errors along the way. Good way for them to learn about lambdas and c#!

So, apologies for the lame - yet long - and simplistic example. This is almost like something we did on a real project at work, just way simpler. Now that we have a simple example to start with, in my next post, I want to go over a more complicated example – one in which I felt a little bit more pain and confusion. That’s going to take me longer to type up though. In the meantime, please comment. Thanks!


 
Categories: bdd

Fellow Seattle ALT.NETter Aaron Jensen and his friends at Eleutian released a really interesting library of functionality several months ago called “Machine.” There are several things in here I can use every day. Not all of it is perfect, but it’s very innovative, clean, and easy to work with.

Today, I want to talk about Machine.Specifications, or MSpec as I will refer to it going forward. If you haven’t read Aaron’s introduction post, read it now, then come back and read this.

I am still learning BDD but I am already enjoying some benefits of it in my tests. For the most part, I have just been adopting BDD style naming of my fixtures/tests (or rather, contexts/specifications), using my existing testing framework (mbunit), and using really detailed versions of the stories our Product Owners come up with. I get two main benefits:

  • Product Owners would know what a test was covering.
  • My tests are coupled to the user stories, not the code.

The advantages are offset by:

  • Product Owners don’t look at the code. It’s a psychological barrier, but it’s there.
  • I am still new to BDD so keep running into things that I find hard to express in BDD terms.

One thing almost all BDD frameworks offer that hits one of those bullets is functionality to format a bullet point list of the specs that are implemented from tests following a naming convention. That alone is enough reason for me to try some of these frameworks.

Concept: MTEGOTAG (pronounced “Em Teg-O-Tag”): Mean Time to (Product Owner) Eyes Glazing Over Thinking About Golf. When I write my specs in c#, this number is very low, approaching zero. I can keep a Product Owners interest for 10, maybe 20 seconds if I am showing specs in c#. The number gets even lower for each repeated application per session.

So that’s kind of my motivation. I recently had some time to sit down and actually do real work in mspec, my next post will be about the results.

P.S.: If you are in the seattle area and don’t already know this, a bunch of us are meeting to talk about BDD this Thursday.


 
Categories: bdd