OrthoCoders

You can code it, I can help!

Testing With Random Values: Is It Really Dangerous?

Last Saturday I presented a session about automation at the Winnipeg Code Camp. I've been doing presentations a the WPG CC for the past three years, and is always a blast! Great sessions, lots of networking and great food is an awesome combination! After my session, some of the attendes approached me and started to talk about CI, their gotchas about testing and woes :). One of them mentioned that he wasn't sure about the way they were using "hardcoded" values with the tests. He felt that probably should be a better way of defining valid inputs that could avoid the "smell". My suggestions was to consider using auto generated values. Random values are a great tool to avoid hardcoding inputs. Using "factories" to create valid test values helps us to make sure we have better coverage of the domain of valid inputs for our tests.

How do I know what input is valid?

You may wonder "But, if the values are random, the test may pass once and then fail at runtime, that's really, really, dangerous!!!". The answer is No, not at all. The key is to identify the domain that makes your test pass and use the random generation tools to get values that are part of that domain. How to do that? Keep reading ... When you write a unit test you may have more than one scenario. For each scenario you need to find a set of inputs that make it pass. For example, let's use a method that verifies if a string is a palindrome. Definition of palindrome in wikipedia:
palindrome is a word, phrase, number or other sequence of units that can be read the same way in either direction
Our method may look something like this in C#: [sourcecode language="csharp"] bool IsPalindrome(string word) [/sourcecode] So now, what should we do? Write one test that takes random strings as input, run it once and if we are lucky and passes then we are done? Woah, slow down your horses..... take a deep breath... maybe two....

Preconditions, postconditions and domains

The precondition of the method is the set of values that are valid in order to call the method. Valid means that we don't get an exception when we use it, no matter what the result of the method is. So, what are the values that are valid to use for IsPalindrome? It seems to me that any string should work, except null, therefore:
  • Precondition: Any not null string instance.
Great! Now let's think about the postcondition and discover our scenarios. The postcondition defines what are the expected results and under what conditions are calculated. Following our definition of what the method does, we can have two possible outputs:
  • True: if the input is palindrome
  • False: if the input is not palindrome
Thus, we have two scenarios:
  • When the word is palindrome it should return true
  • When the word is not palindrome it should return false

Writing the test

We need to test two scenarios and the best way of doing that is to write one test per scenario. The first test should check the first part of the postcondition, I want to get true as result, and to do that the domain I need to use would be all the words that actually are palindrome. Let's write the scenario:
Given I have a palindrome phrase When I check IsPalindrome Then I should get true
I like to use the Given-When-Then syntax, it's very descriptive. I'm using the MT Testing library that enforces GWT. Here is the scenario using C# and just one example: [sourcecode language="csharp"] protected override void GivenThat() { this._phrase = "Madam Im adam"; } protected override void WhenIRun() { this.Actual = PalindromeChecker.IsPalindrome(this._phrase); } [It] public void Should_validate_the_phrase_is_palindrome() { this.Actual.Should().Be.True(); } [/sourcecode] The test looks good, a bit of TDD and all green. Don't worry about the implementation, you can download all the source and check it out at the later ;).

Using Factories

I'd like to make sure that for any palindrome phrase (that's the domain we got from the pre and post condition) the test passes, no just for one example. However, though generating all the palindrome words is a bit ambitious, we could settle for generating a collection of phrases that are palindrome and use that as input. MbUnit provides an attribute named "Factory" that allows a static method to define a collection of values to be passed to the test, let's look at the code: [sourcecode language="csharp"] private readonly string _phrase; [Factory("PalindromeFactory")] public When_palindrome_checker_checks_a_palindrome_phrase(string phrase) { _phrase = phrase; } protected override void WhenIRun() { this.Actual = PalindromeChecker.IsPalindrome(this._phrase); } [It] public void Should_check_the_word_as_palindrome() { this.Actual.Should().Be.True(); } protected static IEnumerable<string> PalindromeFactory() { yield return "Madam I'm adam"; yield return "Draw pupil's lip upward"; yield return .... } [/sourcecode] Using the attribute, we indicate the runner that the test should be run for every value in the collection. We run the test, and we can see in the result window something like: [sourcecode language="txt"] ### Step When_palindrome_checker_checks_a_palindrome_phrase("Madam I\'m adam"): passed ### ### Step When_palindrome_checker_checks_a_palindrome_phrase("Draw pupil\'s lip upward"): passed ### ### Step When_palindrome_checker_checks_a_palindrome_phrase("Gateman sees name, garageman sees name tag"): passed ### ### Step When_palindrome_checker_checks_a_palindrome_phrase("Go hang a salami; I\'m a lasagna hog"): passed ### ### Step When_palindrome_checker_checks_a_palindrome_phrase("I roamed under it as a tired, nude Maori"): passed ### ### Step When_palindrome_checker_checks_a_palindrome_phrase("Live not on evil"): passed ### [/sourcecode]

Using random strings

One case covered, one to go. The second test should verify the second part of the postcondition. I want to get false as result, so the domain I need to use is all the words that are not palindrome. Let's write the scenario:
Given I don't have palindrome phrase When I check IsPalindrome Then I should get false
And the code: [sourcecode language="csharp"] private string _phrase; protected override void GivenThat() { this._phrase = "This is not palindrome"; } protected override void WhenIRun() { this.Actual = PalindromeChecker.IsPalindrome(this._phrase); } [It] public void Should_not_validate_the_phrase_as_palindrome() { this.Actual.Should().Be.False(); } [/sourcecode] The code works fine. However we have a similar situation as before, I'd like to have more values that belong to the domain that makes the method fail, that means any phrase that is not palindrome. Factories are a good idea when we have a set of values that we want to use. In this case I don't want to hardcode non palindrome phrases. I would prefer to have a way of creating those phrases without needing to enumerate all the examples. Luckily MbUnit has another attribute that we can use to generate random strings. It's called RandomStrings and it takes a regular expression to describe the string. Adding the attribute the code looks like this: [sourcecode language="csharp"] private readonly string _phrase; public When_palindrome_checker_checks_a_non_palindrome_phrase( [RandomStrings(Count=10, Pattern=@"Weird [A-Z]{5,8} [0-9]{2}")]string phrase) { _phrase = phrase; } protected override void WhenIRun() { this.Actual = PalindromeChecker.IsPalindrome(this._phrase); } [It] public void Should_not_validate_the_phrase_as_palindrome() { this.Actual.Should().Be.False(); } [/sourcecode] I'm using a regular expression that uses numbers at the end to make sure the phrase is not palindrome. We run the tests, and here is the output: [sourcecode language="txt"] ### Step When_palindrome_checker_checks_a_non_palindrome_phrase("Weird VGSKTTZ 64"): passed ### ### Step When_palindrome_checker_checks_a_non_palindrome_phrase("Weird VLMEWIJ 59"): passed ### ### Step When_palindrome_checker_checks_a_non_palindrome_phrase("Weird NDICJK 08"): passed ### ### Step When_palindrome_checker_checks_a_non_palindrome_phrase("Weird FEMBMSKK 88"): passed ### ### Step When_palindrome_checker_checks_a_non_palindrome_phrase("Weird IHOIOWN 38"): passed ### ### Step When_palindrome_checker_checks_a_non_palindrome_phrase("Weird ITIRCHAA 63"): passed ### [/sourcecode]

The moral of the story

Random values are a powerful tool, not dangerous per se. They are part of our toolbox as developers. Can we use it wrong? Sure, but that's a completely different story .... If you want to get the full code, download it from the examples in MT Testing. Check the Palindrome folder. Do you have any other stories or comments about random values that you would like to share, please leave a comment, feedback is more than welcome. Want to see an example with major complexity? Something that fits better your case? Tell me about it and I'll add it to the MT Testing framework as an example of usage (and I'll give you credit for it too!).

MVC Virtual Conference #2

On Feb 8th I participated on the MVC Conf #2. It was a great experience, though is still a bit weird not to see the crowd :-). My session was about "Quality driven acceptance tests using Capybara" and it was a bit of a shock because many .NET developers weren't expecting Ruby as a tool and many got confused and thought I was using Ruby on Rails... but I guess that's material for another post. Overall I really enjoyed presenting and I hope  to do it again next time. You can download the slides and source code from presentations. I just updated the README file with some troubleshooting. If you have any issues running the code, or with the setup let me know.

Presentation at Codemash

Wow, first day of CodeMash and so far it's being great! It's really an awesome conference! Yesterday at the precompiler, I went to the " Getting Published in an Evolving Industry: How to Survive and Even Thrive" with Jason Gilmore (@wjgilmore) and learnt a lot about writing books and how to publish, I'm really tempted to write an ebook! Then in the afternoon I went to the "Git Immersion" session and was pretty interesting, is a tough call because for every session I go there is another four I can't ... Today I'm going to do my presentation about IronRuby. I uploaded the slides and code to the presentation page, please feel free to download it and check it out before the conference. In order to run ir u need to: * Install Iron Ruby from http://ironruby.codeplex.com/ * Remember to put it in the path * Install rake: igem install rake * Install bundler: igem install bundler * Check the tasks: rake -T * Setup dependencies: rake setup * Build: rake * Run the tests: rake test That's it! Please contact me at @abarylko or amir@barylko.com if you have any issues. Hope to see you at the session! Say hi if you are at codemash! Now, in the afternoon, which session to go.... which session to go.....

Microsoft Techdays 2010

Wow! First day of Techdays is gone and I've done both of my presentations:
  • Test Driven Development Patterns
  • Top 10 mistakes in unit testing
The attendance was pretty good and all the attendees were great! Thank you everyone, I had a great time! I've uploaded to presentations the slides and a link to the source. Both presentations using the same code. Please follow each commit that mimics the steps I did in the presentation. If u want to run the code please do the following:
  • Download the code
  • Install ruby (if u don't have it)
  • Install rake: gem install rake
  • Install bundler: gem install bundler
  • Then do setup: rake setup
  • Run the test to make sure: rake tests
Please let me know if you have any issues or questions.