puts Blog.new(”nonsense”)

Tests increasingly serve multiple roles in today’s projects. They help us design APIs through test-driven development. They provide confidence that new changes aren’t breaking existing functionality. They offer an executable specification of the application. But can we ever get to a point where we have too much testing?

Enough is Enough

Consider the following test that you might come across in an application with a less-than-ideal test suite. (We’ll pick on some badly-written Rails code for this example, but the ideas we’ll discuss are certainly not unique to just Rails or Ruby or even just to dynamic languages. Unfortunately, these problems are quite universal.)

Whoa! That sure is a lot going on in a single test case. What exactly is it that we’re trying to test here? Let’s take a step back and see if we can figure it out. [1]

• Line 5 - We start off by creating a new product. We give the product a name and a price and call #create. Since we’re in a Rails app, without looking elsewhere in the code, we can make an educated guess that Product is an ActiveRecord subclass and that calling #create will persist the product to the database.
• Line 6 - We send an HTTP GET request to a #show method and pass along the ID of the product we just created. Since we’re in ProductsControllerTest, we can safely infer that we’re calling the #show action in the ProductsController class.
• Line 7 - We assert that the controller action responded with HTTP status code 200 (i.e., success).
• Lines 8-9 - We look for an object stored in the assigns hash (i.e., the hash of variables that will be available to the view template) with a key of :product and we assert that it’s not nil.
• Line 10 - We verify that the product object satisfies all the product validation rules (though we can’t know what those rules are without taking a look at the Product class).
• Lines 11-12 - We assert that the attribute values we provided when creating the product (in line 5) match the attribute values stored in the object we fetched from the assigns hash.

For a class called ProductsControllerTest, it sure feels like we’re doing a fair bit more than just testing the code we’d expect to find in ProductsController (assuming ProductsController conforms to the traditional responsibilities of a controller). Let’s take a look at the controller code to better understand exactly what it is that we’re testing.

It turns out that our controller code is notably simpler than our test case would have led us to believe. The #show action does indeed stick to the expected behavior of a controller, and in this case, it’s able to provide that behavior in a single line of code (despite the 8 lines of test code currently employed above). That single line satisfies all of the expectations of the #show action: look up the product with the given ID and place the product object in an instance variable for use by the view.

If the #show action isn’t responsible for knowing how to successfully save a new product record into the database, why does our test case attempt to provide the data for creating a new product? What will happen to our test case when someone adds a new validation rule requiring that all products also include a quantity attribute? Unfortunately, the test as it’s currently written will break. And while a failing test is often a welcome warning that our changes have inadvertently broken a part of our application, it’s far from desirable for a model-level rule related to creating new records to cause a failure in a controller test related simply to displaying existing records. In short, we’ve given our test case too much responsibility, and in doing so, we’ve made it fragile.

We can see from the #show action above that not only is the controller not responsible for knowing how to create a valid product, it’s also not responsible for ensuring that a product’s attributes are properly populated when the record is read from the database. However, if we take another look at the last few lines of the test case, we might think otherwise. That brings us to the second problem with this particular test: it’s a rotten source of documentation for the code being tested. As we increasingly move toward tests as specifications of our application’s behavior, it’s vital that those specifications clearly communicate the expected behavior. As it’s currently implemented, the overspecification in this test case leaves the reader having to do way too much work to figure out the true expectations of the code being tested.

Communicate Essence

Let’s take another pass at writing a test for the #show action, this time with an eye toward removing the fragility of the previous test case and increasing the value of the test as a specification.

In this implementation, we’ve abstracted away the logic for creating a new product in line 5. We’ve defined a helper method for use by any and all tests in our application that have a need to create a new product. If and when the rules for successfully creating a new product change, we’ll update the #create_product method, and we won’t have to touch the code in ProductsController or ProductsControllerTest at all. [2]

As for the four assertions that appeared at the end of our first attempt at this test case, we’ve replaced those assertions with a single (stronger) assertion. Where we previously asserted that the assigns hash held a non-nil product, that the product was valid, and that its attributes matched the attributes used at the beginning of the test, we now simply verify that the product object in the assigns hash is equal to the product object that we created at the beginning of the test. That single line more accurately and more concisely expresses our expectation: that the product whose ID we provide in the request to the #show action is the same object that’s made available to the view.

By focusing our test case on the essence of the code under test, we’ve ended up with less test code to maintain. And with the extraneous setup and assertions removed, the remaining test code provides a clearer and less fragile specification of the behavior being tested. [3]

May I Take Your Order, Please?

In the previous example, we might have detected the overspecification by the significant mismatch between the lines of test code and the lines of production code, or seeing model-specific assertions in a controller-specific test may have caught our eye. But, overspecification comes in more subtle forms as well. Consider the following method provided by ActiveRecord::Base for fetching the list of columns that hold domain-specific content from a model class in Rails.

To test this method, we’ll need a sample ActiveRecord model class. Let’s use the Topic model, which is backed by the following table.

In our first attempt at testing the #content_columns method, we might come up something similar to this:

Can you spot the overspecification? To give you a hint, compare that test to the actual test employed in the ActiveRecord test suite:

If you’ve ever written a test for something that returns an ordered list of items where the actual order either doesn’t matter or isn’t guaranteed, you’ve probably been bitten by this breed of overspecification, and there’s a good chance your solution matched the solution used in the ActiveRecord test code above. Because the #content_columns method doesn’t guarantee that the columns will be returned in any particular order, it’s inappropriate (and fragile) for our test to specify a certain order.

While the actual ActiveRecord solution above works, it too could be better. First, since we know that the two arrays in line 6 won’t be equal unless they have the same length, we can safely remove the extraneous length assertion in line 5. Second, when we see the calls to #sort on line 6, we’re forced to pause (if even for just a moment) to think about why it might be important to sort the two arrays. We don’t really want to sort the arrays; we’re just using that technique to get around the fact that order doesn’t matter to us, but it does matter when Ruby compares two arrays for equality. Since we really only want to assert that the arrays have the same elements, why not do exactly that?

By using an assertion like Shoulda’s assert_same_elements method, our test can clearly and concisely express the expected behavior.

Use It Wisely

Overspecification comes in many flavors, and the examples above in no way represent a comprehensive list. As developers, we frequently strive to write as little code as possible to accomplish the task at hand. When it comes to writing tests, we should very much keep that goal in mind as well. Good tests communicate the essence of the scenario being tested and nothing more. While I doubt a project will ever suffer from too much testing, it can certainly suffer from tests that specify too much.

Notes

[1] If this test seems absurd to you, good! Your ability to detect this type of overspecification will serve you well.

[2] The actual implementation of the #create_product method isn’t particularly pertinent in this discussion. (In the Rails space, there’s certainly no shortage of options.)

[3] With all the excessive thoroughness that went into overspecifying the the current behavior of the #show method, is it possible that we’ve been too distracted to identify other functionality that’s still missing from that method? In the next post, we’ll take a look at overspecification’s more lethargic cousin: underspecification.

Thanks to Justin Gehtland, Muness Alrubaie, and Glenn Vanderburg for reading drafts of this post.

So you’ve taken your project to 100% code coverage, you’ve configured your continuous integration system to fail the build if that coverage ever drops below 100%, and you’re ready to enjoy the fearless refactoring and the rock solid regression testing suite that your software engineering rigor has now earned you. But are you really covered? What does 100% code coverage mean in your project? Is it enough to know that your test suite encounters every line of code? Or don’t you want to be sure that it exercises every line? If you simply encounter the line without asserting that it produces the correct results, are you any better off? [1]

Failure to Assert

Just achieving 100% code coverage is the easy part. Making it mean something: that’s where the real value kicks in. Consider the ease with which we can get to 100% line coverage on the following code (generated using Rails 2.1 scaffolding).

In order to ensure that the #index method is performing all its proper duties, we’ll define the following “test case.”

We’ll use rcov to assess the results.

And just like that, we have 100% code coverage for the #index method. [2] In this case though, that clearly means nothing more than the fact that we encountered 100% of the lines in the method. When code coverage is that easily achieved, it hardly seems cause for celebration. To get any real value from the test, we need to actually assert that we’re getting the expected results. Until we do so, we have nothing more than incidental coverage. [3]

The test code provided by the Rails scaffolding gets us closer to where we want to be.

The tests pass, and our code coverage is still at 100%. At this point we’ve significantly increased the value of that particular test. No longer does the code have to crash spectacularly in order to yield a test error. Instead, exiting the method with anything other than a success response code will result in a test failure. Similarly, failure to set the @products instance variable will cause the test case to flunk. [4]

100% Covered. 50% Tested.

But while we’ve improved on the initial test case, at best we’re really only halfway to where we want to be. What would our test suite tell us if we were to alter line 9 as follows.

Unfortunately, our test suite still blindly gives a thumbs up, despite the fact that any attempt to access the XML-formatted output would yield an exception. While our test suite includes assertions for how the application should prepare an HTML-bound response, our coverage of the XML-specific logic in line 9 remains merely incidental.

If we want our automated test suite to ensure that the #index method remains capable of producing an XML-formatted response as our codebase changes over time, we’d need to add a test case to exercise that code.

rcov doesn’t reward us with any extra credit for writing this test case, but achieving 100% coverage is not the primary goal of a good test suite. [5] The primary goal is to ensure that the code satisfies the requirements. If the application really is required to provide an XML-formatted list of products, then we should seriously consider defining a test for that functionality in our test suite. [6]

Size Matters

In the examples above, we started off with a 5-line method that had 100% line coverage but lacked any assertions. That scenario provides some insight into other places where we might find a high percentage of incidental coverage. While long methods are a bad idea in general, the simple act of invoking a method may be enough to register it as having 100% coverage. If our test suite results in the execution of a 25-line method with no branches, all 25 lines are considered to be covered, regardless of whether we perform any assertions to verify the results of those 25 lines. Even if we refactor the code into smaller methods, if we don’t unit test those new methods, we’re still left with nothing more than incidental coverage.

Use It Wisely

If your goal is to achieve 100% coverage, then incidental coverage is your friend, but your test suite will fall far short of its full potential. You’ll run the risk of acquiring a false sense of security, and the ability to safely and fearlessly refactor is out the window. Changes or enhancements to your application will occur without the safety net of a full regression suite.

If your goal is instead to develop a comprehensive test suite to A) validate your application’s functionality and B) ensure that you build just enough software, then test-driven development (TDD) is your friend (as are peer reviews and/or pair programming). And subsequently, you’ll enjoy the nice side effect of 100% code coverage, because you’ll only build the code necessary to satisfy your tests.

Notes

[1] Sure, you know that the line executes without causing the application to crash (at least for this set of inputs), but does a lack of crashing really constitute success? Not likely.

[2] While we have 100% line coverage for the #index method, we don’t have 100% coverage for the ProductsController class as a whole. As the full coverage report shows, the tests provided with the Rails scaffolding do not cover the exception cases in the #create and #update methods.

[3] Incidental coverage is just as valuable as that guy that puts in “face time” at the office. He’s physically present at least 8 hours every day. People see him there. He’s at the office. He must be doing something. Right?

[4] At this point we’ve gone from covering nothing to covering something. That’s good, but we should ask ourselves whether we’re performing the most appropriate assertions for this test case. For example, is it enough to assert that assigns(:products) is not nil, or should we be making a stronger assertion about its value? Do we want to ensure that we’re rendering the intended view templates? What about verifying the content of the HTML (or XML) that gets rendered? Tackling issues specific to testing Rails controllers would take away from the focus of this post, but the strength of assertions and the layers at which we test is surely fodder for future posts.

[5] However, with anything less than 100% coverage, fearless refactoring and full regression testing simply isn’t something your test suite can provide. In Software Testing Techniques (1996), Boris Beizer takes the hardline on anyone that would consider developing a codebase without at least 100% line coverage: “[Line coverage] is the weakest measure in the family [of structural coverage criteria]: testing less than this for new software is unconscionable …”

[6] You could argue that this new test case is too similar to the original test case, that it needs stronger assertions, or that the current functionality is too simple for this new test case to offer sufficient benefit, but it’s up to you to assess how important automated testing is of that code. If you opt to not test your application’s ability to provide an XML-formatted list of products, you must not allow your 100% line coverage to give you a false sense of security about your code. That code may be covered, but it’s not tested.

Thanks to Stuart Halloway for reading drafts of this post.

In any discussion of code coverage, it’s important to understand the type of coverage that’s being measured. In the same way that you wouldn’t plan a trip to the beach without knowing whether it’s going to be 30 degrees Celsius (perfect!) or 30 degrees Fahrenheit (stay home), we can’t have a truly meaningful discourse on code coverage without first identifying the analysis type.

Lining Up

Many popular code coverage analysis tools currently report what’s known as line coverage (also commonly referred to as statement coverage). Line coverage analysis (as the name implies) identifies which lines were encountered as a result of your tests.

To better illustrate what we can expect from line coverage analysis, let’s first consider the following Ruby module and a possible test case for that module.

Read the rest of this entry »

Andres Almiray interviewed me this week for the Groovy Zone. We cover a breadth of topics, including:

• Just how far Grails has come in the past two years
• Why the GORM DSL likely obviates previous mapping techniques
• Groovy as a gateway drug to more and better developer testing
• Why Grails testing infrastructure improvements deserve top billing in Grails 1.1
• Something called Rails
• New testing-related developments in the Groovy ecosystem

For all that and more, check out the interview at Groovy Zone, a new(ish) and hoppin’ community for Groovy and Grails news.

(Did I mention that we discuss testing?)

Many thanks to Andres and DZone for the interview.

• More evidence that 100% test coverage is just a good place to start.

• And here I thought perhaps it was finally time to drop BASIC from my resume. ^§

• Dave Klein takes on a gnarly Oracle schema using the Grails ORM DSL. If you’re dying to see some XML or annotations in use, then well, you need help, and this tutorial simply ain’t for you.

• Git repo containing the complete Rails source code and it’s entire revision history: 21.9 MB. SVN checkout of the current Rails source code with no history: 23.8 MB. Convinced yet? ^‡

• Safari 3.1 hits the street, now with more cowbell marginally better dev tools. Oh well, it’s still feels like the best fit for day-to-day browsing, but it’s got a long way to go if it’s ever gonna compete with Firebug for some real web developer love.

–

^§Link courtesy of Rob Sanheim.

^‡Link courtesy of Stu Halloway.

In the spirit of Andy Glover’s Weekly Bag and Bill Dupre’s frequent batches of Whatever, herein lies the first installment in a (sure-to-be-sporadic) series of Noteworthy Nonsense.

• iPhone web app authors rejoice! (Yes. You read that right. Web-app authors.) Test your web apps using the official iPhone simulator. (And, oh yeah, you can go native now too…but the lack of RubyCocoa support is a bit of a downer.)

• A plugin that lets you run Struts 1.x code inside a Grails app? Yeah, I cringed too, but this will surely be a welcome migration path for those folks trapped in the land of *.do.

• Are SVN users suffering from the Blub paradox? Linus pulls no punches in offering up his take on this matter.

• Dan Benjamin’s back with the Leopard edition of his definitive “how-to” guide for rolling your own installation of Ruby, Rails, and friends.

• The last time Glen Smith declared a month o’ Grails, he showed the community just how very much is possible with merely an hour a day. Now Glen’s at it again, but this time it’s MockFor(March). If history is any indicator, expect Glen to blaze new trails in the land of Grails unit testing. Glen: It takes GUTs, but I’m rootin’ for you!

• Speaking of good unit tests, Jay Fields announced a new version of the expectations gem, complete with a healthy dose of example code. One expectation per test? Saying “goodbye” to cumbersome test names? Jay’s onto something here.

The inaugural Groovy/Grails Experience (2GX) is in the bag, and as much as I was looking forward to it, it honestly exceeded my expectations rather significantly. The sheer enthusiasm for Groovy (and Grails) in the Java community is almost palpable.

The event capped off with a new installment of Relevance’s Refactotum series. Jay Zimmerman (Director of 2GX and the popular NFJS Symposium Series) was kind enough to dedicate the final afternoon of the conference to this unique session demonstrating the ease of contributing to open source, followed immediately by a workshop for putting those techniques into practice. Coming out of the Refactotum, we had a widely diverse set of contributions, all of which make meaningful improvements to Grails. The resulting set of patches involve increasing test coverage, refactoring for readability, boarding up broken windows, reducing complexity, and a handful of other improvements as well. And perhaps the most rewarding part (for me, at least) of the three-day conference was talking to the several people that stopped by afterward, each to express that while they had never before understood just how easy it is to get involved in open source, now they know, and each seemed downright eager to contribute!

Special thanks to Scott Davis, Jeff Brown, Dierk König, Alex Tkachman, Alexandru Popescu, Ken Kousen, and Daniel Hinojosa for their excellent and insightful contributions to the discussion!

–

Resources

Slides

• Refactotum #6 - Grails (2GX)

Helpful Links and Tools Discussed

• Jay Fields - Tests Reflect Code Quality
• Code Coverage (Wikipedia)
• Grails Code Coverage Report
• Cobertura
• easyb
• Testing Private Methods in Ruby^§

Resulting Patches (Keep ‘em Coming!)

• GRAILS-2412
• GRAILS-2469
• GRAILS-2470
• GRAILS-2471
• GRAILS-2521
• GRAILS-2527

^§The controversy of testing private methods isn’t unique to Groovy…but seems common to Refactotums.

Believe it or not, there are still at least a few of us crazy hold-outs that still haven’t imbibed the increasingly ubiquitous RSpec Kool-Aid. And for the folks in this crowd that still want BDD-style testing, it’s test-spec and test/spec/rails all the way. So when I recently needed a means for validating that a nontrivial route landed in the right controller/action, I naturally went looking for the test/spec/rails equivalent of #assert_recognizes. When I saw that it was nowhere to be found, I knew I’d found another task for Open Source Fridays (er, the first half hour of an Open Source Friday). And thanks to Matthew Bass’s uncanny responsiveness in applying the patch that Rob Sanheim and I submitted, test/spec/rails now offers some handy new specs for testing your routes.

assert_generates => should.route_to

Where you’d use #assert_generates in traditional Rails + test/unit, test/spec/rails now supports #route_to as a more spec-flavored alternative. For example, to verify that a given set of URL options routes to the expected path…

becomes

assert_recognizes => should.route_from

And where our ancestors once used #assert_recognizes, test/spec/rails now offers #you_bettuh_recognize #route_from. “An example would be nice”, you say? Well, to perform essentially the inverse of the above test…

- - -

The rdoc offers additional examples, and check out the test/spec/rails README for more mappings from old-school test/unit assertions to test/spec/rails equivalents.

The Groovy community’s been busy rolling out a steady stream of holiday goodness for the Java developers of the world: Groovy 1.5 was just released, Grails 1.0 RC3 is out, and now the Prags are prepping two new Groovy books for early next year. And to top things off, the folks behind NFJS are hosting a three-day conference with Groovy and Grails experts from all over the world.

The inaugural 2G Experience will take place February 21-23 in Reston, VA, and the agenda is slam-packed! BDD with Andy Glover. DSLs with Venkat Subramaniam. Google Maps with Scott Davis. Metaprogramming with Jeff Brown. A can’t-miss JRuby/Groovy smackdown with Neal Ford. A Grails keynote with Graeme Rocher. The list goes on…

I’ll be presenting sessions on Going Further with Grails and Bending GORM: 5-minute Techniques for Enterprise Integration. And to close out the conference, Relevance’s Refactotum series will make its Groovy/Grails debut. I’m teaming up with Scott Davis and Venkat Subramaniam (and any stray Groovy/Grails devs that happen to wander nearby) to host this hands-on workshop helping attendees make their mark on the Groovy and Grails revolution.

Refactotum: Groovy/Grails (3-hour workshop)

Contributing to open source is great for your career. In a few short hours, you can learn, teach, promote your skills, and improve the quality of the community. In this unique workshop, we will show you how, by doing it. Using Grails as an example, we’ll show you how to:

• download the source code
• build and run tests
• use Cobertura and code review to find problem areas
• refactor some code
• create and submit a patch

Take this opportunity to begin contributing to Groovy, Grails, or any other open source project that interests you. Experts from the Groovy and Grails community will be on hand to help you get started.

So is Groovy really “the next generation” of the Java language? Come decide for yourself. As for me, I couldn’t agree more.

Smart developers. Really.

Working together.

In small teams.

Using the best tools available.

Embracing true agility.

Pushing each other.

And giving back.

In the constant pursuit of excellence.

Are you in?

Then check us out.

And let’s hear it: jobs@thinkrelevance.com