Top unit-testing Questions

List of Tags
516
Mark Levison

Let's consider the state of JavaScript unit tests and testing tools.

JsUnit

We already use it for some of our js code.

Pros

  • can be invoked from an ant build file
  • launches browser to run the tests
  • Eclipse plug-in

Cons

  • launches browser to run the tests
  • Does not support js file to write the unit test code: it has to be embedded inside an html file
  • it has not been updated for a few years

Notes:

  • There is a JsUnit (2).
  • An 'ant' is an open source build tool; "Ant" because it is a little thing that can build big things.

RhinoUnit

Pros

  • ant driven
  • supports js file
  • very simple to use

Cons

  • Simulation of JavaScript engine: not advanced enough to support our code: I tried to run test code working with JsUnit: I encountered issue when loading our common JavaScript files

crosscheck

Pros

  • Can be invoked from ant build file
  • Simulates real browser behaviour

Cons

  • Simulation of JavaScript engine from a limited number of browser versions
  • No activity for 2 years: it does not support Firefox versions 2.x nor 3.x

jsspec

Pros

  • Runs on actual browser

Cons

  • JavaScript only framework: cannot be called from ant build file

jspec

Pros

  • Runs on actual browser

Cons

  • Does not seem to support our code: I tried to run test code running with js unit: I encountered issue when loading our common JavaScript files
  • JavaScript only framework: cannot be called from ant build file,

Screw.unit

Pros

  • Runs on actual browser

Cons

  • JavaScript only framework: cannot be called from ant build file

Note: I did not try it but it is very similar to jsspec and jspec.

It looks like JsUnit is the only choice we have. Please note it is already used in other components. It is not perfect though because it does not provide an easy way to apply the Test-Driven Development (TDD) process for the following reasons: - It does not provide a simple and integrated way to run JavaScript unit test - It forces you to write the unit tests in a html file instead of a .js file, - It forces you to have a local installation of the JsUnit framework in order to avoid absolute hard coded path to reference js unit files.

As a consequence, you have to switch back and forth from you IDE and all the browsers we want to support while "TDDing" in JavaScript. It is feasible but I do not think it is very effective (I may be wrong here).

Also I still need to look at a proper JavaScript editor or Eclipse plug-in allowing easy referencing and refactoring. There is the JSTD one which is part of WTP, however I have not played with it enough in order to have a good idea of it.

As a conclusion, I do not think we have the tools to be fully "TDD" compliant. We can be TOD (Test Oriented Development), but I have not found any tool that allows us to work with JavaScript as efficiently as with Java. Once again, the ideal solution would be to have something identical to JUnit.

What JavaScript unit testing tools do you use?

Recent Entries:

Answered By: gregers ( 359)

Buster.js

Very similar server/client concept as JsTestDriver (see further down). Except that the server is built with JavaScript (node.js) instead of Java, and the API follows JS best practices to a much higher degree.

A browser JavaScript testing toolkit. It does browser testing with browser automation (think JsTestDriver), qunit style static html page testing, testing in headless browsers (phantomjs, jsdom, ...), and more. Take a look at the overview!

A Node.js testing toolkit. You get the same test case library, assertion library, etc. This is also great for hybrid browser and Node.js code. Write your test case with Buster.JS and run it both in Node.js and in a real browser.

Screencast: Buster.js Getting started (2:45)

pros:

  • Uses node.js, so compatible with Win/OS X/Linux
  • Run tests from browser or headless with PhantomJS (soon)
  • Run on multiple clients at once
  • Supports NodeJS testing
  • Don't need to run server/clients on development computer (no need for IE)
  • Run tests from command line (can be integrated in ant/maven) Write tests xUnit or BDD style
  • Supports multiple JavaScript test frameworks
  • Defer tests instead of commenting them out
  • SinonJS built in
  • Auto-run tests on save
  • Proxies requests cross-domain
  • Possible to customize:
    • Extend it to wrap other test-frameworks (JsTestDriver built in)
    • Your own assertions/refutes
    • Reporters (xunit XML, traditional dots, specification, tap, teamcity and more built in)
    • Customize/replace the HTML that is used to run the browser-tests
  • TextMate and Emacs integration

cons:

  • Stil in beta, so can be buggy
  • No plugin for Eclipse/IntelliJ (yet)
  • Doesn't group results by os/browser/version like TestSwarm *. It does however print out the browser name and version in the test results.
  • No history of previous test results like TestSwarm *
  • Doesn't fully work on windows as of nov 2012

* TestSwarm is also a Continuos Integration server, while you need a separate CI server for Buster.js. It does however output xUnit XML reports, so it should be easy to integrate with Hudson, Bamboo or other CI servers.

Testacular

Again, very similar server/client concept as JsTestDriver (see further down). Like Buster.js the server is built with JavaScript (node.js).

Screencast: Testacular Getting started

pros:

  • Uses node.js, so compatible with Win/OS X/Linux
  • Run tests from browser or headless with PhantomJS
  • Run on multiple clients at once
  • Option to launch, capture, and automatically shutdown browsers
  • Option to run server/clients on development computer or separately
  • Run tests from command line (can be integrated in ant/maven)
  • Write tests xUnit or BDD style
  • Supports multiple JavaScript test frameworks
  • Auto-run tests on save
  • Proxies requests cross-domain
  • Possible to customize:
    • Extend it to wrap other test-frameworks (Jasmine, Mocha, QUnit built-in)
    • Your own assertions/refutes
    • Reporters
    • Browser Launchers
  • Plugin for WebStorm

cons:

  • Does not supports NodeJS testing
  • No plugin for Eclipse (yet)
  • No history of previous test results

TestSwarm

John Resig (jQuery) has created a tool for distributed JavaScript testing, TestSwarm. Mainly for open source JavaScript projects, but TestSwarm is open source, so you can set up a server yourself for corporate testing. Although this might require that you to do some modifications.

pros:

  • Continuos integration server for JavaScript
  • Supports all major browsers/operating systems
  • Run on multiple clients at once
  • Don't need to run server/clients on development computer (no need for IE)
  • Automatic run tests on all clients when you commit something (or whenever you modify the script to run the tests)
  • Show history of test results pr commit
  • Supports multiple JavaScript test frameworks
  • Have test results for OS and browser versions
  • Crowdsource to test in a multitude of browsers

cons:

  • Can not break your build through ant/maven
  • Don't notice the test fail before commit
  • No IDEplug-in

http://ejohn.org/blog/javascript-testing-does-not-scale/

TestSwarm architecture:

alt text

JsTestDriver

Some people at Google have also started on a distributed JavaScript tool, JsTestDriver. It is similar to TestSwarm, that it has a server, and clients connected. But it also has support for running tests from command line and plugins for Eclipse and IntelliJ!

pros:

  • Supports all major browsers/operating systems
  • Run on multiple clients at once
  • Don't need to run server/clients on development computer (no need for IE)
  • Run tests from command line (jar) (can be integrated in ant/maven)
  • Eclipse plugin
  • IntelliJ plugin
  • Supports multiple JavaScript test frameworks

cons:

  • Doesn't show results for os or browser version. Only browser names. It does however print out the version in the test results.
  • No history of previous test results
  • Very low project activity

Overview of how JsTestDriver works at runtime: alt text

Eclipse plugin screenshot:

alt text

Short intro video: http://www.youtube.com/watch?v=V4wYrR6t5gE

YUI Yeti

Yahoo now has their own JavaScript test run server, Yeti. Built with node.js. It should be able to run your existing YUI-tests as they are, and works in multiple browsers. Since it runs from the command-line I guess it's mostly similar to JsTestDriver.

alt text

Announcement from August 25th 2010: http://www.yuiblog.com/blog/2010/08/25/introducing-yeti-the-yui-easy-testing-interface/

If you have experience with this test-runner, please contribute with more info :)

Project home: http://yuilibrary.com/projects/yeti/

Jasmine

Jasmine

This is a test-runner that might interest developers familiar with Ruby or Ruby on Rails. The syntax is based on RSpec that's used for testing in Rails projects.

Jasmine is a behavior-driven development framework for testing your JavaScript code. It does not depend on any other JavaScript frameworks. It does not require a DOM.

If you have experience with this test-runner, please contribute with more info :)

Project home: https://github.com/pivotal/jasmine/

QUnit

QUnit focuses on testing JavaScript in the browser, while providing as much convenience to the developer as possible. Blurb from the site:

QUnit is a powerful, easy-to-use JavaScript unit test suite. It's used by the jQuery, jQuery UI and jQuery Mobile projects and is capable of testing any generic JavaScript code

QUnit shares some history with TestSwarm (above):

QUnit was originally developed by John Resig as part of jQuery. In 2008 it got its own home, name and API documentation, allowing others to use it for their unit testing as well. At the time it still dependended on jQuery. A rewrite in 2009 fixed that, now QUnit runs completelty standalone. QUnit's assertion methods follow the CommonJS Unit Testing specification, which was to some degree influenced by QUnit.

Project home: http://qunitjs.com/

Sinon

Another great tool is sinon.js by Christian Johansen, the author of Test-Driven JavaScript Development. Best described by himself:

Standalone test spies, stubs and mocks for JavaScript. No dependencies, works with any unit testing framework.

Test-Driven JavaScript Development http://tddjs.com/

432
MattGrommes

How do I use JUnit to test a class that has internal private methods? It seems bad to change the access modifier for a method just to be able to run a test.

Answered By: Cem Catikkas ( 313)

If you have somewhat of a legacy application, and you're not allowed to change the visibility of your methods, the best way to test private methods is to use reflection.

Internally we're using helpers to get/set private and private static variables as well as invoke private and private static methods. The following patterns will let you do pretty much anything related to the private methods and fields. Of course you can't change private static final variables through reflection.

Method method = targetClass.getDeclaredMethod(methodName, argClasses);
method.setAccessible(true);
return method.invoke(targetObject, argObjects);

And for fields:

Field field = targetClass.getDeclaredField(fieldName);
field.setAccessible(true);
field.set(object, value);

Notes:
* targetClass.getDeclaredMethod(methodName, argClasses) lets you look into private methods. The same thing applies for getDeclaredField.
* The setAccessible(true) is required to play around with privates.

I'd like to unit test my Android application but I found that test driven development in Android is far from trivial at the moment.

Any tips, tricks, war stories for building light weight and preferably fast running tests?

Answered By: Renas ( 147)

You should try Robotium! Go to Robotium.org and download the example test project. Robotium is a really easy to use framework that makes testing of android applications easy and fast. I created it to make testing of advanced android applications possible with minimum effort. Its used in conjunction with ActivityInstrumentationTestCase2.

352
Corin

I want to start using mock objects on my next C# project.

After a quick Google search I've found there are many:

So my question is: what one is your favourite .NET mocking framework and why?

Answered By: Matt Hamilton ( 205)

I've not used most of the ones you've listed, so I can't be objective about it, but I use Moq and it has been awesome. The fluent interface makes it a joy to work with. For example:

mockService.Setup(s => s.GetCustomers()).Returns(new List<Customer>());

@Ngu Soon Hui, I wasn't aware that the other frameworks don't have compile-time checking. Moq certainly does. In my example above, if the service class that mockService is mocking doesn't have a GetCustomers() method, I would get a compile-time error. I'd also get one if the GetCustomers() method didn't return a List<Customer> or an interface like IList<Customer>.

296
The Talking Walnut

I am working to integrate unit testing into the development process on the team I work on and there are some skeptics. What are some good ways to convince the skeptical developers on the team of the value of Unit Testing? In my specific case we would be adding Unit Tests as we add functionality or fixed bugs. Unfortunately our code base does not lend itself to easy testing.

Answered By: reefnet_alex ( 360)

Every day in our office there is an exchange which goes something like this:

"Man, I just love unit tests, I've just been able to make a bunch of changes to the way something works, and then was able to confirm I hadn't broken anything by running the test over it again..."

The details change daily, but the sentiment doesn't. Unit tests and test-driven development (TDD) have so many hidden and personal benefits as well as the obvious ones that you just can't really explain to somebody until they're doing it themselves.

But, ignoring that, here's my attempt!

  1. Unit Tests allow you to make big changes to code quickly. You know it works now because you've run the tests, when you make the changes you need to make, you need to get the tests working again. This saves hours.

  2. TDD helps you to realise when to stop coding. Your tests give you confidence that you've done enough for now and can stop tweaking and move on to the next thing.

  3. The tests and the code work together to achieve better code. Your code could be bad / buggy. Your TEST could be bad / buggy. In TDD you are banking on the chances of BOTH being bad / buggy being low. Often it's the test that needs fixing but that's still a good outcome.

  4. TDD helps with coding constipation. When faced with a large and daunting piece of work ahead writing the tests will get you moving quickly.

  5. Unit Tests help you really understand the design of the code you are working on. Instead of writing code to do something, you are starting by outlining all the conditions you are subjecting the code to and what outputs you'd expect from that.

  6. Unit Tests give you instant visual feedback, we all like the feeling of all those green lights when we've done. It's very satisfying. It's also much easier to pick up where you left off after an interruption because you can see where you got to - that next red light that needs fixing.

  7. Contrary to popular belief unit testing does not mean writing twice as much code, or coding slower. It's faster and more robust than coding without tests once you've got the hang of it. Test code itself is usually relatively trivial and doesn't add a big overhead to what you're doing. This is one you'll only believe when you're doing it :)

  8. I think it was Fowler who said: "Imperfect tests, run frequently, are much better than perfect tests that are never written at all". I interprate this as giving me permission to write tests where I think they'll be most useful even if the rest of my code coverage is woefully incomplete.

  9. Good unit tests can help document and define what something is supposed to do

  10. Unit tests help with code re-use. Migrate both your code AND your tests to your new project. Tweak the code till the tests run again.

This presentation is an excellent introduction to the subject. (this version requires Firefox)

A lot of work I'm involved with doesn't Unit Test well (web application user interactions etc.), but even so we're all test infected in this shop, and happiest when we've got our tests tied down. I can't recommend the approach highly enough.

296
lostInTransit

I was testing my app on the simulator when it crashed on clicking a button of a UIAlertView. I stopped debugging there, made some changes to the code and built the app again. Now when I run the application, I get this error in the console

Couldn't register com.myApp.debug with the bootstrap server. Error: unknown error code. This generally means that another instance of this process was already running or is hung in the debugger.Program received signal: “SIGABRT”.

I tried removing the app from the simulator, doing a clean build but I still get this error when I try to run the app.

What should I do to be able to run the app on my simulator again?

Answered By: Elliot Kroo ( 135)

Try quitting and restarting the simulator? If "worse comes to worst" you can always try restarting: in my experience this should fix it.

226
Paul Osborne

I worked on an embedded system this summer written in straight C. It was an existing project that the company I work for had taken over. I have become quite accustomed to writing unit tests in Java using JUnit but was at a loss as to the best way to write unit tests for existing code (which needed refactoring) as well as new code added to the system.

Are there any projects out there that make unit testing plain C code as easy as unit testing Java code with JUnit? Any insight that would apply specifically to embedded development (cross-compiling to arm-linux platform) would be greatly appreciated.

Answered By: Adam Rosenfield ( 98)

One unit testing framework in C is Check; a list of unit testing frameworks in C can be found here. Depending on how many standard library functions your runtime has, you may or not be able to use one of those.

What frameworks exist to unit test Objective-C code? I would like a framework that integrates nicely with Xcode.

Answered By: Chris Hanson ( 220)

Xcode includes OCUnit, an Objective-C unit testing framework, and support for running unit tests (OCUnit or otherwise) as part of your project's build process. Xcode's unit testing support is described in the Xcode Unit Testing Guide.

I've written a series of weblog posts about how to perform some common tasks with Xcode unit testing:

Finally, I've written a few posts on how to write tests for Cocoa user interfaces; the way Cocoa is structured makes it relatively straightforward, because you don't have to spin an event loop or anything like that in most cases.

This makes it possible to do test-driven development for not just your model-level code but also your controller-level and even view-level code.

Hot-on-the-heels of of my previous unit testing related question, here's another toughie:

I have thus far avoided the nightmare that is testing multi-threaded code since it just seems like too much of a minefield. I'd like to ask how people have gone about testing code that relies on threads for successful execution, or just how people have gone about testing those kinds of issues that only show up when two threads interact in a given manner?

This seems like a really key problem for programmers today, it would be useful to pool our knowledge on this one imho.

Answered By: Will ( 82)

Look, there's no easy way to do this. I'm working on a project that is inherently multithreaded. Events come in from the operating system and I have to process them concurrently.

The simplest way to deal with testing complex, multithhreaded application code is this: If its too complex to test, you're doing it wrong. If you have a single instance that has multiple threads acting upon it, and you can't test situations where these threads step all over each other, then your design needs to be redone. Its both as simple and as complex as this.

There are many ways to program for multithreading that avoids threads running through instances at the same time. The simplest is to make all your objects immutable. Of course, that's not usually possible. So you have to identify those places in your design where threads interract with the same instance and reduce the number of those places. By doing this, you isolate a few classes where multithreading actually occurs, reducing the overall complexity of testing your system.

But you have to realize that even by doing this you still can't test every situation where two threads step on each other. To do that, you'd have to run two threads concurrently in the same test, then control exactly what lines they are executing at any given moment. The best you can do is simulate this situation. But this might require you to code specifically for testing, and that's at best a half step towards a true solution.

Probably the best way to test code for threading issues is through static analysis of the code. If your threaded code doesn't follow a finite set of thread safe patterns, then you might have a problem. I believe Code Analysis in VS does contain some knowledge of threading, but probably not much.

Look, as things stand currently (and probably will stand for a good time to come), the best way to test multithreaded apps is to reduce the complexity of threaded code as much as possible. Minimize areas where threads interact, test as best as possible, and use code analysis to identify danger areas.

208
Josh Brown

What's the best framework for creating mock objects in Java? Why? What are the pros and cons of each framework?

Answered By: Brian Laframboise ( 172)

I've had good success using Mockito.

When I tried learning about JMock and EasyMock, I found the learning curve to be a bit steep (though maybe that's just me).

I like Mockito because of its simple and clean syntax that I was able to grasp pretty quickly. The minimal syntax is designed to support the common cases very well, although the few times I needed to do something more complicated I found what I wanted was supported and easy to grasp.

Here's an (abridged) example from the Mockito homepage:

import static org.mockito.Mockito.*;

List mockedList = mock(List.class);
mockedList.clear();
verify(mockedList).clear();

It doesn't get much simpler than that.

The only major downside I can think of is that it won't mock static methods.