Monday, December 03, 2007

TestNG versus JUnit4

Comparing JUnit 4 and TestNG 5.7

Excerpts from by Andy Glover.

JUnit is geared more towards unit testing - testing an object class in isolation.
TestNG provides more features and flexibility to facilitate its use not only for unit but integration, regression, functional, acceptance testings etc.

1. The setup method (annotated with @BeforeClass) needs to static and public with JUnit 4 but thats not required by TestNG. Thus TestNG is more flexible of the two.

2. Dependency testing:
Unlike JUnit, TestNG welcomes test dependencies through the dependsOnMethods attribute of the Test annotation. With this handy feature, you can easily specify dependent methods, which will execute before a desired method. What's more, if the dependent method fails, then all subsequent tests will be skipped, not marked as failed.

In JUnit 4, you can specify test orders using fixtures but if one test A fails then a test B that depends on test A will also be marked as failed.

TestNG's trick of skipping, rather than failing, can really take the pressure off in large test suites. Rather than trying to figure out why 50 percent of the test suite failed, your team can concentrate on why 50 percent of it was skipped! Better yet, TestNG complements its dependency testing setup with a mechanism for rerunning only failed tests.

3. Fail and rerun:
The ability to rerun failed tests is especially handy in large test suites, and it's a feature you'll only find in TestNG. In JUnit 4, if your test suite consists of 1000 tests and 3 of them fail, you'll likely be forced to rerun the entire suite (with fixes). Needless to say, this sort of thing can take hours.

Anytime there is a failure in TestNG, it creates an XML configuration file (testng-failed.xml) that delineates the failed tests. Running a TestNG runner with this file causes TestNG to only run the failed tests. So, in the previous example, you would only have to rerun the three failed tests and not the whole suite.

This feature doesn't seem like such a big deal when you're running smaller test suites, but you quickly come to appreciate it as your test suites grow in size.

4. Parametric testing:
By placing parametric data in TestNG's XML configuration files, you can reuse a single test case with different data sets and even get different results. This technique is perfect for avoiding tests that only assert sunny-day scenarios or don't effectively verify bounds.

JUnit testers often turn to a framework like FIT in this case because it lets you drive tests with tabular data. But TestNG provides a similar feature right out of the box.

This feature not only facilitates reuse of the test case code but also allows non-programmers to specify test data (since test data is in xml file).

public class TestWebServer {
@Test(parameters = { "number-of-times" })
public void accessPage(int numberOfTimes) {
while (numberOfTimes-- > 0) {
// access the web page

5. Advanced Parametric testing:
While pulling data values into an XML file can be quite handy, tests occasionally require complex types, which can't be represented as a String or a primitive value. TestNG handles this scenario with its @DataProvider annotation, which facilitates the mapping of complex parameter types to a test method.


//This method will provide data to any test method that declares that its Data Provider
//is named "test1"
@DataProvider(name = "test1")
public Object[][] createData1() {
return new Object[][] {
{ "Cedric", new Integer(36) },
{ "Anne", new Integer(37)},

//This test method declares that its data should be supplied by the Data Provider
//named "test1"
@Test(dataProvider = "test1")
public void verifyData1(String n1, Integer n2) {
System.out.println(n1 + " " + n2);

6. Groups:

You can define groups at the class level and then add groups at the method level. You can also specify groups and methods to be included and excluded.

@Test(groups = { "checkin-test" })
public class All {

@Test(groups = { "func-test" )
public void method1() { ... }

public void method2() { ... }

and then in testng.xml:

<test name="Simple example">
<include name="checkin-test"/>
<exclude name="broken"/>

<class name="example1.Test1">
<include name="testMethod" />

Saturday, December 01, 2007

TestNG - java testing framework

Recently i got introduced to TestNG (version 5.7) at work. I was familiar to JUnit from the past and i kind of knew about the existance of TestNG and that it had improvements over JUnit but i never thought that it will gain so much traction that i will be made to use it soon. Here are some of the features:
  • JDK 5 Annotations (JDK 1.4 is also supported with JavaDoc annotations).
  • Flexible test configuration - using multiple testng XML configuration files one per test suite.
  • Support for data-driven testing (with @DataProvider).
  • Support for parameters - you can pass parameters to test methods from the testng.xml file.
  • Allows distribution of tests on slave machines - support for parallel execution of tests and methods.
  • Powerful execution model (no more TestSuite) - test classes are annotated POJOs and don't have to extend any class or implement interface to have test methods.
  • Supported by a variety of tools and plug-ins (Eclipse, IDEA, Ant, Maven, etc...).
  • Embeds BeanShell for further flexibility.
  • Default JDK functions for runtime and logging (no dependencies).
  • Dependent methods for application server testing. - one can specify the dependsOnMethods attribute to the @Test annotation to specify a list of methods that should execute before a certain test method executes. This is a powerful feature and is required for any kind of dependent testing. If a dependent method fails, then all subsequent tests will be skipped, not marked as failed (unlike JUnit).
A good article stating improvements in TestNG over JUnit 4 is found at

I used TestNG today for the first time and found the framework very easy to use and within a day i had it integrated into our build system and made a presentation to the team about its usage in our project. In this post, i am detailing the steps i performed to start using TestNG:

1. Wrote a class using just the 3 basic TestNG annotations to start with:
  • @BeforeClass
  • @Test (groups = {"xyz.groupname"}) - at class level which gets inherited by all public methods in the class.
  • @AfterClass
See for complete list of supported annotations.

2. Wrote a master testng.xml which included all project suite-files and was referenced from the ant build script. Also wrote a testng-regression.xml which was imported in the master testng.xml. In the testng-regression.xml, defined the test runs in the suite. Each such testng-xxx.xml file is for xxx named test suite. Each suite can have one or more test runs. Each test runs identifies the class(es) or package(s) to lookup for annotated test methods. Each test run also identifies filter criteria based on groups to include and exclude in the test run. See for more on testng.xml.

One powerful feature i found was the group names could be specified in dot separated (java package name like) notations and follow a hierarchy akin to the Log4j Logger naming hierarchy. So you can use wildcards in the testng.xml to not only include all classes of a group but also include classes from child groups. For example, i could just say xyz.* to include and xyz.def group classes.

3. Lastly, used the ant build file to call the testng ant task and pass the testng.xml location to it so that testng can execute the tests we wanted. We can have multiple targets defined for different types of tests that we may want to automate. See for examples.

In the latest releases of JUnit 4, it too uses JDK5 annotations and thus makes up for some of the shortcomings that led Cedric Beust to develop TestNG framework.

If you have not had a chance to explore TestNG so far, then i hope after reading this post you will have the good sense to do so now :).

15 sorting algorithms visualized in 5 minutes, with awesome arcade sounds

15 sorting algorithms visualized in 5 minutes, with awesome arcade sounds from r/programming