TestNG versus JUnit4

Comparing JUnit 4 and TestNG 5.7

Excerpts from http://www.ibm.com/developerworks/java/library/j-cq08296/ by Andy Glover.

JUnit is geared more towards unit testing - testing an object class in isolation.
TestNG provides more features and flexibility to facilitate its use not only for unit but integration, regression, functional, acceptance testings etc.

1. The setup method (annotated with @BeforeClass) needs to static and public with JUnit 4 but thats not required by TestNG. Thus TestNG is more flexible of the two.

2. Dependency testing:
Unlike JUnit, TestNG welcomes test dependencies through the dependsOnMethods attribute of the Test annotation. With this handy feature, you can easily specify dependent methods, which will execute before a desired method. What's more, if the dependent method fails, then all subsequent tests will be skipped, not marked as failed.

In JUnit 4, you can specify test orders using fixtures but if one test A fails then a test B that depends on test A will also be marked as failed.

TestNG's trick of skipping, rather than failing, can really take the pressure off in large test suites. Rather than trying to figure out why 50 percent of the test suite failed, your team can concentrate on why 50 percent of it was skipped! Better yet, TestNG complements its dependency testing setup with a mechanism for rerunning only failed tests.

3. Fail and rerun:
The ability to rerun failed tests is especially handy in large test suites, and it's a feature you'll only find in TestNG. In JUnit 4, if your test suite consists of 1000 tests and 3 of them fail, you'll likely be forced to rerun the entire suite (with fixes). Needless to say, this sort of thing can take hours.

Anytime there is a failure in TestNG, it creates an XML configuration file (testng-failed.xml) that delineates the failed tests. Running a TestNG runner with this file causes TestNG to only run the failed tests. So, in the previous example, you would only have to rerun the three failed tests and not the whole suite.

This feature doesn't seem like such a big deal when you're running smaller test suites, but you quickly come to appreciate it as your test suites grow in size.

4. Parametric testing:
By placing parametric data in TestNG's XML configuration files, you can reuse a single test case with different data sets and even get different results. This technique is perfect for avoiding tests that only assert sunny-day scenarios or don't effectively verify bounds.

JUnit testers often turn to a framework like FIT in this case because it lets you drive tests with tabular data. But TestNG provides a similar feature right out of the box.

This feature not only facilitates reuse of the test case code but also allows non-programmers to specify test data (since test data is in xml file).

public class TestWebServer {
  @Test(parameters = { "number-of-times" })
  public void accessPage(int numberOfTimes) {
    while (numberOfTimes-- > 0) {
     // access the web page
    }
  }
}

5. Advanced Parametric testing:
While pulling data values into an XML file can be quite handy, tests occasionally require complex types, which can't be represented as a String or a primitive value. TestNG handles this scenario with its @DataProvider annotation, which facilitates the mapping of complex parameter types to a test method.

Example:

//This method will provide data to any test method that declares that its Data Provider
//is named "test1"
@DataProvider(name = "test1")
public Object[][] createData1() {
 return new Object[][] {
   { "Cedric", new Integer(36) },
   { "Anne", new Integer(37)}, 
 };
}

//This test method declares that its data should be supplied by the Data Provider
//named "test1"
@Test(dataProvider = "test1")
public void verifyData1(String n1, Integer n2) {
 System.out.println(n1 + " " + n2);
} 

6. Groups:

You can define groups at the class level and then add groups at the method level. You can also specify groups and methods to be included and excluded.

@Test(groups = { "checkin-test" })
public class All {

  @Test(groups = { "func-test" )
  public void method1() { ... }

  public void method2() { ... }
}

and then in testng.xml:

<test name="Simple example">
  <groups>
    <run>
      <include name="checkin-test"/>
      <exclude name="broken"/>
    </run>
  </groups>
 
  <classes>
    <class name="example1.Test1">
    <methods>
       <include name="testMethod" />
    </methods>
  </classes>
</test>