Thursday, June 3, 2010

Making JUnit tests run parallely

If you use Selenium tests or other JUnit based tests with long execution time and low CPU usage you will eventually try to make them run concurrently. The good thing is that this is extremely easy thing to do with Apache Ant.

That why I am extremely surprised nobody still does it. Most people try to invent the wheel and try to use other frameworks like TestNG, write new experimental parallel computer classes for JUnit, or even configurable versions of the same classes. This all is a time waste for the people who just want to run some concurrent tests.

Here is how you do that:

1. Make sure that you have "ant-contrib" package installed. Usually it is enough if the appropriate JAR file is in the ant class path.

2. Turn the tasks of "ant-contrib" library. You can do this by adding the following lines to your "build.xml" file.
<taskdef
  resource="net/sf/antcontrib/antcontrib.properties"
/>
3. Add the new target that will execute exactly one test. You will have something like following.
<target name="execute.test">

  <!-- we need to have relative path -->
  <pathconvert property="test.source.relative">
    <fileset file="${test.source.absolute}" />
    <map from="${test.sources}/" to="" />
  </pathconvert>

  <!-- run one particular test -->
  <junit fork="true" printsummary="true">

    <classpath>
      <path refid="${test.classes}" />
    </classpath>

    <formatter type="xml" />

    <batchtest todir="${test.reports}">
      <fileset dir="${test.sources}">
        <filename name="${test.source.relative}" />
      </fileset>
    </batchtest>
  </junit>

</target>
4. Now use built-in "foreach" concurrent execution capabilities to execute this target. You would have to add something like following to your other target.
<foreach
target="execute.test"
maxthreads="5"
inheritall="true"
inheritrefs="true"
parallel="true"
param="test.source.absolute">
  <path>
    <fileset dir="${test.sources}">
      <include name="**/*.java"/>
    </fileset>
  </path>
</foreach>
5. That's all. Easy and nice. You will have all your tests executing in 5 nice threads. You can use any number of threads you want. No additional configuration, classes, code is needed.

9 comments:

  1. I agree, your solution is quite elegant. Nice idea. I like it.

    But what about forking? If you need tests forked you can't use fork="once" because each test is a batch test.

    Forking might be neccessary if you have many tests to avoid PermSpace issues. If you disable class loading for each test (and thus keeping the PermSpace from filling up), you might still run into problems with the much hated singletons.

    ReplyDelete
  2. Actually it seems to work properly even without fork="true". I only enabled it, because all my tests are truly independent.

    Of course nothing will save you from singletons automatically. If the tests are not thread safe and/or independent they will need to be rewritten.

    ReplyDelete
  3. The best thing about my idea that it provides ideal balancing. If one thread is exhausted it will take more job to do.

    ReplyDelete
  4. Very nice! Unfortunately in our case we can't apply a pattern matching to distinguish between test classes and utility classes in the test directory.
    Furthermore, I would like to guarantee random shuffling of test execution order and thread groupings.

    ReplyDelete
  5. You can use JUnit 4 annotations instead of pattern matching.

    The jobs indeed will be given to the thread which is least busy, but of course this will not guarantee totally random behavior.

    For that you could write your own two line extension to ForEach task that will randomize test execution. Or you could provide the list of the tests in already randomized order.

    There is one important problems in the solution you give in your blog.

    Tests are not balanced between threads in run-time. That means if one test is stuck somewhere - you will get all the thread waiting for it to finish while other threads will be doing nothing. This negates huge part of the advantages given by the concurrent execution.

    ReplyDelete
    Replies
    1. THe post is really helpful and I implemented it for my project. However I have one question about how the failure conditions can be handled. I want my build to fail if any of the test cases which run in parallel fails.

      I tried using fail tag and failureProperty in the junit task. I am able to successfully log the message but I need to fail the build in that case.

      Thanks,
      Saurabh V.

      Delete
    2. This comment has been removed by the author.

      Delete
  6. Hi,

    I assume this is to run all tests parallely. what if we need to run different test classes parallely and the tests in each class run sequentially( not in order )?

    Thanks

    ReplyDelete
  7. I like this, but I don't think the haltonfailure="yes" works with this approach.

    ReplyDelete