Limitations of JBench

JBench is best used on an idle machine. This is because as far as I know, there is no way in pure Java of measuring the time spent by the JVM in a thread, as opposed to the real time spent performing an operation. This means that if you run one benchmark on a very heavily loaded system, then another on the same system after the load has reduced, you may well see very different results. If you are running a JVM which supports the JVM profiler interface (JVMPI), such as JDK1.3, it's possible to do this with native code. See Using the CPU timer for details of how to run this.

Another potential limitation is that it is relatively ugly to create a benchmark to test an algorithm with a range of parameters - eg to find out how sorting time increases with data set size. It's perfectly feasible to do this by creating several task configurations with the same parameters other than size - but it would be nice to have some way of making that automatic, eg having
task.1.size = 10..10000 by *10
or
task.1.size = 10..100 by +10
The above syntax is only a very (very) quick idea of what might be feasible in the future. It may happen, but I wouldn't hold your breath. If you really want this, mail me with requirements and suggestions, and I'll see...

JBench Manual