Batchlib

The starting point for the batchlib module is that you have a script that must be run many times for different parameter values. For example, you want to know the result of a function evaluation for a large number of input values.

To solve this problem, the script must be executed a large number of times, each time with different input values. If the execution time of the script is small and the number of evaluations is limited, this can be done sequentially, using for example, a for-loop. If the total execution time needed becomes too large, you want to distribute the different evaluations to different machines. The latter is known as batch scheduling.

Manually performing the distribution is quite difficult to do. At the one hand, you want to make optimal use of the available processing power, which means you want to be able to dynamically shift evaluations from one machine to another (faster) machine if the latter machine becomes idle. At the other hand, results of different evaluations must not get mixed up.

The problem gets worse if the evaluations you need to do are generated by another program, a situation that arises in solving optimisation problems. In that case, running evaluations by hand is practically not even feasible.

The batchlib module aims to provide a solution for the above problems. After customising the framework for your computations, it provides an easy to use interface for starting evaluations and retrieving the answers. The framework takes care of which evaluations need to be done, allocates them to available machines in the network, and makes results of evaluations available.

All text (this text, the example, and the reference documentation) is also included as postscript file in the module you can download.

Download

The batchlib module is available in source form as batchlib-1.3.tar.gz.

You also need a module to run a script at a remote machine. You can either use exec_proxy or use some other module.

January 2007: Development has ceased.