> home > support > FAQ > taskfarm

Task Farming on HPCx

I have written a very straightforward harness called taskfarm, currently in /usr/local/packages/bin, that tries to help people do simple task farming of serial codes. It's basically nothing more than a wrapper MPI program that calls system(...) to run the user's code, but hopefully the interface is flexible enough to be used in most cases.

Basically, in your LoadLeveler script you do something like:

taskfarm "echo hello from %d"

(quotes aren't necessary here). Each process executes the command with %d replaced by its own process id, so if you specify 3 CPUs in the your LoadLeveler script and run the above you should get output like:

hello from 0
hello from 1
hello from 2

In practice you want each executable to read different input files and write to different output, so you would do something like:

  taskfarm "cd rundir%d; serialjob < input > output.log"

  taskfarm "serialjob< input.%d > output.%d.log"

(here the quotes ARE necessary).

The only option is "-v" for verbose in case you want to debug, eg on 3 processes:

  taskfarm -v "echo hello from %d"

  taskfarm: called with 4 arguments: echo hello from %d
  taskfarm: process 0 executing "echo hello from 0"
  taskfarm: process 1 executing "echo hello from 1"
  taskfarm: process 2 executing "echo hello from 2"
  hello from 0
  hello from 1
  hello from 2
  taskfarm: return code on process 0 is 0
  taskfarm: return code on process 1 is 0
  taskfarm: return code on process 2 is 0

I hope that this is useful!

David Henty

http://www.hpcx.ac.uk/support/FAQ/taskfarm.html

contact email - www@hpcx.ac.uk