W2H: How to activate the Batch-Queue-Feature in W2H

Peter Ernst and Martin Senger
This page describes, what you have to do, if you want to activate the Batch-Queue-Feature. This feature is deactivated by default.

What does Batch-Queue-Feature mean?
If not enabled, whenever you press the Run-Button in the Application-Window, the job is started immediately (interatively). With batch queues enabled, you have the possibility to put this job into a batch queue (batch processing/load sharing).

After activating the Batch-Queue-Feature a selection of batch queues appears next to to Run-Button in the Application-Window, where users may select the appropriate queue. Before, the W2H admin (i.e. you)

This Batch-Queue-Feature has to be activated for each application for which you want to use it.

How to activate Batch?

  1. To be able to use batch queues, you must create two small scripts, wrapping the special command-line-options and envrionment variables necessary by your queuing system. These two scripts must be located in the same directroy with w2h.com (specified in Cfg.pl: $COMAPP):

    1. w2h.batch.add: This program puts the jobs into the queue.
    2. w2h.batch.rm: This program removes a job out of a queue.

    How to do those programs will be described later. Within this package there are examples for the following queueing programs: qsub/NQS, batch and at. Select one which is appropriate for your queueing system, rename it to w2h.batch.add (and a corresponding one to w2h.batch.rm), put them into the same directory with w2h.com, and change their access rights to make them executable.

  2. [Note:
    Before W2H 3.2 a different scheme how to specify QUEUEDEFAULTs was used. The old scheme is now deprecated. For those, who already have a configuration according to the old scheme, there is a small program (util.extract_queues_from_configs) that converts the old configurations to the new scheme.
    ]

    Create a list file, with the specifications what applications should use what queue. Example:

    ..
BLAST    QUEUEDEFAULT short    QUEUES none,short,long
FASTA    QUEUEDEFAULT long
BESTFIT  QUEUEDEFAULT none     QUEUES none,short,long

    Specify the location of this file in the Cfg.pl variable $QUEUES_DEFINITION_TABLE. W2H comes with an example table in the file application.queues (as a partner file of application.menu).

    If you have a queueing system like 'qsub/NQS', you can offer your users quite a list of queues. The specification of a QUEUEDEFAULT is required (otherwise no queues will be used for this application).

    Additionaly you can specify an individual list of queues for the individual applications.

    A special is the keyword none: If you don't add this special keyword to the list of QUEUES, running the application interactively is not possible (it is always in batch). You can use none as a QUEUEDEFAULT, too.

    In the example above, BLAST runs by default in the short queue, but the user may change this to none (i.e. interactively) or to long. FASTA always runs in the batch queue long (the user can't change this). BESTFIT usually doesn't run in a batch queue, but the user can choose to use short or long as well.

    Instead of specifying the same QUEUES list over and over again for every application, you may specify a default list of queues for all applications with no QUEUES entry in the list file (the QUEUEDEFAULT entry is still required). You can specify this default list in Cfg.pl: 

    $QUEUES = ("veryshort", "short", "long" "verylong", "FastA");

    The name of the queue will be passed to you by W2H in the w2h.batch.add-program.

  3. The w2h.batch.add-program has to do the following things:

    • Get the desired batch queue from the command line
    • Put a call to w2h.com.queue into the batch queue.
      The arguments for this call are on the command-line, too.
    • Write an identifier for this job on STDOUT. W2H will only store this identifier for the case, that the user wishes to remove the job from the queue. In this case W2H will call your 2nd script w2h.batch.rm with this identifier as input, so that you can remove this special job from the queue.
    • If there was an error, write a message to STDOUT.

    Please look into the provided examples for details:

    • bin/w2h.batch.add.qsub (the main example)
    • bin/w2h.batch.add.batch
    • bin/w2h.batch.add.at

  4. The w2h.batch.rm program is more simple. It gets the batch-job-ID from the w2h.batch.add program on its command-line, and it just has to remove this job from the queue

    There are examples, too, for this file:

    • bin/w2h.batch.rm.qsub
    • bin/w2h.batch.rm.batch
    • bin/w2h.batch.rm.at

  5. Finally, you have to activate $QUEUES_ENABLED in Cfg.pl.

If it is not working:

  1. look for error messages in the .log file (either from the Results Manager in W2H or directly in the ~/.w2h-www directory)
  2. look for error messages in the httpd server log
p.ernst@dkfz-heidelberg.de
Last modified: Thu Dec 28 17:57:38 2000