runners Package

runners Package

Base classes for job runner plugins.

class galaxy.jobs.runners.AsynchronousJobRunner(app, nworkers, **kwargs)

Bases: galaxy.jobs.runners.BaseJobRunner

Parent class for any job runner that runs jobs asynchronously (e.g. via a distributed resource manager). Provides general methods for having a thread to monitor the state of asynchronous jobs and submitting those jobs to the correct methods (queue, finish, cleanup) at appropriate times..

check_watched_item(job_state)

check_watched_items()

This method is responsible for iterating over self.watched and handling state changes and updating self.watched with a new list of watched job states. Subclasses can opt to override this directly (as older job runners will initially) or just override check_watched_item and allow the list processing to reuse the logic here.

fail_job(job_state)

finish_job(job_state)

Get the output/error for a finished job, pass to job_wrapper.finish and cleanup all the job’s temporary files.

handle_stop()

mark_as_failed(job_state)

mark_as_finished(job_state)

monitor()

Watches jobs currently in the monitor queue and deals with state changes (queued to running) and job completion.

monitor_job(job_state)

shutdown()

Attempts to gracefully shut down the monitor thread

class galaxy.jobs.runners.AsynchronousJobState(files_dir=None, job_wrapper=None, job_id=None, job_file=None, output_file=None, error_file=None, exit_code_file=None, job_name=None, job_destination=None)

Bases: galaxy.jobs.runners.JobState

Encapsulate the state of an asynchronous job, this should be subclassed as needed for various job runners to capture additional information needed to communicate with distributed resource manager.

check_limits(runtime=None)

cleanup()

register_cleanup_file_attribute(attribute)

running

class galaxy.jobs.runners.BaseJobRunner(app, nworkers, **kwargs)

Bases: object

DEFAULT_SPECS = {'recheck_missing_job_retries': {'default': 0, 'map': <type 'int'>, 'valid': <function <lambda> at 0x7f85dd6d01b8>}}

build_command_line(job_wrapper, include_metadata=False, include_work_dir_outputs=True)

get_job_file(job_wrapper, **kwds)

get_work_dir_outputs(job_wrapper, job_working_directory=None)

Returns list of pairs (source_file, destination) describing path to work_dir output file and ultimate destination.

mark_as_queued(job_wrapper)

mark_as_resubmitted(job_state, info=None)

parse_destination_params(params)

Parse the JobDestination params dict and return the runner’s native representation of those params.

prepare_job(job_wrapper, include_metadata=False, include_work_dir_outputs=True)

Some sanity checks that all runners’ queue_job() methods are likely to want to do

put(job_wrapper)

Add a job to the queue (by job identifier), indicate that the job is ready to run.

queue_job(job_wrapper)

recover(job, job_wrapper)

run_next()

Run the next item in the work queue (a job waiting to run)

shutdown()

Attempts to gracefully shut down the worker threads

stop_job(job)

url_to_destination(url)

Convert a legacy URL to a JobDestination.

Job runner URLs are deprecated, JobDestinations should be used instead. This base class method converts from a URL to a very basic JobDestination without destination params.

class galaxy.jobs.runners.JobState

Bases: object

Encapsulate state of jobs.

static default_exit_code_file(files_dir, id_tag)

static default_job_file(files_dir, id_tag)

runner_states = <galaxy.util.bunch.Bunch object>

set_defaults(files_dir)

class galaxy.jobs.runners.RunnerParams(specs=None, params=None)

Bases: galaxy.util.ParamsWithSpecs

cli Module

Job control via a command line interface (e.g. qsub/qstat), possibly over a remote connection (e.g. ssh).

class galaxy.jobs.runners.cli.ShellJobRunner(app, nworkers)

Bases: galaxy.jobs.runners.AsynchronousJobRunner

Job runner backed by a finite pool of worker threads. FIFO scheduling

check_watched_items()

Called by the monitor thread to look at each watched job and deal with state changes.

finish_job(job_state)

For recovery of jobs started prior to standardizing the naming of files in the AsychronousJobState object

get_cli_plugins(shell_params, job_params)

parse_destination_params(params)

queue_job(job_wrapper)

Create job script and submit it to the DRM

recover(job, job_wrapper)

Recovers jobs stuck in the queued/running state when Galaxy started

runner_name = 'ShellRunner'

stop_job(job)

Attempts to delete a dispatched job

url_to_destination(url)

condor Module

Job control via the Condor DRM.

class galaxy.jobs.runners.condor.CondorJobRunner(app, nworkers)

Bases: galaxy.jobs.runners.AsynchronousJobRunner

Job runner backed by a finite pool of worker threads. FIFO scheduling

check_watched_items()

Called by the monitor thread to look at each watched job and deal with state changes.

queue_job(job_wrapper)

Create job script and submit it to the DRM

recover(job, job_wrapper)

Recovers jobs stuck in the queued/running state when Galaxy started

runner_name = 'CondorRunner'

stop_job(job)

Attempts to delete a job from the DRM queue

drmaa Module

Job control via the DRMAA API.

class galaxy.jobs.runners.drmaa.DRMAAJobRunner(app, nworkers, **kwargs)

Bases: galaxy.jobs.runners.AsynchronousJobRunner

Job runner backed by a finite pool of worker threads. FIFO scheduling

check_watched_items()

Called by the monitor thread to look at each watched job and deal with state changes.

external_runjob(jobtemplate_filename, username)

runs an external script the will QSUB a new job. The external script will be run with sudo, and will setuid() to the specified user. Effectively, will QSUB as a different user (then the one used by Galaxy).

get_native_spec(url)

Get any native DRM arguments specified by the site configuration

queue_job(job_wrapper)

Create job script and submit it to the DRM

recover(job, job_wrapper)

Recovers jobs stuck in the queued/running state when Galaxy started

runner_name = 'DRMAARunner'

stop_job(job)

Attempts to delete a job from the DRM queue

store_jobtemplate(job_wrapper, jt)

Stores the content of a DRMAA JobTemplate object in a file as a JSON string. Path is hard-coded, but it’s no worse than other path in this module. Uses Galaxy’s JobID, so file is expected to be unique.

url_to_destination(url)

Convert a legacy URL to a job destination

local Module

Job runner plugin for executing jobs on the local system via the command line.

class galaxy.jobs.runners.local.LocalJobRunner(app, nworkers)

Bases: galaxy.jobs.runners.BaseJobRunner

Job runner backed by a finite pool of worker threads. FIFO scheduling

queue_job(job_wrapper)

recover(job, job_wrapper)

runner_name = 'LocalRunner'

stop_job(job)

lwr Module

class galaxy.jobs.runners.lwr.LwrJobRunner(app, nworkers, **kwds)

Bases: galaxy.jobs.runners.AsynchronousJobRunner

LWR Job Runner

check_pid(pid)

check_watched_item(job_state)

fail_job(job_state)

Seperated out so we can use the worker threads for it.

finish_job(job_state)

get_client(job_destination_params, job_id, env=[])

get_client_from_state(job_state)

get_client_from_wrapper(job_wrapper)

get_input_files(job_wrapper)

get_output_files(job_wrapper)

queue_job(job_wrapper)

recover(job, job_wrapper)

Recovers jobs stuck in the queued/running state when Galaxy started

runner_name = 'LWRRunner'

shutdown()

stop_job(job)

url_to_destination(url)

Convert a legacy URL to a job destination

pbs Module

tasks Module

class galaxy.jobs.runners.tasks.TaskedJobRunner(app, nworkers)

Bases: galaxy.jobs.runners.BaseJobRunner

Job runner backed by a finite pool of worker threads. FIFO scheduling

queue_job(job_wrapper)

recover(job, job_wrapper)

runner_name = 'TaskRunner'

stop_job(job)

Subpackages

• cli_job Package
  • cli_job Package
  • torque Module
• cli_shell Package
  • cli_shell Package
  • rsh Module