jobs Package

jobs Package

Support for running a tool in Galaxy via an internal job management system

class galaxy.jobs.ComputeEnvironment

Bases: object

Definition of the job as it will be run on the (potentially) remote compute server.

config_directory()

Directory containing config files (potentially remote)

input_paths()

Input DatasetPaths defined by job.

new_file_path()

Absolute path to dump new files for this job on compute server.

output_paths()

Output DatasetPaths defined by job.

sep()

os.path.sep for the platform this job will execute in.

tool_directory()

Absolute path to tool files for this job on compute server.

unstructured_path_rewriter()

Return a function that takes in a value, determines if it is path to be rewritten (will be passed non-path values as well - onus is on this function to determine both if its input is a path and if it should be rewritten.)

version_path()

Location of the version file for the underlying tool.

working_directory()

Job working directory (potentially remote)

class galaxy.jobs.JobConfiguration(app)

Bases: object

A parser and interface to advanced job management features.

These features are configured in the job configuration, by default, job_conf.xml

DEFAULT_NWORKERS = 4

convert_legacy_destinations(job_runners)

Converts legacy (from a URL) destinations to contain the appropriate runner params defined in the URL.

Parameters:job_runners (list of job runner plugins) – All loaded job runner plugins.

default_job_tool_configuration

The default JobToolConfiguration, used if a tool does not have an explicit defintion in the configuration. It consists of a reference to the default handler and default destination.

Returns:JobToolConfiguration – a representation of a <tool> element that uses the default handler and destination

get_destination(id_or_tag)

Given a destination ID or tag, return the JobDestination matching the provided ID or tag

Parameters:id_or_tag (str) – A destination ID or tag. Returns:JobDestination – A valid destination

Destinations are deepcopied as they are expected to be passed in to job runners, which will modify them for persisting params set at runtime.

get_destinations(id_or_tag)

Given a destination ID or tag, return all JobDestinations matching the provided ID or tag

Parameters:id_or_tag (str) – A destination ID or tag. Returns:list or tuple of JobDestinations

Destinations are not deepcopied, so they should not be passed to anything which might modify them.

get_handler(id_or_tag)

Given a handler ID or tag, return the provided ID or an ID matching the provided tag

Parameters:id_or_tag (str) – A handler ID or tag. Returns:str – A valid job handler ID.

get_job_runner_plugins(handler_id)

Load all configured job runner plugins

Returns:list of job runner plugins

get_job_tool_configurations(ids)

Get all configured JobToolConfigurations for a tool ID, or, if given a list of IDs, the JobToolConfigurations for the first id in ids matching a tool definition.

Note

You should not mix tool shed tool IDs, versionless tool shed IDs, and tool config tool IDs that refer to the same tool.

Parameters:ids (list or str.) – Tool ID or IDs to fetch the JobToolConfiguration of. Returns:list – JobToolConfiguration Bunches representing <tool> elements matching the specified ID(s).

Example tool ID strings include:

• Full tool shed id: toolshed.example.org/repos/nate/filter_tool_repo/filter_tool/1.0.0
• Tool shed id less version: toolshed.example.org/repos/nate/filter_tool_repo/filter_tool
• Tool config tool id: filter_tool

get_tool_resource_parameters(tool_id)

Given a tool id, return XML elements describing parameters to insert into job resources.

Tool id:A tool ID (a string) Returns:List of parameter elements.

is_handler(server_name)

Given a server name, indicate whether the server is a job handler

Parameters:server_name (str) – The name to check Returns:bool

is_id(collection)

Given a collection of handlers or destinations, indicate whether the collection represents a tag or a real ID

Parameters:collection (tuple or list) – A representation of a destination or handler Returns:bool

is_tag(collection)

Given a collection of handlers or destinations, indicate whether the collection represents a tag or a real ID

Parameters:collection (tuple or list) – A representation of a destination or handler Returns:bool

class galaxy.jobs.JobDestination(**kwds)

Bases: galaxy.util.bunch.Bunch

Provides details about where a job runs

class galaxy.jobs.JobToolConfiguration(**kwds)

Bases: galaxy.util.bunch.Bunch

Provides details on what handler and destination a tool should use

A JobToolConfiguration will have the required attribute ‘id’ and optional attributes ‘handler’, ‘destination’, and ‘params’

get_resource_group()

class galaxy.jobs.JobWrapper(job, queue, use_persisted_destination=False)

Bases: object

Wraps a ‘model.Job’ with convenience methods for running processes and state management.

can_split()

change_ownership_for_run()

change_state(state, info=False)

check_limits(runtime=None)

check_tool_output(stdout, stderr, tool_exit_code, job)

cleanup(delete_files=True)

clear_working_directory()

commands_in_new_shell

compute_outputs()

create_working_directory()

default_compute_environment(job=None)

fail(message, exception=False, stdout='', stderr='', exit_code=None)

Indicate job failure by setting state and message on all output datasets.

finish(stdout, stderr, tool_exit_code=None, remote_working_directory=None)

Called to indicate that the associated command has been run. Updates the output datasets based on stderr and stdout from the command, and the contents of the output files.

galaxy_lib_dir

galaxy_system_pwent

get_command_line()

get_dataset_finish_context(job_context, dataset)

get_env_setup_clause()

get_id_tag()

get_input_dataset_fnames(ds)

get_input_fnames()

get_input_paths(job=None)

get_job()

get_job_runner()

get_job_runner_url()

get_mutable_output_fnames()

get_output_destination(output_path)

Destination for outputs marked as from_work_dir. This is the normal case, just copy these files directly to the ulimate destination.

get_output_file_id(file)

get_output_fnames()

get_output_hdas_and_fnames()

get_output_sizes()

get_parallelism()

get_param_dict()

Restore the dictionary of parameters from the database.

get_session_id()

get_state()

get_tool_provided_job_metadata()

get_version_string_path()

has_limits()

invalidate_external_metadata()

job_destination

Return the JobDestination that this job will use to run. This will either be a configured destination, a randomly selected destination if the configured destination was a tag, or a dynamically generated destination from the dynamic runner.

Calling this method for the first time causes the dynamic runner to do its calculation, if any.

Returns:JobDestination

mark_as_resubmitted(info=None)

pause(job=None, message=None)

prepare(compute_environment=None)

Prepare the job to run by creating the working directory and the config files.

reclaim_ownership()

requires_setting_metadata

set_job_destination(job_destination, external_id=None)

Persist job destination params in the database for recovery.

self.job_destination is not used because a runner may choose to rewrite parts of the destination (e.g. the params).

set_runner(runner_url, external_id)

setup_external_metadata(exec_dir=None, tmp_dir=None, dataset_files_path=None, config_root=None, config_file=None, datatypes_config=None, set_extension=True, **kwds)

user

user_system_pwent

class galaxy.jobs.NoopQueue

Bases: object

Implements the JobQueue / JobStopQueue interface but does nothing

put(*args, **kwargs)

put_stop(*args)

shutdown()

class galaxy.jobs.ParallelismInfo(tag)

Bases: object

Stores the information (if any) for running multiple instances of the tool in parallel on the same set of inputs.

class galaxy.jobs.SharedComputeEnvironment(job_wrapper, job)

Bases: galaxy.jobs.SimpleComputeEnvironment

Default ComputeEnviornment for job and task wrapper to pass to ToolEvaluator - valid when Galaxy and compute share all the relevant file systems.

input_paths()

new_file_path()

output_paths()

tool_directory()

version_path()

working_directory()

class galaxy.jobs.SimpleComputeEnvironment

Bases: object

config_directory()

sep()

unstructured_path_rewriter()

class galaxy.jobs.TaskWrapper(task, queue)

Bases: galaxy.jobs.JobWrapper

Extension of JobWrapper intended for running tasks. Should be refactored into a generalized executable unit wrapper parent, then jobs and tasks.

can_split()

change_state(state, info=False)

cleanup(delete_files=True)

fail(message, exception=False)

finish(stdout, stderr, tool_exit_code=None)

Called to indicate that the associated command has been run. Updates the output datasets based on stderr and stdout from the command, and the contents of the output files.

get_command_line()

get_dataset_finish_context(job_context, dataset)

get_exit_code()

get_id_tag()

get_job()

get_output_destination(output_path)

Destination for outputs marked as from_work_dir. These must be copied with the same basenme as the path for the ultimate output destination. This is required in the task case so they can be merged.

get_output_file_id(file)

get_param_dict()

Restore the dictionary of parameters from the database.

get_session_id()

get_state()

get_task()

get_tool_provided_job_metadata()

prepare(compute_environment=None)

Prepare the job to run by creating the working directory and the config files.

set_runner(runner_url, external_id)

setup_external_metadata(exec_dir=None, tmp_dir=None, dataset_files_path=None, config_root=None, config_file=None, datatypes_config=None, set_extension=True, **kwds)

galaxy.jobs.config_exception(e, file)

handler Module

Galaxy job handler, prepares, runs, tracks, and finishes Galaxy jobs

class galaxy.jobs.handler.DefaultJobDispatcher(app)

Bases: object

put(job_wrapper)

recover(job, job_wrapper)

shutdown()

stop(job)

Stop the given job. The input variable job may be either a Job or a Task.

url_to_destination(url)

This is used by the runner mapper (a.k.a. dynamic runner) and recovery methods to have runners convert URLs to destinations.

New-style runner plugin IDs must match the URL’s scheme for this to work.

class galaxy.jobs.handler.JobHandler(app)

Bases: object

Handle the preparation, running, tracking, and finishing of jobs

shutdown()

start()

class galaxy.jobs.handler.JobHandlerQueue(app, dispatcher)

Bases: object

Job Handler’s Internal Queue, this is what actually implements waiting for jobs to be runnable and dispatching to a JobRunner.

STOP_SIGNAL = <object object>

get_total_job_count_per_destination()

get_user_job_count(user_id)

get_user_job_count_per_destination(user_id)

increase_running_job_count(user_id, destination_id)

job_pair_for_id(id)

job_wrapper(job, use_persisted_destination=False)

put(job_id, tool_id)

Add a job to the queue (by job identifier)

shutdown()

Attempts to gracefully shut down the worker thread

start()

Starts the JobHandler’s thread after checking for any unhandled jobs.

class galaxy.jobs.handler.JobHandlerStopQueue(app, dispatcher)

Bases: object

A queue for jobs which need to be terminated prematurely.

STOP_SIGNAL = <object object>

monitor()

Continually iterate the waiting jobs, stop any that are found.

monitor_step()

Called repeatedly by monitor to stop jobs.

put(job_id, error_msg=None)

shutdown()

Attempts to gracefully shut down the worker thread

manager Module

Top-level Galaxy job manager, moves jobs to handler(s)

class galaxy.jobs.manager.JobManager(app)

Bases: object

Highest level interface to job management.

TODO: Currently the app accesses “job_queue” and “job_stop_queue” directly.

This should be decoupled.

shutdown()

start()

class galaxy.jobs.manager.NoopHandler(*args, **kwargs)

Bases: object

shutdown(*args)

start()

mapper Module

exception galaxy.jobs.mapper.JobMappingException(failure_message)

Bases: exceptions.Exception

exception galaxy.jobs.mapper.JobNotReadyException(job_state=None, message=None)

Bases: exceptions.Exception

class galaxy.jobs.mapper.JobRunnerMapper(job_wrapper, url_to_destination, job_config)

Bases: object

This class is responsible to managing the mapping of jobs (in the form of job_wrappers) to job runner url strings.

cache_job_destination(raw_job_destination)

get_job_destination(params)

Cache the job_destination to avoid recalculation.

transfer_manager Module

Manage transfers from arbitrary URLs to temporary files. Socket interface for IPC with multiple process configurations.

class galaxy.jobs.transfer_manager.Sleeper

Bases: object

Provides a ‘sleep’ method that sleeps for a number of seconds unless the notify method is called (from a different thread).

sleep(seconds)

wake()

class galaxy.jobs.transfer_manager.TransferManager(app)

Bases: object

Manage simple data transfers from URLs to temporary locations.

get_state(transfer_jobs, via_socket=False)

new(path=None, **kwd)

run(transfer_jobs)

This method blocks, so if invoking the transfer manager ever starts taking too long, we should move it to a thread. However, the transfer_manager will either daemonize or return after submitting to a running daemon, so it should be fairly quick to return.

shutdown()

Subpackages

• actions Package
  • actions Package
  • post Module
• deferred Package
  • deferred Package
  • data_transfer Module
  • genome_index Module
  • genome_transfer Module
  • liftover_transfer Module
  • manual_data_transfer Module
  • pacific_biosciences_smrt_portal Module
• runners Package
  • runners Package
  • cli Module
  • condor Module
  • drmaa Module
  • local Module
  • lwr Module
  • pbs Module
  • tasks Module
  • Subpackages
    • cli_job Package
      • cli_job Package
      • torque Module
    • cli_shell Package
      • cli_shell Package
      • rsh Module
• splitters Package
  • basic Module
  • multi Module