util Package¶
util
Package¶
Utility functions used systemwide.
-
class
galaxy.util.
Params
(params, sanitize=True)[source]¶ Bases:
object
Stores and ‘sanitizes’ parameters. Alphanumeric characters and the non-alphanumeric ones that are deemed safe are let to pass through (see L{valid_chars}). Some non-safe characters are escaped to safe forms for example C{>} becomes C{__lt__} (see L{mapped_chars}). All other characters are replaced with C{X}.
Operates on string or list values only (HTTP parameters).
>>> values = { 'status':'on', 'symbols':[ 'alpha', '<>', '$rm&#!' ] } >>> par = Params(values) >>> par.status 'on' >>> par.value == None # missing attributes return None True >>> par.get('price', 0) 0 >>> par.symbols # replaces unknown symbols with X ['alpha', '__lt____gt__', 'XrmX__pd__!'] >>> sorted(par.flatten()) # flattening to a list [('status', 'on'), ('symbols', 'XrmX__pd__!'), ('symbols', '__lt____gt__'), ('symbols', 'alpha')]
-
NEVER_SANITIZE
= ['file_data', 'url_paste', 'URL', 'filesystem_paths']¶
-
-
galaxy.util.
compare_urls
(url1, url2, compare_scheme=True, compare_hostname=True, compare_path=True)[source]¶
-
galaxy.util.
docstring_trim
(docstring)[source]¶ Trimming python doc strings. Taken from: http://www.python.org/dev/peps/pep-0257/
-
galaxy.util.
file_iter
(fname, sep=None)[source]¶ This generator iterates over a file and yields its lines splitted via the C{sep} parameter. Skips empty lines and lines starting with the C{#} character.
>>> lines = [ line for line in file_iter(__file__) ] >>> len(lines) != 0 True
-
galaxy.util.
file_reader
(fp, chunk_size=65536)[source]¶ This generator yields the open fileobject in chunks (default 64k). Closes the file at the end
-
galaxy.util.
in_directory
(file, directory, local_path_module=<module 'posixpath' from '/home/docs/checkouts/readthedocs.org/user_builds/jmchilton-galaxy/envs/latest/lib/python2.7/posixpath.pyc'>)[source]¶ Return true, if the common prefix of both is equal to directory e.g. /a/b/c/d.rst and directory is /a/b, the common prefix is /a/b
-
galaxy.util.
is_binary
(value, binary_chars=None)[source]¶ File is binary if it contains a null-byte by default (e.g. behavior of grep, etc.). This may fail for utf-16 files, but so would ASCII encoding. >>> is_binary( string.printable ) False >>> is_binary( ‘xcex94’ ) False >>> is_binary( ‘000’ ) True
-
galaxy.util.
is_uuid
(value)[source]¶ This method returns True if value is a UUID, otherwise False. >>> is_uuid( “123e4567-e89b-12d3-a456-426655440000” ) True >>> is_uuid( “0x3242340298902834” ) False
-
galaxy.util.
listify
(item, do_strip=False)[source]¶ Make a single item a single item list, or return a list if passed a list. Passing a None returns an empty list.
-
galaxy.util.
mask_password_from_url
(url)[source]¶ Masks out passwords from connection urls like the database connection in galaxy.ini
>>> mask_password_from_url( 'sqlite+postgresql://user:password@localhost/' ) 'sqlite+postgresql://user:********@localhost/' >>> mask_password_from_url( 'amqp://user:amqp@localhost' ) 'amqp://user:********@localhost' >>> mask_password_from_url( 'amqp://localhost') 'amqp://localhost'
-
galaxy.util.
merge_sorted_iterables
(operator, *iterables)[source]¶ >>> operator = lambda x: x >>> list( merge_sorted_iterables( operator, [1,2,3], [4,5] ) ) [1, 2, 3, 4, 5] >>> list( merge_sorted_iterables( operator, [4, 5], [1,2,3] ) ) [1, 2, 3, 4, 5] >>> list( merge_sorted_iterables( operator, [1, 4, 5], [2], [3] ) ) [1, 2, 3, 4, 5]
-
galaxy.util.
mkstemp_ln
(src, prefix='mkstemp_ln_')[source]¶ From tempfile._mkstemp_inner, generate a hard link in the same dir with a random name. Created so we can persist the underlying file of a NamedTemporaryFile upon its closure.
-
galaxy.util.
nice_size
(size)[source]¶ Returns a readably formatted string with the size
>>> nice_size(100) '100 bytes' >>> nice_size(10000) '9.8 KB' >>> nice_size(1000000) '976.6 KB' >>> nice_size(100000000) '95.4 MB'
-
galaxy.util.
pretty_print_time_interval
(time=False, precise=False)[source]¶ Get a datetime object or a int() Epoch timestamp and return a pretty string like ‘an hour ago’, ‘Yesterday’, ‘3 months ago’, ‘just now’, etc credit: http://stackoverflow.com/questions/1551382/user-friendly-time-format-in-python
-
galaxy.util.
read_build_sites
(filename, check_builds=True)[source]¶ read db names to ucsc mappings from file, this file should probably be merged with the one above
-
galaxy.util.
ready_name_for_url
(raw_name)[source]¶ General method to convert a string (i.e. object name) to a URL-ready slug.
>>> ready_name_for_url( "My Cool Object" ) 'My-Cool-Object' >>> ready_name_for_url( "!My Cool Object!" ) 'My-Cool-Object' >>> ready_name_for_url( "Hello₩◎ґʟⅾ" ) 'Hello'
-
galaxy.util.
restore_text
(text, character_map={'@': '__at__', '\t': '__tc__', '\n': '__cn__', '\r': '__cr__', '[': '__ob__', ']': '__cb__', '#': '__pd__', '"': '__dq__', "'": '__sq__', '{': '__oc__', '}': '__cc__', '<': '__lt__', '>': '__gt__'})[source]¶ Restores sanitized text
-
galaxy.util.
roundify
(amount, sfs=2)[source]¶ Take a number in string form and truncate to ‘sfs’ significant figures.
-
galaxy.util.
safe_str_cmp
(a, b)[source]¶ safely compare two strings in a timing-attack-resistant manner
-
galaxy.util.
sanitize_for_filename
(text, default=None)[source]¶ Restricts the characters that are allowed in a filename portion; Returns default value or a unique id string if result is not a valid name. Method is overly aggressive to minimize possible complications, but a maximum length is not considered.
-
galaxy.util.
sanitize_lists_to_string
(values, valid_characters=set(['!', ' ', ')', '(', '+', '*', '-', ', ', '/', '.', '1', '0', '3', '2', '5', '4', '7', '6', '9', '8', ':', '=', '?', 'A', 'C', 'B', 'E', 'D', 'G', 'F', 'I', 'H', 'K', 'J', 'M', 'L', 'O', 'N', 'Q', 'P', 'S', 'R', 'U', 'T', 'W', 'V', 'Y', 'X', 'Z', '_', '^', 'a', 'c', 'b', 'e', 'd', 'g', 'f', 'i', 'h', 'k', 'j', 'm', 'l', 'o', 'n', 'q', 'p', 's', 'r', 'u', 't', 'w', 'v', 'y', 'x', 'z']), character_map={'@': '__at__', '\t': '__tc__', '\n': '__cn__', '\r': '__cr__', '[': '__ob__', ']': '__cb__', '#': '__pd__', '"': '__dq__', "'": '__sq__', '{': '__oc__', '}': '__cc__', '<': '__lt__', '>': '__gt__'}, invalid_character='X')[source]¶
-
galaxy.util.
sanitize_param
(value, valid_characters=set(['!', ' ', ')', '(', '+', '*', '-', ', ', '/', '.', '1', '0', '3', '2', '5', '4', '7', '6', '9', '8', ':', '=', '?', 'A', 'C', 'B', 'E', 'D', 'G', 'F', 'I', 'H', 'K', 'J', 'M', 'L', 'O', 'N', 'Q', 'P', 'S', 'R', 'U', 'T', 'W', 'V', 'Y', 'X', 'Z', '_', '^', 'a', 'c', 'b', 'e', 'd', 'g', 'f', 'i', 'h', 'k', 'j', 'm', 'l', 'o', 'n', 'q', 'p', 's', 'r', 'u', 't', 'w', 'v', 'y', 'x', 'z']), character_map={'@': '__at__', '\t': '__tc__', '\n': '__cn__', '\r': '__cr__', '[': '__ob__', ']': '__cb__', '#': '__pd__', '"': '__dq__', "'": '__sq__', '{': '__oc__', '}': '__cc__', '<': '__lt__', '>': '__gt__'}, invalid_character='X')[source]¶ Clean incoming parameters (strings or lists)
-
galaxy.util.
sanitize_text
(text, valid_characters=set(['!', ' ', ')', '(', '+', '*', '-', ', ', '/', '.', '1', '0', '3', '2', '5', '4', '7', '6', '9', '8', ':', '=', '?', 'A', 'C', 'B', 'E', 'D', 'G', 'F', 'I', 'H', 'K', 'J', 'M', 'L', 'O', 'N', 'Q', 'P', 'S', 'R', 'U', 'T', 'W', 'V', 'Y', 'X', 'Z', '_', '^', 'a', 'c', 'b', 'e', 'd', 'g', 'f', 'i', 'h', 'k', 'j', 'm', 'l', 'o', 'n', 'q', 'p', 's', 'r', 'u', 't', 'w', 'v', 'y', 'x', 'z']), character_map={'@': '__at__', '\t': '__tc__', '\n': '__cn__', '\r': '__cr__', '[': '__ob__', ']': '__cb__', '#': '__pd__', '"': '__dq__', "'": '__sq__', '{': '__oc__', '}': '__cc__', '<': '__lt__', '>': '__gt__'}, invalid_character='X')[source]¶ Restricts the characters that are allowed in text; accepts both strings and lists of strings; non-string entities will be cast to strings.
-
galaxy.util.
shrink_stream_by_size
(value, size, join_by='..', left_larger=True, beginning_on_size_error=False, end_on_size_error=False)[source]¶
-
galaxy.util.
shrink_string_by_size
(value, size, join_by='..', left_larger=True, beginning_on_size_error=False, end_on_size_error=False)[source]¶
-
galaxy.util.
size_to_bytes
(size)[source]¶ Returns a number of bytes if given a reasonably formatted string with the size
-
galaxy.util.
smart_str
(s, encoding='utf-8', strings_only=False, errors='strict')[source]¶ Returns a bytestring version of ‘s’, encoded as specified in ‘encoding’.
If strings_only is True, don’t convert (some) non-string-like objects.
Adapted from an older, simpler version of django.utils.encoding.smart_str.
-
galaxy.util.
string_as_bool_or_none
(string)[source]¶ - Returns True, None or False based on the argument:
- True if passed True, ‘True’, ‘Yes’, or ‘On’ None if passed None or ‘None’ False otherwise
Note: string comparison is case-insensitive so lowecase versions of those function equivalently.
-
galaxy.util.
synchronized
(func)[source]¶ This wrapper will serialize access to ‘func’ to a single thread. Use it as a decorator.
-
galaxy.util.
umask_fix_perms
(path, umask, unmasked_perms, gid=None)[source]¶ umask-friendly permissions fixing
-
galaxy.util.
unicodify
(value, encoding='utf-8', error='replace', default=None)[source]¶ Returns a unicode string or None
aliaspickler
Module¶
bunch
Module¶
-
class
galaxy.util.bunch.
Bunch
(**kwds)[source]¶ Bases:
object
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52308
Often we want to just collect a bunch of stuff together, naming each item of the bunch; a dictionary’s OK for that, but a small do-nothing class is even handier, and prettier to use.
debugging
Module¶
-
class
galaxy.util.debugging.
SimpleProfiler
(log=None)[source]¶ Bases:
object
Simple profiler that captures the duration between calls to report and stores the results in a list.
-
REPORT_FORMAT
= '%20f: %s'¶
-
-
galaxy.util.debugging.
stack_trace_string
(max_depth=None, line_format='{index}:{file}:{function}:{line}')[source]¶ Returns a string representation of the current stack.
Parameters: depth – positive integer to control how many levels of the stack are returned. max_depth=None returns the entire stack (default).
expressions
Module¶
Expression evaluation support.
For the moment this depends on python’s eval. In the future it should be replaced with a “safe” parser.
hash_util
Module¶
Utility functions for bi-directional Python version compatibility. Python 2.5 introduced hashlib which replaced sha in Python 2.4 and previous versions.
heartbeat
Module¶
-
class
galaxy.util.heartbeat.
Heartbeat
(name='Heartbeat Thread', period=20, fname='heartbeat.log')[source]¶ Bases:
threading.Thread
Thread that periodically dumps the state of all threads to a file
-
galaxy.util.heartbeat.
get_current_thread_object_dict
()[source]¶ Get a dictionary of all ‘Thread’ objects created via the threading module keyed by thread_id. Note that not all interpreter threads have a thread objects, only the main thread and any created via the ‘threading’ module. Threads created via the low level ‘thread’ module will not be in the returned dictionary.
- HACK: This mucks with the internals of the threading module since that
- module does not expose any way to match ‘Thread’ objects with intepreter thread identifiers (though it should).
inflection
Module¶
-
class
galaxy.util.inflection.
Base
[source]¶ Locale inflectors must inherit from this base class inorder to provide the basic Inflector functionality
-
camelize
(word)[source]¶ Returns given word as CamelCased Converts a word like “send_email” to “SendEmail”. It will remove non alphanumeric character from the word, so “who’s online” will be converted to “WhoSOnline”
-
classify
(table_name)[source]¶ Converts a table name to its class name according to rails naming conventions. Example: Converts “people” to “Person”
-
cond_plural
(number_of_records, word)[source]¶ Returns the plural form of a word if first parameter is greater than 1
-
foreignKey
(class_name, separate_class_name_and_id_with_underscore=1)[source]¶ Returns class_name in underscored form, with “_id” tacked on at the end. This is for use in dealing with the database.
-
humanize
(word, uppercase='')[source]¶ Returns a human-readable string from word Returns a human-readable string from word, by replacing underscores with a space, and by upper-casing the initial character by default. If you need to uppercase all the words you just have to pass ‘all’ as a second parameter.
-
ordinalize
(number)[source]¶ Converts number to its ordinal English form. This method converts 13 to 13th, 2 to 2nd ...
-
string_replace
(word, find, replace)[source]¶ This function returns a copy of word, translating all occurrences of each character in find to the corresponding character in replace
-
tableize
(class_name)[source]¶ Converts a class name to its table name according to rails naming conventions. Example. Converts “Person” to “people”
-
titleize
(word, uppercase='')[source]¶ Converts an underscored or CamelCase word into a English sentence. The titleize function converts text like “WelcomePage”, “welcome_page” or “welcome page” to this “Welcome Page”. If second parameter is set to ‘first’ it will only capitalize the first character of the title.
-
unaccent
(text)[source]¶ Transforms a string to its unaccented version. This might be useful for generating “friendly” URLs
-
underscore
(word)[source]¶ Converts a word “into_it_s_underscored_version” Convert any “CamelCased” or “ordinary Word” into an “underscored_word”. This can be really useful for creating friendly URLs.
-
-
class
galaxy.util.inflection.
English
[source]¶ Bases:
galaxy.util.inflection.Base
Inflector for pluralize and singularize English nouns.
This is the default Inflector for the Inflector obj
-
class
galaxy.util.inflection.
Inflector
(Inflector=<class galaxy.util.inflection.English>)[source]¶ Inflector for pluralizing and singularizing nouns.
It provides methods for helping on creating programs based on naming conventions like on Ruby on Rails.
-
camelize
(word)[source]¶ Returns given word as CamelCased Converts a word like “send_email” to “SendEmail”. It will remove non alphanumeric character from the word, so “who’s online” will be converted to “WhoSOnline”
-
classify
(table_name)[source]¶ Converts a table name to its class name according to rails naming conventions. Example: Converts “people” to “Person”
-
cond_plural
(number_of_records, word)[source]¶ Returns the plural form of a word if first parameter is greater than 1
-
foreignKey
(class_name, separate_class_name_and_id_with_underscore=1)[source]¶ Returns class_name in underscored form, with “_id” tacked on at the end. This is for use in dealing with the database.
-
humanize
(word, uppercase='')[source]¶ Returns a human-readable string from word Returns a human-readable string from word, by replacing underscores with a space, and by upper-casing the initial character by default. If you need to uppercase all the words you just have to pass ‘all’ as a second parameter.
-
ordinalize
(number)[source]¶ Converts number to its ordinal form. This method converts 13 to 13th, 2 to 2nd ...
-
tableize
(class_name)[source]¶ Converts a class name to its table name according to rails naming conventions. Example. Converts “Person” to “people”
-
titleize
(word, uppercase='')[source]¶ Converts an underscored or CamelCase word into a sentence. The titleize function converts text like “WelcomePage”, “welcome_page” or “welcome page” to this “Welcome Page”. If the “uppercase” parameter is set to ‘first’ it will only capitalize the first character of the title.
-
unaccent
(text)[source]¶ Transforms a string to its unaccented version. This might be useful for generating “friendly” URLs
-
underscore
(word)[source]¶ Converts a word “into_it_s_underscored_version” Convert any “CamelCased” or “ordinary Word” into an “underscored_word”. This can be really useful for creating friendly URLs.
-
json
Module¶
-
galaxy.util.json.
dumps
(obj, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, cls=None, indent=None, separators=None, encoding='utf-8', default=None, sort_keys=False, **kw)[source]¶ Serialize
obj
to a JSON formattedstr
.If
skipkeys
is false thendict
keys that are not basic types (str
,unicode
,int
,long
,float
,bool
,None
) will be skipped instead of raising aTypeError
.If
ensure_ascii
is false, all non-ASCII characters are not escaped, and the return value may be aunicode
instance. Seedump
for details.If
check_circular
is false, then the circular reference check for container types will be skipped and a circular reference will result in anOverflowError
(or worse).If
allow_nan
is false, then it will be aValueError
to serialize out of rangefloat
values (nan
,inf
,-inf
) in strict compliance of the JSON specification, instead of using the JavaScript equivalents (NaN
,Infinity
,-Infinity
).If
indent
is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines.None
is the most compact representation. Since the default item separator is', '
, the output might include trailing whitespace whenindent
is specified. You can useseparators=(',', ': ')
to avoid this.If
separators
is an(item_separator, dict_separator)
tuple then it will be used instead of the default(', ', ': ')
separators.(',', ':')
is the most compact JSON representation.encoding
is the character encoding for str instances, default is UTF-8.default(obj)
is a function that should return a serializable version of obj or raise TypeError. The default simply raises TypeError.If sort_keys is
True
(default:False
), then the output of dictionaries will be sorted by key.To use a custom
JSONEncoder
subclass (e.g. one that overrides the.default()
method to serialize additional types), specify it with thecls
kwarg; otherwiseJSONEncoder
is used.
-
galaxy.util.json.
loads
(s, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw)[source]¶ Deserialize
s
(astr
orunicode
instance containing a JSON document) to a Python object.If
s
is astr
instance and is encoded with an ASCII based encoding other than utf-8 (e.g. latin-1) then an appropriateencoding
name must be specified. Encodings that are not ASCII based (such as UCS-2) are not allowed and should be decoded tounicode
first.object_hook
is an optional function that will be called with the result of any object literal decode (adict
). The return value ofobject_hook
will be used instead of thedict
. This feature can be used to implement custom decoders (e.g. JSON-RPC class hinting).object_pairs_hook
is an optional function that will be called with the result of any object literal decoded with an ordered list of pairs. The return value ofobject_pairs_hook
will be used instead of thedict
. This feature can be used to implement custom decoders that rely on the order that the key and value pairs are decoded (for example, collections.OrderedDict will remember the order of insertion). Ifobject_hook
is also defined, theobject_pairs_hook
takes priority.parse_float
, if specified, will be called with the string of every JSON float to be decoded. By default this is equivalent to float(num_str). This can be used to use another datatype or parser for JSON floats (e.g. decimal.Decimal).parse_int
, if specified, will be called with the string of every JSON int to be decoded. By default this is equivalent to int(num_str). This can be used to use another datatype or parser for JSON integers (e.g. float).parse_constant
, if specified, will be called with one of the following strings: -Infinity, Infinity, NaN, null, true, false. This can be used to raise an exception if invalid JSON numbers are encountered.To use a custom
JSONDecoder
subclass, specify it with thecls
kwarg; otherwiseJSONDecoder
is used.
lrucache
Module¶
Kanwei Li, 03/2010
Simple LRU cache that uses a dictionary to store a specified number of objects at a time.
memdump
Module¶
none_like
Module¶
Objects with No values
odict
Module¶
Ordered dictionary implementation.
-
class
galaxy.util.odict.
odict
(dict=None)[source]¶ Bases:
UserDict.UserDict
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/107747
This dictionary class extends UserDict to record the order in which items are added. Calling keys(), values(), items(), etc. will return results in this order.
sanitize_html
Module¶
HTML Sanitizer (ripped from feedparser)
shed_util
Module¶
shed_util_common
Module¶
streamball
Module¶
A simple wrapper for writing tarballs as a stream.
topsort
Module¶
Topological sort.
- From Tim Peters, see:
- http://mail.python.org/pipermail/python-list/1999-July/006660.html
topsort takes a list of pairs, where each pair (x, y) is taken to mean that x <= y wrt some abstract partial ordering. The return value is a list, representing a total ordering that respects all the input constraints. E.g.,
topsort( [(1,2), (3,3)] )
Valid topological sorts would be any of (but nothing other than)
[3, 1, 2] [1, 3, 2] [1, 2, 3]
... however this variant ensures that ‘key’ order (first element of tuple) is preserved so the following will be result returned:
[1, 3, 2]
because those are the permutations of the input elements that respect the “1 precedes 2” and “3 precedes 3” input constraints. Note that a constraint of the form (x, x) is really just a trick to make sure x appears somewhere in the output list.
If there’s a cycle in the constraints, say
topsort( [(1,2), (2,1)] )
then CycleError is raised, and the exception object supports many methods to help analyze and break the cycles. This requires a good deal more code than topsort itself!