util Package¶
util Package¶
Utility functions used systemwide.
-
class
galaxy.util.Params(params, sanitize=True)[source]¶ Bases:
objectStores and ‘sanitizes’ parameters. Alphanumeric characters and the non-alphanumeric ones that are deemed safe are let to pass through (see L{valid_chars}). Some non-safe characters are escaped to safe forms for example C{>} becomes C{__lt__} (see L{mapped_chars}). All other characters are replaced with C{X}.
Operates on string or list values only (HTTP parameters).
>>> values = { 'status':'on', 'symbols':[ 'alpha', '<>', '$rm&#!' ] } >>> par = Params(values) >>> par.status 'on' >>> par.value == None # missing attributes return None True >>> par.get('price', 0) 0 >>> par.symbols # replaces unknown symbols with X ['alpha', '__lt____gt__', 'XrmX__pd__!'] >>> par.flatten() # flattening to a list [('status', 'on'), ('symbols', 'alpha'), ('symbols', '__lt____gt__'), ('symbols', 'XrmX__pd__!')]
-
NEVER_SANITIZE= ['file_data', 'url_paste', 'URL', 'filesystem_paths']¶
-
-
galaxy.util.compare_urls(url1, url2, compare_scheme=True, compare_hostname=True, compare_path=True)[source]¶
-
galaxy.util.docstring_trim(docstring)[source]¶ Trimming python doc strings. Taken from: http://www.python.org/dev/peps/pep-0257/
-
galaxy.util.file_iter(fname, sep=None)[source]¶ This generator iterates over a file and yields its lines splitted via the C{sep} parameter. Skips empty lines and lines starting with the C{#} character.
>>> lines = [ line for line in file_iter(__file__) ] >>> len(lines) != 0 True
-
galaxy.util.file_reader(fp, chunk_size=65536)[source]¶ This generator yields the open fileobject in chunks (default 64k). Closes the file at the end
-
galaxy.util.in_directory(file, directory, local_path_module=<module 'posixpath' from '/home/docs/checkouts/readthedocs.org/user_builds/jmchilton-galaxy/envs/stable/lib/python2.7/posixpath.pyc'>)[source]¶ Return true, if the common prefix of both is equal to directory e.g. /a/b/c/d.rst and directory is /a/b, the common prefix is /a/b
-
galaxy.util.is_binary(value, binary_chars=None)[source]¶ File is binary if it contains a null-byte by default (e.g. behavior of grep, etc.). This may fail for utf-16 files, but so would ASCII encoding. >>> is_binary( string.printable ) False >>> is_binary( ‘xcex94’ ) False >>> is_binary( ‘000’ ) True
-
galaxy.util.is_uuid(value)[source]¶ This method returns True if value is a UUID, otherwise False. >>> is_uuid( “123e4567-e89b-12d3-a456-426655440000” ) True >>> is_uuid( “0x3242340298902834” ) False
-
galaxy.util.listify(item, do_strip=False)[source]¶ Make a single item a single item list, or return a list if passed a list. Passing a None returns an empty list.
-
galaxy.util.merge_sorted_iterables(operator, *iterables)[source]¶ >>> operator = lambda x: x >>> list( merge_sorted_iterables( operator, [1,2,3], [4,5] ) ) [1, 2, 3, 4, 5] >>> list( merge_sorted_iterables( operator, [4, 5], [1,2,3] ) ) [1, 2, 3, 4, 5] >>> list( merge_sorted_iterables( operator, [1, 4, 5], [2], [3] ) ) [1, 2, 3, 4, 5]
-
galaxy.util.mkstemp_ln(src, prefix='mkstemp_ln_')[source]¶ From tempfile._mkstemp_inner, generate a hard link in the same dir with a random name. Created so we can persist the underlying file of a NamedTemporaryFile upon its closure.
-
galaxy.util.nice_size(size)[source]¶ Returns a readably formatted string with the size
>>> nice_size(100) '100 bytes' >>> nice_size(10000) '9.8 KB' >>> nice_size(1000000) '976.6 KB' >>> nice_size(100000000) '95.4 MB'
-
galaxy.util.read_build_sites(filename, check_builds=True)[source]¶ read db names to ucsc mappings from file, this file should probably be merged with the one above
-
galaxy.util.ready_name_for_url(raw_name)[source]¶ General method to convert a string (i.e. object name) to a URL-ready slug.
>>> ready_name_for_url( "My Cool Object" ) 'My-Cool-Object' >>> ready_name_for_url( "!My Cool Object!" ) 'My-Cool-Object' >>> ready_name_for_url( "Hello₩◎ґʟⅾ" ) 'Hello'
-
galaxy.util.restore_text(text, character_map={'@': '__at__', '\t': '__tc__', '\n': '__cn__', '\r': '__cr__', '[': '__ob__', ']': '__cb__', '#': '__pd__', '"': '__dq__', "'": '__sq__', '{': '__oc__', '}': '__cc__', '<': '__lt__', '>': '__gt__'})[source]¶ Restores sanitized text
-
galaxy.util.roundify(amount, sfs=2)[source]¶ Take a number in string form and truncate to ‘sfs’ significant figures.
-
galaxy.util.sanitize_for_filename(text, default=None)[source]¶ Restricts the characters that are allowed in a filename portion; Returns default value or a unique id string if result is not a valid name. Method is overly aggressive to minimize possible complications, but a maximum length is not considered.
-
galaxy.util.sanitize_lists_to_string(values, valid_characters=set(['!', ' ', ')', '(', '+', '*', '-', ', ', '/', '.', '1', '0', '3', '2', '5', '4', '7', '6', '9', '8', ':', '=', '?', 'A', 'C', 'B', 'E', 'D', 'G', 'F', 'I', 'H', 'K', 'J', 'M', 'L', 'O', 'N', 'Q', 'P', 'S', 'R', 'U', 'T', 'W', 'V', 'Y', 'X', 'Z', '_', '^', 'a', 'c', 'b', 'e', 'd', 'g', 'f', 'i', 'h', 'k', 'j', 'm', 'l', 'o', 'n', 'q', 'p', 's', 'r', 'u', 't', 'w', 'v', 'y', 'x', 'z']), character_map={'@': '__at__', '\t': '__tc__', '\n': '__cn__', '\r': '__cr__', '[': '__ob__', ']': '__cb__', '#': '__pd__', '"': '__dq__', "'": '__sq__', '{': '__oc__', '}': '__cc__', '<': '__lt__', '>': '__gt__'}, invalid_character='X')[source]¶
-
galaxy.util.sanitize_param(value, valid_characters=set(['!', ' ', ')', '(', '+', '*', '-', ', ', '/', '.', '1', '0', '3', '2', '5', '4', '7', '6', '9', '8', ':', '=', '?', 'A', 'C', 'B', 'E', 'D', 'G', 'F', 'I', 'H', 'K', 'J', 'M', 'L', 'O', 'N', 'Q', 'P', 'S', 'R', 'U', 'T', 'W', 'V', 'Y', 'X', 'Z', '_', '^', 'a', 'c', 'b', 'e', 'd', 'g', 'f', 'i', 'h', 'k', 'j', 'm', 'l', 'o', 'n', 'q', 'p', 's', 'r', 'u', 't', 'w', 'v', 'y', 'x', 'z']), character_map={'@': '__at__', '\t': '__tc__', '\n': '__cn__', '\r': '__cr__', '[': '__ob__', ']': '__cb__', '#': '__pd__', '"': '__dq__', "'": '__sq__', '{': '__oc__', '}': '__cc__', '<': '__lt__', '>': '__gt__'}, invalid_character='X')[source]¶ Clean incoming parameters (strings or lists)
-
galaxy.util.sanitize_text(text, valid_characters=set(['!', ' ', ')', '(', '+', '*', '-', ', ', '/', '.', '1', '0', '3', '2', '5', '4', '7', '6', '9', '8', ':', '=', '?', 'A', 'C', 'B', 'E', 'D', 'G', 'F', 'I', 'H', 'K', 'J', 'M', 'L', 'O', 'N', 'Q', 'P', 'S', 'R', 'U', 'T', 'W', 'V', 'Y', 'X', 'Z', '_', '^', 'a', 'c', 'b', 'e', 'd', 'g', 'f', 'i', 'h', 'k', 'j', 'm', 'l', 'o', 'n', 'q', 'p', 's', 'r', 'u', 't', 'w', 'v', 'y', 'x', 'z']), character_map={'@': '__at__', '\t': '__tc__', '\n': '__cn__', '\r': '__cr__', '[': '__ob__', ']': '__cb__', '#': '__pd__', '"': '__dq__', "'": '__sq__', '{': '__oc__', '}': '__cc__', '<': '__lt__', '>': '__gt__'}, invalid_character='X')[source]¶ Restricts the characters that are allowed in text; accepts both strings and lists of strings; non-string entities will be cast to strings.
-
galaxy.util.shrink_stream_by_size(value, size, join_by='..', left_larger=True, beginning_on_size_error=False, end_on_size_error=False)[source]¶
-
galaxy.util.shrink_string_by_size(value, size, join_by='..', left_larger=True, beginning_on_size_error=False, end_on_size_error=False)[source]¶
-
galaxy.util.size_to_bytes(size)[source]¶ Returns a number of bytes if given a reasonably formatted string with the size
-
galaxy.util.smart_str(s, encoding='utf-8', strings_only=False, errors='strict')[source]¶ Returns a bytestring version of ‘s’, encoded as specified in ‘encoding’.
If strings_only is True, don’t convert (some) non-string-like objects.
Adapted from an older, simpler version of django.utils.encoding.smart_str.
-
galaxy.util.string_as_bool_or_none(string)[source]¶ - Returns True, None or False based on the argument:
- True if passed True, ‘True’, ‘Yes’, or ‘On’ None if passed None or ‘None’ False otherwise
Note: string comparison is case-insensitive so lowecase versions of those function equivalently.
-
galaxy.util.synchronized(func)[source]¶ This wrapper will serialize access to ‘func’ to a single thread. Use it as a decorator.
-
galaxy.util.umask_fix_perms(path, umask, unmasked_perms, gid=None)[source]¶ umask-friendly permissions fixing
-
galaxy.util.unicodify(value, encoding='utf-8', error='replace', default=None)[source]¶ Returns a unicode string or None
aliaspickler Module¶
bunch Module¶
-
class
galaxy.util.bunch.Bunch(**kwds)[source]¶ Bases:
objecthttp://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52308
Often we want to just collect a bunch of stuff together, naming each item of the bunch; a dictionary’s OK for that, but a small do-nothing class is even handier, and prettier to use.
debugging Module¶
-
class
galaxy.util.debugging.SimpleProfiler(log=None)[source]¶ Bases:
objectSimple profiler that captures the duration between calls to report and stores the results in a list.
-
REPORT_FORMAT= '%20f: %s'¶
-
-
galaxy.util.debugging.stack_trace_string(max_depth=None, line_format='{index}:{file}:{function}:{line}')[source]¶ Returns a string representation of the current stack.
Parameters: depth – positive integer to control how many levels of the stack are returned. max_depth=None returns the entire stack (default).
expressions Module¶
Expression evaluation support.
For the moment this depends on python’s eval. In the future it should be replaced with a “safe” parser.
hash_util Module¶
Utility functions for bi-directional Python version compatibility. Python 2.5 introduced hashlib which replaced sha in Python 2.4 and previous versions.
heartbeat Module¶
-
class
galaxy.util.heartbeat.Heartbeat(name='Heartbeat Thread', period=20, fname='heartbeat.log')[source]¶ Bases:
threading.ThreadThread that periodically dumps the state of all threads to a file
-
galaxy.util.heartbeat.get_current_thread_object_dict()[source]¶ Get a dictionary of all ‘Thread’ objects created via the threading module keyed by thread_id. Note that not all interpreter threads have a thread objects, only the main thread and any created via the ‘threading’ module. Threads created via the low level ‘thread’ module will not be in the returned dictionary.
- HACK: This mucks with the internals of the threading module since that
- module does not expose any way to match ‘Thread’ objects with intepreter thread identifiers (though it should).
inflection Module¶
-
class
galaxy.util.inflection.Base[source]¶ Locale inflectors must inherit from this base class inorder to provide the basic Inflector functionality
-
camelize(word)[source]¶ Returns given word as CamelCased Converts a word like “send_email” to “SendEmail”. It will remove non alphanumeric character from the word, so “who’s online” will be converted to “WhoSOnline”
-
classify(table_name)[source]¶ Converts a table name to its class name according to rails naming conventions. Example: Converts “people” to “Person”
-
cond_plural(number_of_records, word)[source]¶ Returns the plural form of a word if first parameter is greater than 1
-
foreignKey(class_name, separate_class_name_and_id_with_underscore=1)[source]¶ Returns class_name in underscored form, with “_id” tacked on at the end. This is for use in dealing with the database.
-
humanize(word, uppercase='')[source]¶ Returns a human-readable string from word Returns a human-readable string from word, by replacing underscores with a space, and by upper-casing the initial character by default. If you need to uppercase all the words you just have to pass ‘all’ as a second parameter.
-
ordinalize(number)[source]¶ Converts number to its ordinal English form. This method converts 13 to 13th, 2 to 2nd ...
-
string_replace(word, find, replace)[source]¶ This function returns a copy of word, translating all occurrences of each character in find to the corresponding character in replace
-
tableize(class_name)[source]¶ Converts a class name to its table name according to rails naming conventions. Example. Converts “Person” to “people”
-
titleize(word, uppercase='')[source]¶ Converts an underscored or CamelCase word into a English sentence. The titleize function converts text like “WelcomePage”, “welcome_page” or “welcome page” to this “Welcome Page”. If second parameter is set to ‘first’ it will only capitalize the first character of the title.
-
unaccent(text)[source]¶ Transforms a string to its unaccented version. This might be useful for generating “friendly” URLs
-
underscore(word)[source]¶ Converts a word “into_it_s_underscored_version” Convert any “CamelCased” or “ordinary Word” into an “underscored_word”. This can be really useful for creating friendly URLs.
-
-
class
galaxy.util.inflection.English[source]¶ Bases:
galaxy.util.inflection.BaseInflector for pluralize and singularize English nouns.
This is the default Inflector for the Inflector obj
-
class
galaxy.util.inflection.Inflector(Inflector=<class galaxy.util.inflection.English>)[source]¶ Inflector for pluralizing and singularizing nouns.
It provides methods for helping on creating programs based on naming conventions like on Ruby on Rails.
-
camelize(word)[source]¶ Returns given word as CamelCased Converts a word like “send_email” to “SendEmail”. It will remove non alphanumeric character from the word, so “who’s online” will be converted to “WhoSOnline”
-
classify(table_name)[source]¶ Converts a table name to its class name according to rails naming conventions. Example: Converts “people” to “Person”
-
cond_plural(number_of_records, word)[source]¶ Returns the plural form of a word if first parameter is greater than 1
-
foreignKey(class_name, separate_class_name_and_id_with_underscore=1)[source]¶ Returns class_name in underscored form, with “_id” tacked on at the end. This is for use in dealing with the database.
-
humanize(word, uppercase='')[source]¶ Returns a human-readable string from word Returns a human-readable string from word, by replacing underscores with a space, and by upper-casing the initial character by default. If you need to uppercase all the words you just have to pass ‘all’ as a second parameter.
-
ordinalize(number)[source]¶ Converts number to its ordinal form. This method converts 13 to 13th, 2 to 2nd ...
-
tableize(class_name)[source]¶ Converts a class name to its table name according to rails naming conventions. Example. Converts “Person” to “people”
-
titleize(word, uppercase='')[source]¶ Converts an underscored or CamelCase word into a sentence. The titleize function converts text like “WelcomePage”, “welcome_page” or “welcome page” to this “Welcome Page”. If the “uppercase” parameter is set to ‘first’ it will only capitalize the first character of the title.
-
unaccent(text)[source]¶ Transforms a string to its unaccented version. This might be useful for generating “friendly” URLs
-
underscore(word)[source]¶ Converts a word “into_it_s_underscored_version” Convert any “CamelCased” or “ordinary Word” into an “underscored_word”. This can be really useful for creating friendly URLs.
-
json Module¶
-
galaxy.util.json.dumps(obj, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, cls=None, indent=None, separators=None, encoding='utf-8', default=None, sort_keys=False, **kw)[source]¶ Serialize
objto a JSON formattedstr.If
skipkeysis false thendictkeys that are not basic types (str,unicode,int,long,float,bool,None) will be skipped instead of raising aTypeError.If
ensure_asciiis false, all non-ASCII characters are not escaped, and the return value may be aunicodeinstance. Seedumpfor details.If
check_circularis false, then the circular reference check for container types will be skipped and a circular reference will result in anOverflowError(or worse).If
allow_nanis false, then it will be aValueErrorto serialize out of rangefloatvalues (nan,inf,-inf) in strict compliance of the JSON specification, instead of using the JavaScript equivalents (NaN,Infinity,-Infinity).If
indentis a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines.Noneis the most compact representation. Since the default item separator is', ', the output might include trailing whitespace whenindentis specified. You can useseparators=(',', ': ')to avoid this.If
separatorsis an(item_separator, dict_separator)tuple then it will be used instead of the default(', ', ': ')separators.(',', ':')is the most compact JSON representation.encodingis the character encoding for str instances, default is UTF-8.default(obj)is a function that should return a serializable version of obj or raise TypeError. The default simply raises TypeError.If sort_keys is
True(default:False), then the output of dictionaries will be sorted by key.To use a custom
JSONEncodersubclass (e.g. one that overrides the.default()method to serialize additional types), specify it with theclskwarg; otherwiseJSONEncoderis used.
-
galaxy.util.json.loads(s, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw)[source]¶ Deserialize
s(astrorunicodeinstance containing a JSON document) to a Python object.If
sis astrinstance and is encoded with an ASCII based encoding other than utf-8 (e.g. latin-1) then an appropriateencodingname must be specified. Encodings that are not ASCII based (such as UCS-2) are not allowed and should be decoded tounicodefirst.object_hookis an optional function that will be called with the result of any object literal decode (adict). The return value ofobject_hookwill be used instead of thedict. This feature can be used to implement custom decoders (e.g. JSON-RPC class hinting).object_pairs_hookis an optional function that will be called with the result of any object literal decoded with an ordered list of pairs. The return value ofobject_pairs_hookwill be used instead of thedict. This feature can be used to implement custom decoders that rely on the order that the key and value pairs are decoded (for example, collections.OrderedDict will remember the order of insertion). Ifobject_hookis also defined, theobject_pairs_hooktakes priority.parse_float, if specified, will be called with the string of every JSON float to be decoded. By default this is equivalent to float(num_str). This can be used to use another datatype or parser for JSON floats (e.g. decimal.Decimal).parse_int, if specified, will be called with the string of every JSON int to be decoded. By default this is equivalent to int(num_str). This can be used to use another datatype or parser for JSON integers (e.g. float).parse_constant, if specified, will be called with one of the following strings: -Infinity, Infinity, NaN, null, true, false. This can be used to raise an exception if invalid JSON numbers are encountered.To use a custom
JSONDecodersubclass, specify it with theclskwarg; otherwiseJSONDecoderis used.
lrucache Module¶
Kanwei Li, 03/2010
Simple LRU cache that uses a dictionary to store a specified number of objects at a time.
memdump Module¶
none_like Module¶
Objects with No values
odict Module¶
Ordered dictionary implementation.
-
class
galaxy.util.odict.odict(dict=None)[source]¶ Bases:
UserDict.UserDicthttp://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/107747
This dictionary class extends UserDict to record the order in which items are added. Calling keys(), values(), items(), etc. will return results in this order.
sanitize_html Module¶
HTML Sanitizer (ripped from feedparser)
shed_util Module¶
shed_util_common Module¶
streamball Module¶
A simple wrapper for writing tarballs as a stream.
topsort Module¶
Topological sort.
- From Tim Peters, see:
- http://mail.python.org/pipermail/python-list/1999-July/006660.html
topsort takes a list of pairs, where each pair (x, y) is taken to mean that x <= y wrt some abstract partial ordering. The return value is a list, representing a total ordering that respects all the input constraints. E.g.,
topsort( [(1,2), (3,3)] )
Valid topological sorts would be any of (but nothing other than)
[3, 1, 2] [1, 3, 2] [1, 2, 3]
... however this variant ensures that ‘key’ order (first element of tuple) is preserved so the following will be result returned:
[1, 3, 2]
because those are the permutations of the input elements that respect the “1 precedes 2” and “3 precedes 3” input constraints. Note that a constraint of the form (x, x) is really just a trick to make sure x appears somewhere in the output list.
If there’s a cycle in the constraints, say
topsort( [(1,2), (2,1)] )
then CycleError is raised, and the exception object supports many methods to help analyze and break the cycles. This requires a good deal more code than topsort itself!