`dlb.ex` — Execution contexts and dependency-aware tool execution¶

Context objects¶

An (execution) context describes how running tool instances shall interact with the execution environment outside the working tree and with each other. E.g:

number of asynchronously running tool instances

search paths for dynamic helper files

environment variables to be imported from os.environ for use in tool instances

It also controls how diagnostic messages of tool instances are handled and helps with filesystem abstraction (e.g. working tree time, case sensitivity of names in the working tree).

A context is represented as an instance of dlb.ex.Context used as a context manager. The context is entered with the call of _enter__() and exited with the return of __exit__().

Contexts can be nested:

import dlb.ex

# no active context

with dlb.ex.Context():                # A: root context, outer context of B, C, D
    # A is the active context
    with dlb.ex.Context():            # B: inner context of A, outer context of C
        # B is the active context
        with dlb.ex.Context():        # C: inner context of A, B
           # C is the active context
    with dlb.ex.Context():            # D: inner context of A
        # D is the active context
    # A is the active context

# no active context

Combine contexts with message clusters to describe what happens in the context:

with dlb.di.Cluster('this happens in the context'), dlb.ex.Context():
    ...

class dlb.ex.Context(path_cls=dlb.fs.Path, max_parallel_redo_count=1, find_helpers=None)¶

An instance does nothing unless used as a context manager.

When used as a context manager, it embodies an (execution) context and activates it:

a root context, if dlb is not yet running;

an inner context of the active context, otherwise.

When a root context is entered, the working directory of the Python process must be a working tree’s root whose absolute path does not contain unresolved symbolic link.

When a context (root or not) is entered, the path of the working tree’s root must be representable as as path_cls. This allows you to impose restrictions on the accepted paths.

If find_helpers is None for a root context, True is used instead. If find_helpers is None for an active context that is not the root context, find_helpers of the root context is used.

Parameters:	path_cls (dlb.fs.Path) – the subclass of `dlb.fs.Path` to be used to represent the working tree’s root max_parallel_redo_count (int) – maximum number of redos started in this context than can be pending at the same time find_helpers – are dynamic helpers not defined explicitly to be searched for in `executable_search_paths`? find_helpers – None \| bool
Raises:	TypeError – if path_cls is not a subclass of `dlb.fs.Path`

Entering or exiting a context may raise the following exceptions:

exception	meaning	when
`NoWorkingTreeError`	the working directory is not a working tree’s root	entering root context
`ManagementTreeError`	the management tree cannot be setup inside the working tree	entering root context
`ValueError`	the working tree’s root path violates the requested restrictions	entering (any) context
`ContextNestingError`	the contexts are not properly nested	exiting (any) context
`WorkingTreeTimeError` \| working tree time behaved unexpectedly \| exiting root context

Note

Most attributes and methods are available “on the class” as well as “on the instance”, and refer to the corresponding attribute of the active context:

with dlb.ex.Context:
    with dlb.ex.Context as c:
        ... = dlb.ex.Context.working_tree_time_ns   # preferred
        ... c.active.working_tree_time_ns           # also possible
        ... c.working_tree_time_ns                  # also possible

The Context class supports the following methods and attributes:

active¶

The active context.

Same on class and instance.

Raises:	NotRunningError – if dlb is not running).

path_cls¶

The subclass of dlb.fs.Path defined in the constructor.

When called on class, it refers to the active context.

Raises:	NotRunningError – if dlb is not running).

max_parallel_redo_count¶

The maximum number of redos started in this context than can be pending at the same time, as defined in the constructor.

When called on class, it refers to the active context.

Raises:	NotRunningError – if dlb is not running).

find_helpers¶

Find dynamic helpers not defined explicitly are in executable_search_paths? This is defined in the constructor.

When called on class, it refers to the active context.

Raises:	NotRunningError – if dlb is not running).

root_path¶

The absolute path to the working tree’s root.

It is an instance of Context.active.path_cls and is representable as an instance of path_cls of the active context and every possible outer context.

Same on class and instance.

Raises:	NotRunningError – if dlb is not running).

executable_search_paths¶

A duplicate-free tuple of absolute directory paths where this process should look for executables according to the operating system.

It is compiled from the members os.get_exec_path() when the root context is entered.

Same on class and instance.

Raises:	NotRunningError – if dlb is not running).

find_path_in(path, search_prefixes=None)¶

Find the first existing and accessible path in search_prefixes and return its absolute path. Returns None if path is not found in search_prefixed.

If path is a dlb.fs.Path with path.is_dir() = True, existing non-directories in search_prefixes are ignored. If path is a dlb.fs.Path with path.is_dir() = False, existing directories in search_prefixes are ignored.

Relative paths in search_prefixes are treated as relative to root_path.

If search_prefixes is None, executable_search_paths is used instead.

Does not raise OSError.

Parameters:	path (`dlb.fs.Path` or anything a `dlb.fs.Path` can be constructed from) – the relative path to find search_prefixes (an iterable other than `str` or `bytes` or `None`) – paths of directories to search in
Returns:	an absolute path or `None`.

working_tree_time_ns¶

The current working tree time in nanoseconds as an integer.

Same on class and instance.

Raises:	NotRunningError – if dlb is not running).

temporary(suffix='', is_dir=False)¶

Return a dlb.ex.Temporary object, representing a temporary regular file (for is_dir = False) or a temporary directory (for is_dir = True) in the management tree with a unique path.

Usage example:

with context.temporary(suffix='.o') as p:
    ...  # an empty file with absolute path *p* exists

... = context.temporary().path  # just get the absolute path, do not create the file

The path attribute of the returned object is an absolute path in the same directory for all calls in the root context, as a dlb.fs.Path object. Its last component is unique among all calls in the root context. path.is_dir() is is_dir.

The unique path component starts with a lower-case letter and ends with suffix. It contains only lower-case letters and decimal digits between its first characters and the suffix. If suffix is not empty, is must start with a character from strings.punctuation and must not contain '/'. The the unique path component without the suffix is at most 12 characters long for the first 2**61 calls.

When used as a context manager, an empty regular file or directory with path is created when entered and removed (with its content) on exit. Raises FileExistError if the regular file or directory exists.

Same on class and instance.

Parameters:	suffix (str) – suffix of the unique path component
Raises:	ValueError – if suffix is invalid FileExistsError – if the regular file or directory exists (when used as content manager) NotRunningError – if dlb is not running).

working_tree_path_of(path, *, is_dir=None, existing=False, collapsable=False,

allow_nontemporary_management=False, allow_temporary=False)

Return the managed tree path of the path of a filesystem object in the managed tree.

For path to be considered as the path of a filesystem object in the managed tree, path must either be a relative path or it must have root_path as a prefix.

The arguments existing and collapsable describe the assumptions on the filesystem content that may be used to increase the speed and reduce the number of filesystem accesses.

If existing and collapsable are True and path is relative, the filesystem is never accessed.

If existing is False, is_dir() of the returned path reflects the type of the actual filesystem object. Raises dlb.fs.PathNormalizationError if path does not exist.

If allow_nontemporary_management is True, the resulting path may denote a filesystem object in the management tree except in .dlbroot/t. If allow_temporary is True, the resulting path may denote a filesystem object in .dlbroot/t of the management tree.

Does not raise OSError.

Same on class and instance.

Parameters:	path (`dlb.fs.Path` or anything a `dlb.fs.Path` can be constructed from) – a path of a filesystem object in the managed tree is_dir (NoneType \| bool) – `True` if this is a directory path, `False` if not and `None` for derivation from path existing (bool) – assume that all involved filesystem objects exist? collapsable (bool) – assume that any relative to the working tree root is collapsable? allow_nontemporary_management (bool) – is the path permitted to denote a filesystem object the :term`management tree` except ones in `.dlbroot/t`? allow_temporary (bool) – is the path permitted to denote a filesystem object in `.dlbroot/t` of the management tree?
Returns:	a `dlb.fs.Path` p with `p.is_absolute() == False` and `p.is_normalized() == True`
Return type:	same class as path if path is a `dlb.fs.Path` and `dlb.fs.Path` otherwise
Raises:	dlb.fs.PathNormalizationError – if path does not exist in the parts of the working tree that are described by allow_nontemporary_management and allow_temporary ValueError – if the resulting path is not representable NotRunningError – if dlb is not running).

env¶

The environment variable dictionary object with this context as its associated context.

When called on class, it refers to the active context.

Raises:	NotRunningError – if dlb is not running).

helper¶

The dynamic helper dictionary object with this context as its associated context.

The dynamic helper dictionary object maps dynamic helpers to absolute paths, either explicitly or implicitly with the help of find_path_in().

If the active context and the root context both have find_helpers = False and no paths was explicitly assigned to the dynamic helper p in the active context or one of its outer contexts, a look-up with dlb.ex.Context.helper[p] performs a search with dlb.ex.Context.find_path_in(p). (Each such search is performed only once for a given path; the result is stored.)

Examples:

>>> dlb.ex.Context.helper['gcc']
Path('/usr/bin/gcc')

>>> dlb.ex.Context.helper['gcc'] = '/usr/local/bin/my-very-own-very-special-gcc'  # set the path explicitly
>>> dlb.ex.Context.helper['gcc']
Path('/usr/local/bin/my-very-own-very-special-gcc')

>>> dlb.ex.Context.helper['tmp/'] = 'out/t/'  # relative path: relative to the working tree's root path
>>> dlb.ex.Context.helper['tmp/']
Path('/home/schmutzli/projects/esel/out/t')   # with '/home/schmutzli/projects/esel' as the working tree's root

When called on class, it refers to the active context.

Raises:	NotRunningError – if dlb is not running).

Environment variable dictionary objects¶

The environment variable dictionary object env returned by c.env for a context c is a dictionary-like object of all environment variables defined in this c. c is called the associated context of env.

In addition, the environment variable dictionary object manages the import of environment variables from environment variables of the outer context and restriction of imported or assigned values in the form of regular expressions.

The environment variables of the outer context of the root context is defined by os.environ.

Example:

# os.environ usually contains the environment variables in the shell that called the Python interpreter

with dlb.ex.Context():  # takes a snapshot of os.environ

    # import the environment variable 'LANG' into the context
    dlb.ex.Context.active.env.import_from_outer(
        'LANG', restriction=r'[a-z]{2}_[A-Z]{2}', example='sv_SE')

    # now the environment variable is either undefined or matches the regular expression given
    # (in this context and all future inner contexts)

    ... = dlb.ex.Context.active.env['LANG']
        # value in snapshot of os.environ complying to the restriction or KeyError

    dlb.ex.Context.active.env['LANG'] = 'de_AT'

    with dlb.ex.Context():

        # further restrict the value and make sure it is defined
        dlb.ex.Context.active.env.import_from_outer(
            'LANG', restriction='(?P<language>de).*', example='de_CH')

        ... = dlb.ex.Context.active.env['LANG']  # 'de_AT'
        del dlb.ex.Context.active.env['LANG']

        dlb.ex.Context.active.env['LANG'] = 'de_CH'
        # dlb.ex.Context.active.env['LANG'] = 'fr_FR'  # would raise ValueError

    ... = dlb.ex.Context.active.env['LANG']  # 'de_AT'

    del dlb.ex.Context.active.env['LANG']  # undefine 'LANG'
    dlb.ex.Context.active.env['LANG'] = 'fr_FR'  # ok

Environment variable dictionary object support the following methods and attributes:

EnvVarDict.import_from_outer(name, restriction, value_if_undefined=None, example=None)¶

Sets the value of the environment variable named name from the innermost outer context that defines it. If no outer context defines it, the environment variable remains undefined.

Also sets the importing restriction for the value of the environment variable; when it is or later becomes defined, it regular expression restriction must match its value.

The possible imported value and the importing restriction apply to the context and all its future inner contexts.

When called for a root contest, the environment variables are imported from os.environ at the time is was entered.

Parameters:	name (str) – (non-empty) name of the environment variable restriction (str \| `typing.Pattern`) – regular expression example (str) – typical value of a environment variable, restriction must match this
Raises:	ValueError – if an environment variable named name is defined in the associated or an outer context and restriction does not match its value NonActiveContextAccessError – if the associated context is not an active context

EnvVarDict.is_imported(name)¶

Returns True if name is the name of an environment variable imported in the associated context or any of its outer contexts, else False.

Parameters:	name (str) – non-empty name of an environment variable
Raises:	TypeError – if name is not a string ValueError – if name is an empty string

EnvVarDict.get(name, default=None)¶

Return its value if name is the name of a defined environment variable in the associated context, else default.

Parameters:	name (str) – non-empty name of an environment variable
Raises:	TypeError – if name is not a string ValueError – if name is an empty string

EnvVarDict.items()¶: Returns a new view of the dictionary’s items (name, value) pairs of all defined environment variables.

name in env: Returns True if there is a environment variable named name defined in env, else False.

name not in env: Equivalent to not name in env

env[name] = value

Defines an imported environment variable named name with value value in the associated context and all its future inner contexts.

Raises KeyError, if name was not imported in the associated context or one of its outer contexts.

Raises ValueError, if name was imported in the associated context or one of its outer contexts, but is invalid with respect to the restriction an importing context (can be this context and any outer context).

Raises NonActiveContextAccessError, if the associated context is not an active context.

del env[name]

Undefines a defined environment variable named name in the associated context and all its future inner contexts.

Raises KeyError, if name is not defined in the context.

Raises NonActiveContextAccessError, if the associated context is not an active context.

Tool objects¶

Every tool is represented by a subclass of Tool that describes its abstract behaviour and the way it is run (e.g. meaning of command line and output, interaction with file system and environment variables).

Tools are usually parametrized by dependency roles (e.g. input files) and execution parameters.

Each tool instance represents a concrete behaviour and can be run in an active context. Running a tool results in an awaitable result object.

Tool instances are immutable and hashable and fast to construct; the heavy lifting takes place while the tool instance is running.

Tools are customized by inheritance and defining class attributes.

class dlb.ex.Tool¶

A tool declares its dependency roles (e.g. map_file_dependency) and execution parameters (e.g. DO_INCLUDE_DEBUG_INFO, PAPER_FORMAT) as class attributes.

Every tool instance assigns concrete dependencies for the tool’s dependency roles (e.g. a filesystem path './out/hello.map' for a dependency role map_file_dependency), while the execution parameters are the same of all instances of the some tool.

Dependency roles are instances of subclasses of Tool.Dependency.

A new tool can be defined by inheriting from one or more other tools. When overriding a dependency roles, its overriding value must be of the same type as the overridden value and it must be at least as restrictive (e.g. if required dependency must not be overridden by a non-required one). When overriding an execution parameters, its overriding value must be of the same type as the overridden value.

Each subclass of Tool must be defined in a source code location unique among all subclasses of Tool. The definition raises DefinitionAmbiguityError, if its location is cannot be determined or if another subclass of Tool was defined before at the same location.

Example:

class Compiler(dlb.ex.Tool):
   WARNINGS = ('all',)
   source_file = dlb.ex.Tool.Input.RegularFile()
   object_file = dlb.ex.Tool.Output.RegularFile()

class Linker(dlb.ex.Tool):
   object_files = dlb.ex.Tool.Input.RegularFile[1:]()
   linked_file = dlb.ex.Tool.Output.RegularFile()
   map_file = dlb.ex.Tool.Output.RegularFile(required=False)

compiler = Compiler(source_file='main.cpp', object_file='main.cpp.o')
linker = Linker(object_files=[compiler.object_file], linked_file='main')

At construction of a tool, the dependencies given as keyword arguments to the constructor are validated by the tool’s dependency roles and made accessible (for reading only) as an attribute with the name of the corresponding dependency role and a type determined by the dependency role (e.g. dlb.fs.Path for Tool.Input.RegularFile):

>>> Compiler.object_file  # dependency role
<dlb.ex.Tool.Input.RegularFile object at ...>

>>> compiler.object_file  # dependency
Path('main.cpp.o')

run(force_redo=False)¶

Run the tool instance in the active context and returns a result (proxy) object result.

bool(result) is True if a redo is performed and False otherwise.

A redo is performed if force_redo is True or if it is necessary.

If a redo is performed, this method returns before the (asynchronous) redo is complete. After each of the following actions the redo is guaranteed to be complete (either successfully or by raising an exception):

read of a “public” attribute of the result proxy object

exit of the context run() was called in

enter of an inner context of the context run() was called in

modification of env or helper of the context run() was called in

call of run() of the same tool instance

The result object contains an attribute for every dependency role of the tool which contains the concrete dependencies.

If bool(result) is True, all attributes for dependencies have an assigned value. If bool(result) is False, only the attributes for explicit dependencies have an assigned value; the value of all attributes for non-explicit dependencies is NotImplemented.

redo(result, context)¶

Overwrite this method to implement a new Tool.

result is the result object that will by returned by the calling run(). context is the redo context (see Tool.RedoContext).

Use context.execute_helper() and context.replace_output().

Assign to attributes of result to define a non-explicit concrete dependency for the dependency role with the same name.

For a redo to be successful, this method must perform the following tasks:

Create all explicit output dependencies

Assign values to each required non-explicit dependencies

For a filesystem object whose path p is contained in an output dependency, it is recommended to first write to a temporary filesystem object q and then replace it with context.replace_output(p, q). This guarantees that no incomplete output dependency is left behind (like an only half-written object file) when the redo is aborted.

A filesystem object that is an output dependencies is treated as modified be the redo if it is a non-explicit dependency or if it is a explicit dependency that was replaced with context.replace_output().

Raises RuntimeError on the attempt to enter a new dlb.ex.Context as a context manager or to modify the active context.

Return True if the next run this tool instance should perform a redo, regardless of the necessity according to its dependencies.

Example:

class ATool(dlb.ex.Tool):
   EXECUTABLE = 'atool'

   source_file = dlb.ex.Tool.Input.RegularFile()
   output_file = dlb.ex.Tool.Output.RegularFile()
   included_files = dlb.ex.Tool.Input.RegularFile[:](explicit=False)

   async def redo(self, result, context):
       if ...:
           raise ValueException('invalid ...')
       with context.temporary() as temp_file_
          await context.execute_helper(self.EXECUTABLE, ['-o', temp_file, result.source_file])
          result.included_files = ...
          context.replace_output(result.output_file, temp_file)

definition_location¶

The definition location of the class.

It is a tuple of the form (file_path, in_archive_path, lineno) and uniquely identifies the tool among all subclasses of Tool.

in_archive_path is None, if the class was defined in an existing Python source file, and file_path is the os.path.realpath() of this file.

in_archive_path is the path relative of the source file in the zip archive, if the class was defined in an existing zip archive with a filename ending in .zip (loaded by zipimport) and file_path is the os.path.realpath() of this zip archive.

lineno is the 1-based line number in the source file.

fingerprint¶

The permanent local tool instance fingerprint of this instance.

This is a bytes object of fixed size, calculated from all its concrete dependencies d with d.explicit = True.

If two instances of the same subclass of Tool have “similar” explicit dependencies, their fingerprints are equal. If two instances of the same subclass of Tool have explicit dependencies that are not “similar”, their fingerprints are different with very high probability.

The explicit dependencies of two instances are considered “similar”, if they are equal or differ in a way that does not affect the meaning of the dependencies while the tool instance is running.

Redo context¶

A redo context is a read-only view for a dlb.ex.Context with some additional methods related to dynamic helpers and dependencies.

class Tool.RedoContext¶

A redo context is constructed automatically by Tool.run().

execute_helper(helper_file, arguments=(), *, cwd=None, expected_returncodes=frozenset([0]),

forced_env={}, stdin=None, stdout=None, stderr=None, limit=2**16)

Execute the helper_file with command-line arguments arguments in a subprocess with cwd as its working directory and wait for it to complete. The execution is considered successful if an only if its returncode is one in expected_returncodes.

If cwd is not None, is must be the path of directory in the managed tree or in .dlbroot/t/ of the management tree. Otherwise the working tree’s root is used as the working directory.

All members of arguments are converted to str objects.

If a member of arguments is a dlb.fs.Path object p with p.is_absolute() = True, is is replaced by str(p.native). If a member of arguments is a dlb.fs.Path object p with p.is_absolute() = False, is is replaced by str(q.native), where q is p expressed relative to the working directory. Is must denote a filesystem object in the managed tree or in .dlbroot/t/ of the management tree.

env of this object, modified by forced_env, forms the environment for the subprocess.

Parameters:

helper_file – dynamic helper to be executed as a relative path
arguments (iterable of objects that can be converted to str) – command-line arguments
cwd (None or a dlb.fs.Path or anything a dlb.fs.Path can be constructed from) – working directory of the subprocess to be started
expected_returncodes (collection of integers) – expected return codes of the dynamic helper helper_file
forced_env (None | Dict[str, str]) – dictionary of values to override in env or None
stdin – If not None: either a file-like object representing a pipe to be connected to the subprocess’s standard input stream using asyncio.loop.connect_read_pipe(), or the subprocess.PIPE constant.
stdout – If not None: either a file-like object representing the pipe to be connected to the subprocess’s standard output stream using asyncio.loop.connect_read_pipe(), or the subprocess.PIPE constant.
stderr – If not None: either a file-like object representing the pipe to be connected to the subprocess’s standard error stream using asyncio.loop.connect_read_pipe(), or one of subprocess.PIPE or subprocess.STDOUT constants.
limit – the buffer limit for StreamReader wrappers for Process.stdout and Process.stderr (if subprocess.PIPE is passed to stdout and stderr arguments).

Raises:

HelperExecutionError – if the subprocess exits with a returncode not in expected_returncodes.

Returns the tuple (returncode, stdout_data, stderr_data). returncode is the returncode (contained in expected_returncodes). stdout_data and stderr_data are bytes object with the received data from stdout and stderr, respectively.

replace_output(path, source):

Replace the - existing or non-existent - filesystem object path by source. path must be contained in a dependency of the tool instance.

path and source must be different filesystem objects.

After successful completion, path exists and source does not exist. If the parent directory of path does not exist, it is created (with all its parent directories).

The actual operation depends on the corresponding dependency role. If is it a dlb.ex.Tool.Output.RegularFile with replace_by_same_content = False and path and source both exist with the same content, path is no replaced and treated as unchanged.

If path is replaced, this is always done by an atomic operation. If it fails, path is either source afterwards or it does not exist.

Parameters:	path (`dlb.fs.Path` or anything a `dlb.fs.Path` can be constructed from) – a path of a future filesystem object in the managed tree source (`dlb.fs.Path` or anything a `dlb.fs.Path` can be constructed from) – a path of a filesystem object in the managed tree
Raises:	ValueError – if path is not a managed tree path contained in an explicit output dependency or source is not a working tree path of a filesystem object in the managed tree or in `.dlbroot/t/` of the management tree that is different from path.

Dependency classes¶

A dependency class is a subclass of Tool.Dependency. Its instances describe dependency roles (as attributes of a Tool).

The Tool.Dependency.validate() methods of dependency classes are used by tool instances to create concrete dependencies from their constructor arguments.

Each dependency role has an multiplicity specification:

An instance d of a dependency class D created with D(...) has a multiplicity of None which means that its concrete dependency must be a single object (its type depends on D only) or None.

An instance d of a dependency class D created with D[m](...) has a multiplicity of m which means that its concrete dependencies are a sequence of objects (their type depends on D only) or None. The accepted number of members is specified by m.

m can be any non-negative integer or any meaningful proper_slice (of non-negative integers). A number of members is accepted if and only if is either equal to m or contained in range(n + 1)[m].

Example:

class Tool(dlb.ex.Tool):
    # these are dependency roles of the tool 'Tool':
    include_search_paths = dlb.ex.Tool.Input.Directory[1:]()  # a sequence of at least one dlb.ex.Tool.Input.Directory
    cache_dir_path = dlb.ex.Tool.Input.Directory()  # a single dlb.ex.Tool.Input.Directory

tool = Tool(include_search_paths=['build/out/Generated/', 'src/Implementation/'])

# these are concrete dependencies of the tool instance 'tool':
tool.include_search_paths  # (Path('build/out/Generated/'), Path('src/Implementation/'))
tool.cache_dir_path  # (Path('build/out/Generated/'), Path('src/Implementation/'))

Dependency classes are organized in an a hierarchy according to their meaning to a tool by the means of the following abstract classes:

class Tool.Input¶

A Tool.Dependency that describes an input dependency of a tool.

The tool instance must be redone if it (e.g. the mtime of a file) has changed compared to the state before the last successful redo of the tool instance.

An redo must not modify it, successful or not (the same object can be an output dependency of the same tool instance though which can be modified).

class Tool.Output¶

A Tool.Dependency that describes an output dependency of a tool.

A successful redo must generate it (e.g. create a regular file).

These are all abstract classes and contain inner classes derived from them. Example: Tool.Output.Directory is a non-abstract dependency class derived from Tool.Output.

Note

dlb identifies filesystem objects by their managed tree path. It assumes that different managed tree paths point to different filesystem objects.

If a filesystem object serves as an output dependency of one tool instance and as an input dependency of another: Make sure both dependencies use the same path. A redo miss could happen otherwise.

You are always safe without hard links, symbolic links and case-insensitive filesystems.

Concrete dependency role classes support the following methods and attributes:

class Tool.Dependency(required=True, explicit=True)¶

If required is True, a concrete dependency of this dependency role will never be None.

If explicit is True, the concrete dependency can and must be fully defined when the tool instance is created. Otherwise, it cannot and must not be, but automatically assigned by Tool.run().

Each supported constructor argument is available as a property of the same name.

Raises:	DependencyError – if the arguments of the constructor do not match the declared dependency roles of the class

class Value¶

A (potentially abstract) class such that isinstance(v, Value) is True for each validated single value v of each instance t of this class.

This is the type of t.validate() if multiplicity is None and the type of each member of t.validate() otherwise.

validate(value)¶

Parameters:	value (Any type the concrete dependency can convert to T) – The concrete dependency to convert and validate except `None`
Returns:	The validated value of type T
Raises:	TypeError – If `multiplicity` is not `None` and value is not iterable or is a string

compatible_and_no_less_restrictive(other)¶

Is this dependency role an instance of the same class as other with a multiplicity and properties no less restrictive than the ones of other?

Parameters:	other (Tool.Dependency) – reference dependency role
Return type:	bool

multiplicity¶

The multiplicity of the dependency role.

Is None or a dlb.ex.mult.MultiplicityRange.

tuple_from_value(value)¶

Return value if multiplicity is None and a tuple of its members otherwise.

Example:

# returns a tuple of t.Value objects or raises an exception:
>>> v = t.tuple_from_value(t.validate(...))

Input dependency role classes¶

Dependency role class	Keyword arguments of constructor
Dependency role class	Name	Default value
`Tool.Input.RegularFile`	cls	`dlb.fs.Path`
`Tool.Input.NonRegularFile`	cls	`dlb.fs.Path`
`Tool.Input.Directory`	cls	`dlb.fs.Path`
`Tool.Input.EnvVar`	name
	restriction
	example

In addition to the keyword arguments of the specific constructors described here, all constructors also accept the keyword arguments of the constructor of Tool.Dependency.

class Tool.Input.RegularFile(cls=dlb.fs.Path)¶

Constructs a dependency role for a regular files, identified by their paths.

If a path is relative, is it treated as relative to dlb.ex.Context.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

Files outside the managed tree are assumed to remain unchanged between runs of dlb.

The validated value of a concrete dependency is the file’s path as an instance of cls if multiplicity is None and a tuple of the file’s paths otherwise.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    source_files = dlb.ex.Tool.Input.RegularFile[1:](cls=dlb.fs.NoSpacePath)
>>> tool = Tool(source_files=['src/main.cpp'])
>>> tool.source_files
(NoSpacePath('src/main.cpp'),)

Parameters:	cls (dlb.fs.Path) – class to be used to represent the path

class Value¶: Is dlb.fs.Path.

class Tool.Input.NonRegularFile(cls=dlb.fs.Path)¶

Constructs a dependency role for filesystem objects that are neither directories nor regular files, identified by their paths.

If a path is relative, is it treated as relative to dlb.ex.Context.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

Files outside the managed tree are assumed to remain unchanged between runs of dlb.

The validated value of a concrete dependency is the file’s path as an instance of cls if multiplicity is None and a tuple of the file’s paths otherwise.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    symlinks = dlb.ex.Tool.Input.NonRegularFile[:](cls=dlb.fs.NoSpacePath)
>>> tool = Tool(symlinks=['src/current'])
>>> tool.symlinks
(NoSpacePath('src/current'),)

Parameters:	cls (dlb.fs.Path) – class to be used to represent the path

class Value¶: Is dlb.fs.Path.

class Tool.Input.Directory(cls=dlb.fs.Path)¶

Constructs a dependency role for directories, identified by their paths.

If a path is relative, is it treated as relative to dlb.ex.Context.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

Directories outside the managed tree are assumed to remain unchanged between runs of dlb.

The validated value of a concrete dependency is the directory’s path as an instance of cls if multiplicity is None and a tuple of the directory’s paths otherwise.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    cache_directory = dlb.ex.Tool.Input.Directory(required=False)
>>> tool = Tool(cache_directory='/tmp/')
>>> tool.cache_directory
Path('tmp/')

Parameters:	cls (dlb.fs.Path) – class to be used to represent the path

class Value¶: Is dlb.fs.Path.

class Tool.Input.EnvVar(name, restriction, example)¶

Constructs a dependency role for a environment variable named name. It must not have a multiplicity (other than None).

If explicit is False, the value assign in the constructor of the tool instance is used for all future runs of the tool instance. Otherwise, the current value of the active context is used each time Tool.run() is called.

The value of the environment variable is valid if it a string that matches the regular expression restriction, or if it is None and required is False.

The validated value of a concrete dependency is a Value instance with the environment variable’s name and value.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    language = dlb.ex.Tool.Input.EnvVar(
>>>                   name='LANG',
>>>                   restriction=r'(?P<language>[a-z]{2})_(?P<territory>[A-Z]{2})',
>>>                   example='sv_SE')
>>>     flags = dlb.ex.Tool.Input.EnvVar(name='CFLAGS', restriction=r'.+', example='-Wall')
>>> tool = Tool(language='de_CH')  # use 'de_CH' as value of the environment variable for all
>>> tool.language.value['territory']
'CH'
>>> tool.flags
NotImplemented
>>> tool.run().flags.value  # assuming dlb.ex.Context.env['CFLAGS'] of '-O2'
'-O2'

Parameters:	restriction (str \| `typing.Pattern`) – regular expression example (str) – typical value of a environment variable, restriction must match this

class Value¶

A dataclasses.dataclass object with the following attributes:

name¶: The name of the environment variable, as in the corresponding concrete dependency.

raw¶: The value of the environment variable.

groups¶: The named groups of restriction of the corresponding concrete dependency when matched against raw.

Concrete output dependency role classes¶

Dependency role class	Keyword arguments of constructor
Dependency role class	Name	Default value
`Tool.Output.RegularFile`	cls replace_by_same_content	`dlb.fs.Path` `True`
`Tool.Output.NonRegularFile`	cls	`dlb.fs.Path`
`Tool.Output.Directory`	cls	`dlb.fs.Path`
`Tool.Output.Object`

In addition to the keyword arguments of the specific constructors described here, all constructors also accept the keyword arguments of the constructor of Tool.Dependency.

class Tool.Output.RegularFile(cls=dlb.fs.Path, replace_by_same_content=True)¶

Constructs a dependency role for regular files in the managed tree, identified by their paths.

If a path is relative, is it treated as relative to dlb.ex.Context.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

The validated value of a concrete dependency is the file’s path as an instance of cls if multiplicity is None and a tuple of the file’s paths otherwise.

If replace_by_same_content is False for a dependency role containing p, context.replace_output(p, q) in redo(..., context) does not replace p if p and q both exist as accessible regular files and have the same content.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    object_file = dlb.ex.Tool.Output.RegularFile(cls=dlb.fs.NoSpacePath)
>>> tool = Tool(object_file=['main.cpp.o'])
>>> tool.object_file
(NoSpacePath('main.cpp.o'),)

Parameters:	cls (dlb.fs.Path) – class to be used to represent the path

class Value¶: Is dlb.fs.Path.

class Tool.Output.NonRegularFile(cls=dlb.fs.Path)¶

Constructs a dependency role for filesystem objects in the managed tree that are neither directories nor regular files, identified by their paths.

If a path is relative, is it treated as relative to dlb.ex.Context.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

The validated value of a concrete dependency is the file’s path as an instance of cls if multiplicity is None and a tuple of the file’s paths otherwise.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    symlinks = dlb.ex.Tool.Output.NonRegularFile[:](cls=dlb.fs.NoSpacePath)
>>> tool = Tool(symlinks=['dist'])
>>> tool.symlinks
(NoSpacePath('src/current'),)

Parameters:	cls (dlb.fs.Path) – class to be used to represent the path

class Value¶: Is dlb.fs.Path.

class Tool.Output.Directory(cls=dlb.fs.Path)¶

Constructs a dependency role for directories in the managed tree, identified by their paths.

If a path is relative, is it treated as relative to dlb.ex.Context.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

The validated value of a concrete dependency is the directory’s path as an instance of cls if multiplicity is None and a tuple of the directory’s paths otherwise.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    html_root_directory = dlb.ex.Tool.Output.Directory(required=False)
>>> tool = Tool(html_root_directory='html/')
>>> tool.html_root_directory
Path('html/')

Parameters:	cls (dlb.fs.Path) – class to be used to represent the path

class Value¶: Is dlb.fs.Path.

class Tool.Output.Object¶

Constructs a dependency role for any Python object other than None and NotImplemented. It must not be explicit.

The validated value of a concrete dependency is a deep copy of the value.

class Value¶: Is typing.Any.

Exceptions¶

exception dlb.ex.NotRunningError¶: Raised, when an action requires an active context while dlb was not running.

exception dlb.ex.NoWorkingTreeError¶: Raised, when the working directory of the calling process is not a working tree’s root.

exception dlb.ex.ManagementTreeError¶: Raised, when an attempt to prepare or access the management tree failed.

exception dlb.ex.ContextNestingError¶: Raised, when some contexts were not properly nested. I.e. the calls of __exit__() did not occur in the opposite order of the corresponding calls of __enter__().

exception dlb.ex.WorkingTreeTimeError¶: Raised, when the working tree time behaved unexpectedly.

exception dlb.ex.ContextModificationError¶: Raised, when an environment variable dictionary object or a helper dictionary object is modified while its associated context is not the active context.

exception dlb.ex.WorkingTreePathError¶: Raised, when a path is not a working tree path with certain properties where it should be.

exception dlb.ex.DefinitionAmbiguityError¶: Raised at the definition of a subclass of Tool, when the location is unknown or another subclass of Tool was defined before at the same location.

exception dlb.ex.DependencyError¶: Raised when a running tool instance detects a problem with its dependencies before a redo.

exception dlb.ex.ExecutionParameterError¶: Raised when a running tool instance detects a problem with its execution parameters before a redo.

exception dlb.ex.RedoError¶: Raised when a running tool instance detects a problem with its dependencies during or after a redo.

exception dlb.ex.HelperExecutionError¶: Raised, when the execution of a dynamic helper file failed.

Table of Contents

Previous topic

`dlb.ex` — Execution contexts and dependency-aware tool execution¶

Context objects¶

Environment variable dictionary objects¶

Tool objects¶

Redo context¶

Dependency classes¶

Input dependency role classes¶

Concrete output dependency role classes¶

Exceptions¶

dlb.ex — Execution contexts and dependency-aware tool execution¶

Context objects¶

Environment variable dictionary objects¶

Tool objects¶

Redo context¶

Dependency classes¶

Input dependency role classes¶

Concrete output dependency role classes¶

Exceptions¶

`dlb.ex` — Execution contexts and dependency-aware tool execution¶