Reference

Terms

run of dlb
The execution of a part of a script inside the same root context.
script
A Python program that potentially creates an active context at least once.
context

An (execution) context describes how running tool instances shall interact with the execution environment outside the working tree and with each other.

It is represented as an instance of dlb.ex.Context used as a context manager.

active context
The innermost context, if any.
root context
The outermost context, if any.
tool

Describes the abstract behaviour and the way a tool is run by a subclass of dlb.ex.Tool.

It is concretized by instantiation.

tool instance

A (concrete) instance of an (abstract) tool with concrete dependencies.

Can be run in an active context (once or multiple times), which happens sequentially in the following stages:

  1. start of execution: call of r = t.start().
  2. redo check: decide whether a redo is necessary by inspecting its state and memorize it if it does.
  3. if redo necessary or explicitly requested:
    1. redo: generate all output dependencies.
    2. dependency finalization: update dependencies and their state in the run-database.
  4. completion of execution:
    1. successful when no redo was necessary or
    2. successful after awaiting on r has been completed the redo or
    3. unsuccessful after exception has been risen.

Between start and completion of execution the tool instance is said to be running.

dynamic helper

An external filesystem object in the managed tree or outside the working tree with a path to be determined at run-time.

A dynamic helper is identified by a relative path (non-directory or directory).

Typical example: globally installed executable (e.g. compiler) with an absolute path determined by a search for the first match in the environment variable PATH.

redo
The phase of the execution of a tool instance t that generates all its output dependencies.
redo necessity

A redo of a tool instance t is considered necessary if at least one of the following statements is true:

  • There is no last known successful redo of t (see below).
  • t has an output dependency that does not exist in the expected form.
  • t has a non-explicit input dependency set by the the last known successful redo of t that does not exist in the expected form.
  • t has an input dependency whose depended-upon state (e.g. a filesystem attribute or a value of environment variable) has changed since the start of the last known successful redo of t.
  • t has an execution parameter that has changed since the start of the last known successful redo of t.
  • The last known successful redo of t requested a redo for the next run.
  • The mtime, size, UID, GID, or set of filesystem permissions of any of the definition files for t in the managed tree has changed since the last known successful redo of t; a definition file for t is a regular file that contains the class definition of the class of t or one of its (direct or indirect) subclasses.

The last known successful redo of t is the last redo that was completed (without throwing an exception) for a tool instance of the same class and the same fingerprint as t according to the run-database.

redo miss

A redo miss of a tool instance is an undesirable situation that occurs when the redo check considers a redo not necessary while it actually is.

It is caused by bugs of dlb or dynamic helpers and violations of assumptions (e.g. on the behaviour of the filesystem).

working tree
The filesystem tree rooted at a directory (directly) containing a directory .dlbroot/ (a symbolic link to directory is not considered a directory here).
management tree

The filesystem tree rooted at .dlbroot/ in the root of the working tree.

Do not modify its content manually unless you are told so by dlb.

managed tree

The filesystem tree rooted at at the root of the working tree without the management tree. Contains the files involved in the build that are to be managed by dlb.

May and (typically will) be manually modified while there is no active context (e.g. by editing source files).

mtime
The time of last data modification of a filesystem object in the sense of ISO 1003.1-2008.
working tree time
The time according to the mtime of an imaginary filesystem object created in the working tree at a certain instant, assuming a single filesystem (A-F1).
mtime update
Setting the mtime of a filesystem object (on the same filesystem as the working tree) to the current working tree time.
working tree’s system time
The system time used as source for every mtime update of every filesystem object in the working tree, assuming there is one (A-T1).
effective mtime resolution

The effective mtime resolution for a filesystem object p is defined by the following thought experiment:

  • p is modified at ideal time t, resulting in a mtime m of p.
  • p is modified at ideal time t + dt, resulting in a mtime m + dm of p.
  • The effective mtime resolution for p is the minimum dm > 0 for any pair of t and dt > 0.

Nominal resolution of timestamps for some filesystems: XFS: 1 ns, NTFS: 100 ns, ext2: 1 s, FAT32: 2 s. The effective mtime resolution depends also on the filesystem driver and the operating system, but it can (nominally) not be finer that the timestamp resolution of the filesystem.

ideal time
The (strictly increasing) physical time at the place the dlb process is running.
non-upwards path

A relative path that has no prefix denoting its parent directory.

Examples: a/../b is an non-upwards path, a/../../a/b is not.

collapsable path

A path p of an existing filesystem object with the following property: No prefix path of p’ that ends with a component other than .. is the path of a symbolic link, where p’ is p with all . components removed.

Example: a/b/../c/.. is collapsable if and only if neither a/b/ nor a/b/../c is a symbolic link.

canonical-case path

A path whose components are all exactly as listed in their parent directory.

On a case-insensitive filesystem or directory, multiple paths that differ in case or character encoding can point to the same filesystem object. Only one of them is a canonical-case path.

normalized path
A path without .. components.
working tree path
The normalized path of an existing filesystem object relative to the working tree’s root.
managed tree path
A working tree path of a file system object in the managed tree.
run-database

The database in the management tree that stores information on the current and past runs of dlb, primarily related to dependencies.

Its removal (permitted when dlb is not running) typically leads to unnecessary redos in the following two runs.

true input dependency
A true input dependency of a tool instance t is an input of t that is not known to have been generated by a previously running tool instance (in the current or a previous run of dlb).
redo-safe
An action (e.g. a modification of the managed tree) is said to be redo-safe if it cannot lead to a redo miss for any tool instance in the current run or any future run of dlb.
benign managed tree modification

A modification of the managed tree is benign if it consist only of an arbitrary number of the following actions in any order:

  • Remove or create a filesystem object (this includes symbolic links and hard links)
  • Write to a regular file

Examples of modifications of the managed tree that are not benign managed tree modifications:

  • Replace a regular file by another one with mv (does not update mtime of the target)
  • Swap two directories
  • Set the mtime of a filesystem object to something different from the current working tree time

Layout of working tree

The directory .dlbroot/ is mandatory — it marks its parent directory as the root of a dlb working tree. Everything else is optional. dlb does never touch filesystem objects outside .dlbroot/ unless explicitly directed by a script to do so.

The directory .dlbroot/u/ is optional. dlb never touches its content. Each regular file or a symbolic links to a regular file in .dlbroot/u/ whose name ends in .zip should be a zip archive loadable by zipimport. You can place dlb as .dlbroot/u/dlb.zip in this directory (under version control). This makes the working tree almost self-contained; only an external Python interpreter is needed.

The file or directory .dlbroot/z is optional. dlb never touches it.

If you use Git for version control which does not support empty directories, add .dlbroot/z or any file in .dlbroot/u/.

The lines marked with * show filesystem objects only given as an example.

Before first run of a dlb script:

.dlbroot/
src/                    *
   a.c                  *
   a.h                  *
   b.c                  *
test/                   *
...

During a run of a dlb script:

.dlbroot/
    o                   empty regular file, used to probe the "current" mtime
    runs-*.sqlite       run-database
    t/                  temporary files
        a               *
        b               *
 src/                   *
   a.c                  *
   a.h                  *
   b.c                  *
 test/                  *
 out/                   *
   p                    *
 dist/                  *
 ...

After a successful run of a dlb script:

.dlbroot/
    o                   empty regular file
    runs-*.sqlite       run-database
 src/                   *
   a.c                  *
   a.h                  *
   b.c                  *
 test/                  *
 out/                   *
   a.c.o                *
   b.c.o                *
 dist/                  *
   p                    *
 ...

Top-level specification

Assumptions

An assumption (A-…) is a requirement for the intended behaviour of dlb that cannot be checked at runtime in a reliable and efficient way.

For every assumption at set of acceptable effects of its violation is given:

repair
The working tree needs manual repair.
obscure-fail
A build may fail in an obscure way. The diagnostic messages do neither clearly indicate the problem nor a way to fix it.
vulnerable
The build becomes vulnerable to attacks. Running tool instances might overwrite any filesystem object the process has permission to, potentially with information generated during the build or stored in the working tree.
redo-miss
A redo miss in the current or a future run of dlb can occur.
graceful-fail
A build may fail in a meaningful way. The diagnostic messages clearly indicate the problem or a way to fix it.
performance-degradation
A build may take significantly longer than expected (e.g. due to unnecessary redos).

Modification of the working tree

A-A1 (access to management tree)

The management tree is not modified while dlb is running and only as suggested by diagnostic messages of dlb.

Exceptions:

  • Modifications by dlb internals.
  • Modifications of u and z (including their content if they are directories) in the management tree.

Changing the absolute path of the working tree’s root is considered a modification of the management tree.

Acceptable when violated:

A-A2 (access to managed tree)

While a tool instance in running, the managed tree is modified only by running tool instances.

Changing the absolute path of the working tree’s root is considered a modification.

Acceptable when violated:

A-A3 (manual modification of mtime)

Except from modifications requested by a running tool instance, every modification of the mtime of a filesystem object in the working tree is an mtime update. [3]

Acceptable when violated:

A-A4

No part of the filesystem outside of the working tree is modified while a tool instance t is running unless it cannot affect the behaviour of t.

Acceptable when violated:

Filesystems behaviour of working tree

A-F1 (one filesystem)

Every filesystem object w/p, where w is the path of the working tree’s root and p is a relative path without .. components, resides on the same (local or remote) file system.

Acceptable when violated:

A-F2 (mtime update at creation)

Every creation of a filesystem object in the working tree updates its mtime. [5]

Acceptable when violated:

A-F3 (mtime update at write to regular file)

Every write to a regular file in the working tree updates its mtime as soon as it is completed. [5] [2]

Between start and completion of a write, a reader of the file may observe an intermediate state of the file’s content.

[--------------] content change

^              ^
start          mtime update (write complete)

-------------------> ideal time

Acceptable when violated:

A-F4 (mtime update for directory)

Every creation, removal, renaming, and attribute change of a filesystem object in the working tree updates the mtime of the (directly) containing directory. [5]

Acceptable when violated:

A-F5 (moving is atomic)

Moving a regular file, a directory, or a symbolic link in the working tree to a different directory in the working tree is possible in an reasonably secure, efficient and atomic operation that does not affect the moved object’s filesystem attributes (including mtime in full resolution).

Acceptable when violated:

A-F6 (moving makes invisible)

Immediately after a regular file, a directory, or a symbolic link in the working tree has been successfully moved to a different directory within the same working tree, no other process “sees” it in the original directory.

Acceptable when violated:

A-F7 (no corruption)

A filesystem object in the working tree is never corrupted (e.g. by failure of software, memory or power).

Acceptable when violated:

Timing and concurrency

A-T1 (working tree time exists)

The mtime of every filesystem object in the working tree is updated from the same system time (local or remote), the working tree’s system time.

Whenever an mtime update occurs for a filesystem object p at working tree’s system time tp and one occurs for a different filesystem object q at working tree’s system time tq, where p and q are in the working tree and tp is not later than tq, the mtime of p is not later than the mtime of q. [4]

Acceptable when violated:

A-T2 (working tree time mostly monotonically increasing)

With the exception of rare backward jumps, the working tree time is monotonically increasing.

The time between consecutive backward jumps is longer than the duration of a run of dlb.

Acceptable when violated:

A-T3 (effective mtime resolution)

The regular file o in the management tree has an effective mtime resolution no coarser than 100 ms.

Acceptable when violated:

A-T4 (working tree time of true input dependencies in the past)

The mtime of every filesystem object in the managed tree that is an true input dependency of a tool instance t is earlier than the time t starts running.

Acceptable when violated:

A violation of this assumption means that an mtime update after the run of dlb does not necessarily change the mtime, even with a monotonically increasing working tree time. To avoid redo misses, otherwise unnecessary redos may therefore have to be performed.

Dependencies

A-D1 (regular files)
Most of the filesystem objects in the managed tree that serve as input dependencies of tool instances are regular files.
A-D2 (no shared target of input and output dependencies)

If i and o are two different managed tree paths where i serves as an input dependency of a tool instance and o as an output dependency of a different tool instance, then i and o never point to the same existing filesystem object while one of the tool instances is running.

Acceptable when violated:

Possible cause of a violation of this assumption: Use of hard links, symbolic links or union mounting of file systems.

A-D3 (no implicit symbolic links in paths)

A filesystem object in the managed tree that serves as a dependency of a tool instance t does not have a parent directory p in its path that is a symbolic link, unless p is an input dependency of t and in the working tree.

Acceptable when violated:

Guarantees

A guarantee (G-…) is a specification of behaviour observable by the user.

Filesystems access in working tree

G-F1 (reserved names)

dlb does not create or access a file system object z (or its contents if it is a directory) in its management tree in any way.

dlb does not create or modify a directory u (or its contents) in its management tree.

Dependencies

G-D1 (no redo miss when working tree time monotonic)

A benign managed tree modification is redo-safe, provided the assumptions A-F1, A-F2, A-F3, A-F4 hold and the working tree time is monotonically increasing (at least since the oldest mtime of all filesystem objects that are true input dependencies of a tool instance).

This is true even when assumption A-A2 is violated.

G-D2 (no redo miss when file size changes)
Modifying the content of a regular file in the managed tree while a tool instance is running (in violation of A-A2) is redo-safe if it also changes the size of the regular file.
G-D3 (redo miss unlikely when modification intervals relatively long)

A benign managed tree modification is likely to be redo-safe, provided the assumptions A-F1, A-F2, A-F3, A-F4 hold and the “modification intervals are relatively long” for every filesystem object that is a true input dependency of a tool instance.

Here a modification interval of a filesystem object p is considered to be relatively long if it is unlikely that the working tree time at the ideal time t is the same as at t + T, where T is the ideal time between two consecutive mtime updates of p.

This is true even when assumption A-A2 is violated.

G-D4
When assumption A-T4 is violated at a certain time at the start of the “redo check” phase of a running tool instance, a redo of this tool instance is performed.

Timing and concurrency

G-T1 (active context exit)
An active context is not left as long as a tool instance is running in it.
G-T2 (root context exit)
A root context is not left other than by a raised exception before there has been a time window with the following property: The mtime of a regular file o in the management tree would have been different from the mtime of the last filesystem object modified by a running tool instance.
G-T3 (multiple dlb processes)
When multiple scripts are run by different processes on the same working tree, at most one of them is in an active context at the same time.
G-T4 (threads)
dlb does not (directly) create threads.

dlb — Command-line utility for convenience

The command-line utility dlb calls a dlb script; the real work is done entirely by the dlb script with the help of the package dlb.

dlb can save you some typing, however, with the following features:

  • Changes the current working directory to the working tree’s root from anywhere in the working tree.
  • Remembers command-line arguments of the last successful call.
  • Adds ZIP archives to the module search path.

Here is the output of dlb --help:

Run a dlb script in the root of the working tree that contains the current
working directory.

When called with '--help' as the first parameter, displays this help and exits.

When called with a least one parameter and the first parameter is not '--help',
the first parameter must be a dlb script as a normalized, non-upwards path
relative to the root of the working tree that does not start with '-'. '.py' is
appended if it does not end with '.py'. All other parameters are forwarded to
the dlb script.

When called without a parameter, the parameters from the last successful call of
this script with the same 'os.name' are used.

Each regular file or symbolic link to a regular file in the directory
'.dlbroot/u/' of the working tree whose name ends in '.zip' is added to the list
of module search paths of the Python interpreter.

Exit status:

   0  if called with '--help'
   1  if the specified dlb script could not be executed
   2  if no command-line arguments were given and the command-line arguments
      of the last successful call are not available
   e  otherwise, where e is the exit status of the specified dlb script
      (0 if it finished successfully)

Examples:

   dlb build/all         # executes the dlb script 'build/all.py' in the
                         # working tree's root
   dlb                   # same as dlb build/all if the previous call
                         # was successful
   PYTHONVERBOSE=1 dlb   # when called from a POSIX-compliant shell
   dlb --help

dlb version: ?.
Full documentation at: <https://dlb.readthedocs.io/>.

dlb — Version

dlb.__version__

The version of dlb as a non-empty string. Example: '1.2.3'.

Conforms to PEP 396, PEP 386 and PEP 440.

Contains the string '.dev' if and only if this a development version. For a development version, __version__ looks like this: '1.2.3.dev30+317f'.

dlb.version_info

The version of dlb as a non-empty tuple of non-negative integers and strings 'a' … 'z', similar to sys.version_info. Example: (1, 2, 3).

Contains at least 3 members, the first 3 of them integers. Contains more than 3 members if and only if this is an unreleased version. For an unreleased version, version_info looks like this: (1, 2, 3, 'c', 4).

Use it like this to compare (released) versions:

assert (1, 0, 3) <= dlb.version_info  < (2,)

Note

For a development version, version_info carries less information than __version__.

dlb.fs — Filesystem paths

This module provides classes to represent and access filesystem objects in a safe and platform-independent manner, most prominently dlb.fs.Path and its subclasses.

The interface is similar to pathlib. dlb.fs.Path is much faster than pathlib.PurePath. (Lossy) conversion from and to pathlib paths is supported.

Path objects

class dlb.fs.Path

A dlb.fs.Path represents the path of a filesystem object in a platform-independent manner and expresses whether the object is a directory or not. The represented path is called a directory path in the former case and a non-directory path in the latter case.

All represented paths are either absolute or relative to some other path (usually the working directory of the running process).

All operations on instances of dlb.fs.Path show the same behaviour on every platform (with the exception of native). Its instances are immutable and hashable (they can be used in sets or as keys in dictionaries).

If the path represented by a dlb.fs.Path is meaningful as a concrete path on the platform the code is running on, native returns it in the form of a dlb.fs.Path.Native instance, which can then be used to access the filesystem.

On all platform the following properties hold:

  • A path does not contain '\0' (NUL).
  • A component '..' means the parent directory of the path before it.
  • A component '.' means the directory of the path before it.
  • If the platform supports symbolic links, it resolves them as specified in ISO 1003.1-2008 (which means that '..' cannot be collapsed without knowing the filesystem’s content).
  • '/' is used as a path component separator.
  • A path is absolute if and only if it starts with '/'. a non-empty path with all such components removed is equivalent to the original one.
  • A sequence of two or more consequent '/' is equivalent to a single '/', except at the beginning of the path.
  • At the beginning of the path:
    • Three or more consequent '/' are equivalent to a single '/'
    • Exactly two consequent '/' (followed be a character other than '/') means that the following component is to be interpreted in a implementation-defined manner (e.g. ISO 1003.1-2008 or UNC paths)

dlb.fs.Path instances are comparable with each other by ==, < et cetera which define a total ordering. Comparison is case-sensitive and component-wise. For two dlb.fs.Path instances p and q:

  • p = q if and only if p.components == q.component and p.is_dir == q.is_dir.
  • p < q if p.components < q.component (a true prefix of a path is smaller than the prefixed path, and every relative path is smaller than every absolute path).
  • p < q if p.components == q.component and p.is_dir < q.is_dir (a non-directory path is smaller than an otherwise equal directory path).

Usage example:

>>> p = dlb.fs.PortablePath('a/b/c/') / 'x/y'
>>> p.relative_to('a/b/)
PortablePath('c/x/y')

>>> dlb.fs.Path('x/y/z.tar.gz')[:-2]
Path('x/')

>>> dlb.fs.Path('x/y/z.tar.gz').components[-1]
'z.tar.gz'

The dlb.fs.Path class supports the following methods and attributes:

Path(path, *, is_dir=None)

Construct a path from another path, string, or path component sequence:

  • If path is a string, it is interpreted as the string representation of a path in POSIX style with / as a component separator. It must not be empty.
  • If path is a sequence, it is interpreted as a path component sequence after all its members have been converted to str (see components).
  • If path is an instance of dlb.fs.Path, it is copied. This is very fast..
  • If path is an instance of pathlib.PurePath (including pathlib.Path), it is interpreted according to its meaning expressed by the class (a pathlib.PureWindowsPath instance is interpreted as an an MS Windows path, for example). If path is neither absolute nor relative, a ValueError exception is raised.
  • If path is an instance of dlb.fs.Path.Native, its pathlib.Path representation is used. native of the constructed path is equal to path.

If is_dir is True, the path is considered a directory path irrespective of path.

If is_dir is False, the path is considered a non-directory path irrespective of path. However, if path represents '.' or ends with a '..' component, a ValueError exception is raised.

If is_dir is None, the ending of path determines whether it is considered a directory path or not. If path is a string that ends with '/' or if the last component of path is '.' or '..' component, it is considered a directory path.

Parameters:
  • path (str | dlb.fs.Path | dlb.fs.Path.Native | pathlib.PurePath | sequence of str) – path object, portable string representation, or path component sequence
  • is_dir (NoneType | bool) – True if this is a directory path, False if not and None for derivation from path
Raises:

Examples:

>>> p = dlb.fs.Path('a/b/').is_dir()
True

>>> p = dlb.fs.Path(pathlib.PureWindowsPath('C:\\Windows'), is_dir=True)
>>> p
Path('/C:/Windows/')
>>> p.is_dir()
True

>>> p = dlb.fs.Path('x/y/..', is_dir=False)
Traceback (most recent call last):
...
ValueError: cannot be the path of a non-directory: 'x/y/..'
is_dir()
Returns:True if and only if this represents a directory path.
Return type:bool
is_absolute()
Returns:True if and only if this represents an absolute path.
Return type:bool

Note

While POSIX considers paths starting with exactly two '/' not as absolute paths, this class does (and so does pathlib).

is_normalized()
Returns:True if and only if this represents a normalized path (without '..' components)
Return type:bool
relative_to(other, *, collapsable=False)

Return a version of this path relative to the directory path represented by C(other) where C is the class of this path.

If collapsable is False, this path must be a prefix of other. If collapsable is True, other is treated as a collapsable path and the minimum of necessary .. components is prepended.

Parameters:
Returns:

relative path
Return type:

class of this object
Raises:
  • ValueError – if C(other) is not a directory path
  • ValueError – if one of this path and C(other) is a relative path and the other one is an absolute path
  • ValueError – if collapsable is False and this path is not a prefix of C(other)
with_appended_suffix(suffix)

Return a version of this path with suffix appended to its last component.

Parameters:suffix (str) – suffix that does not change the meaning of the path (number of components or the containing directory)
Return type:class of this object
Raises:ValueError – if this is '.' or '..' or suffix does contain '/' or '\0' (NUL)
with_replacing_suffix(suffix)

Return a version of this path with the extension suffix of its last component replaced by suffix.

A string s is an extension suffix of a path component c if and only if

  • c ends with s and s differs from c, and
  • s starts with '.' and contains exactly one '.'.
Parameters:

suffix (str) – suffix that does not change the meaning of the path (number of components or the containing directory)
Return type:

class of this object
Raises:
  • ValueError – if this is '.' or '..' or suffix does contain '/' or '\0' (NUL)
  • ValueError – if the last component does not have an extension suffix.
as_string()

The unique representation as a POSIX-style string.

For every dlb.fs.Path instance p and s = p.as_string(), the following holds:

  • s is not empty
  • s[0] is '/' if and only if p is an absolute path
  • s[-1] is '/' if and only if p is a directory path
  • dlb.fs.Path(s) == p
Return type:str
iterdir(*, name_filter='', is_dir=None, recurse_name_filter=None, follow_symlinks=True, cls=None)

Yield all path objects of the directory contents denoted by this path and matched by the name filters. The paths are duplicate-free and in a platform-independent and deterministic order but not necessarily sorted. Their class is the class of this object if cls is None and cls otherwise.

The path p of an existing filesystem object is eventually yielded if and only if

  • its name matches the name filter name_filter and
  • is_dir is None or is_dir is False and p is a non-directory or is_dir is True and p is a directory
  • it is contained in a matched directory.

A directory is a matched directory if and only if it is the directory d denoted by this path or a direct subdirectory of a matched directory whose name matches the name filter recurse_name_filter. If follow_symlinks is True, a symbolic link to an existing directory is considered a direct subdirectory of the director containing the symbolic link. If follow_symlinks is False or the target of the symbolic link does not exist, it is considered a non-directory.

name_filter and recurse_name_filter are name filters. A name filter can be

  • None — no name matches this filter
  • a callable c accepting exactly one argument — a name n matches this filter if and only if bool(c(n)) is True
  • a compiled regular expression r — a name n matches this filter if and only if r.fullmatch(n)) is not None
  • a non-empty regular expression string s— a name n matches this filter if and only if re.compile(s).fullmatch(n)) is not None
  • an empty string — every name matches this filter

Example:

for p in dlb.fs.Path('src/').iterdir(name_filter=r'(?i).+\.cpp',
                                     recurse_name_filter=lambda n: '.' not in n):
    ...
Return type:

cls | class of this object
Raises:
iterdir_r(*, name_filter='', is_dir=None, recurse_name_filter=None, follow_symlinks=True, cls=None)

Like iterdir(), but all yielded paths are relative to this path.

list(*, name_filter='', is_dir=None, recurse_name_filter=None, follow_symlinks=True, cls=None)

Return all paths yielded by iterdir() as a sorted list.

Example:

>>> dlb.fs.NoSpacePath('src/').list(name_filter=r'(?i).+\.cpp')
[NoSpacePath('src/Clock.cpp'), NoSpacePath('src/main.cpp')]
list_r(*, name_filter='', is_dir=None, recurse_name_filter=None, follow_symlinks=True, cls=None)

Return all paths yielded by iterdir_r() as a sorted list.

Example:

>>> dlb.fs.NoSpacePath('src/').list(name_filter=r'(?i).+\.cpp')
[NoSpacePath('Clock.cpp'), NoSpacePath('main.cpp')]
find_latest_mtime(*, name_filter='', is_dir=None, recurse_name_filter=None, follow_symlinks=True, cls=None)

Return the path of the filesystem object with the latest mtime among all paths yielded by iterdir() or None if there is no such path.

If more than one filesystem object with a path yielded by iterdir() has the same mtime, one of them is chosen in a platform-independent and deterministic manner.

propagate_mtime(*, name_filter='', is_dir=None, recurse_name_filter='')

For each path yielded by iterdir(..., follow_symlinks=False) with an mtime later than the mtime of its containing directory, update the mtime of the containing directory (and its parents up to and including this directory if the resulting mtime is later than their mtime). Return the resulting mtime of this directory in nanoseconds or None if it was not updated.

Note

In contrast to iterdir() the default value for recurse_name_filter is '' which means that by default no subdirectory is ignored.

components

The path component sequence of this path as a tuple of strings.

The first element is the root component. It is '/' or '//' if the path is absolute and '' otherwise. All other elements are the path’s non-root components. They do not contain '/' and are not '.' or '..'.

Example:

>>> dlb.fs.Path('/usr/bin/python3').component
('/', 'usr', 'bin', 'python3')
>>> dlb.fs.Path('x/y/').component
('', 'x', 'y')
>>> dlb.fs.Path('.').component
('',)

This attribute cannot be written.

Return type:tuple of str
parts

The path component sequence of this path with an empty root-component removed:

>>> dlb.fs.Path('/usr/bin/python3').parts
('/', 'usr', 'bin', 'python3')
>>> dlb.fs.Path('x/y/').parts
('x', 'y')
>>> dlb.fs.Path('.').parts
()

This attribute cannot be written.

Return type:tuple of str
native

This path as a (non-portable) native path.

Use it to access the filesystem:

p = dlb.fs.Path('/usr/bin/')
... = os.stat(p.native)       # p.native is a os.PathLike
... = os.stat(str(p.native))
... = p.native.raw.stat()     # pathlib.Path (slower)

Note

Whether a path is a directory path can only by reconstructed from its native path if it ends with a '.' or '..' component.

This attribute cannot be written.

Return type:dlb.fs.Path.Native
Raises:ValueError – if this path is not representable as Path.Native
pure_posix

This path as a pathlib.PurePosixPath:

>>> dlb.fs.Path('/usr/bin/').pure_posix
PurePosixPath('/usr/bin')

This attribute cannot be written.

Return type:pathlib.PurePosixPath
pure_windows

This path as a pathlib.PureWindowsPath:

>>> Path('/C:/Program Files/').pure_windows
PureWindowsPath('C:/Program Files')

This attribute cannot be written.

Return type:pathlib.PureWindowsPath
path[key]

Returns the path represented by a path.parts[key] where key is a slice of indices with a positive slice step.

The resulting path is absolute (with the same root-component) if and only if the slice starts at 0 and path is an absolute path.

The resulting path is a non-directory path if and only if it contains the last component and if path is a non-directory path.

Raises TypeError if key is not a slice. Raises ValueError if path is an absolute path and key is an empty slice.

Examples:

>>> dlb.fs.Path('src/comp/lib/Core.cpp')[:-2]
Path('src/comp/'

>>> dlb.fs.Path('src/comp/..')[1:-1]
Path('comp/')
path / other

Concatenates the paths path and other.

path must be a dlb.fs.Path. other can be a dlb.fs.Path or anything a dlb.fs.Path can be constructed from.

The returned path result is of type C and is constructed from path.components + C(other).parts where C is the class of path. result.is_dir is C(other).is_dir.

Raises ValueError if path is not a directory path. Raises ValueError if C(other) is an absolute path.

Examples:

>>> dlb.fs.NoSpacePath('path/to/glory/') / 'Stanley_Kubrick'
NoSpacePath('path/to/glory/Stanley_Kubrick')

>>> dlb.fs.NoSpacePath('path/to/glory/') / 'Stanley Kubrick'
Traceback (most recent call last):
...
ValueError: invalid path for 'NoSpacePath': 'Stanley Kubrick' (must not contain space)

>>> title = 'Paths to Glory'; year = 1957
>>> dlb.fs.Path('films/') / f'{title} ({year})/' / 'video.mp4'
Path('films/Paths to Glory (1957)/video.mp4')
class Path.Native

A native path whose instances can be used much like instances of pathlib.Path. It implements the os.PathLike protocol.

For each subclass P of dlb.fs.Path there is a corresponding subclass P.Native that imposes the same restrictions on its representable paths as P.

If Q is a subclass of P and P is a subclass of dlb.fs.Path, then Q.Native is a subclass of P.Native.

Example (on a POSIX system):

>>> dlb.fs.NoSpacePath.Native('/tmp/x y')
Traceback (most recent call last):
...
ValueError: invalid path for 'NoSpacePath': '/tmp/x y' (must not contain space)

In contrast to pathlib.Path, conversion to string is done in a safe way: relative paths are guaranteed to start with '.'.

Example (on a POSIX system):

>>> str(dlb.fs.Path.Native('-rf'))
'./-rf'

Instances of dlb.fs.Path.Native and its subclasses should not be constructed directly but by accessing dlb.fs.Path.native.

Example (on a POSIX system):

with open(dlb.fs.NoSpacePath('/tmp/x/a').native) as f:
    ... = f.read()
raw

This path as a pathlib.Path. Use it to access the filesystem in an object-oriented manner:

p = dlb.fs.Path('/usr/bin/')
... = p.native.raw.stat()

This attribute cannot be written.

Constructing a pathlib.Path is an expensive operation. For performance-critical tasks, use p.native and functions for string-like paths instead: e.g. os.path.isdir(p.native) instead of p.native.raw.is_dir().

Restricting paths

Trying to construct a dlb.fs.Path from an invalid value raises an ValueError exception. Additional restrictions to the set of valid values can be imposed on instances of dlb.fs.Path by subclassing.

A subclass of dlb.fs.Path should only implement check_restriction_to_base(). p.check_restriction_to_base() should raise ValueError if it considers p.components or p.is_dir as invalid:

class NoBullshitPath(Path):

    def check_restriction_to_base(self, components_checked: bool):
        # This is called during construction of a NoBullshitPath after all
        # attributes have been assigned and the check_restriction_to_base() of all its
        # base classes have been called successfully.
        # *components_checked* is True if this method was called before on a path p
        # whose p.components contained all members of self.components.

        if not components_checked and any(c == 'bullshit' for c in self.components):
            raise ValueError('no bullshit, please')

NoBullshitPath('thats/bullshit/)  # raises ValueError

Inheritance diagram of dlb.fs.Path, dlb.fs.RelativePath, dlb.fs.AbsolutePath, dlb.fs.NoSpacePath, dlb.fs.PosixPath, dlb.fs.PortablePosixPath, dlb.fs.PortableWindowsPath, dlb.fs.WindowsPath, dlb.fs.PortablePath

class dlb.fs.RelativePath

A dlb.fs.Path that represents a relative path.

class dlb.fs.AbsolutePath

A dlb.fs.Path that represents an absolute path.

class dlb.fs.NormalizedPath

A dlb.fs.Path that represents a normalized path (without '..' components).

class dlb.fs.NoSpacePath

A dlb.fs.Path whose components do not contain ' '.

class dlb.fs.PosixPath

A dlb.fs.Path that represents a POSIX-compliant (ISO 1003.1-2008) paths in its least-constricted form.

Every non-empty string that does not contain '/' or U+0000 (NUL) is a valid component.

For every path prefix (in the POSIX sense) {NAME_MAX} and {PATH_MAX} are considered unlimited.

Relevant parts of ISO 1003.1-2008:

  • section 4.12 Pathname Resolution
  • section 4.5 File Hierarchy
  • section 4.6 Filenames
  • section 4.7 Filename Portability
  • section 3.267 Pathname
  • section 3.269 Path Prefix
  • limits.h
class dlb.fs.PortablePosixPath

A dlb.fs.PosixPath that represents a POSIX-compliant (ISO 1003.1-2008) path in its strictest form. Any path whose support is not required by POSIX or is declared as non-portable is considered invalid.

A component cannot be longer than 14 characters, which must all be members of the Portable Filename Character Set.

The length of the string representation of the path is limited to 255 characters.

No absolute path prefix other than '/' is allowed (because implementation-defined).

class dlb.fs.WindowsPath

A dlb.fs.Path that represents a Microsoft Windows-compliant file or directory path in its least-constricted form, which is either relative or absolute and is not a reserved non-directory path (e.g. NUL).

It cannot represent incomplete paths which are neither absolute nor relative to the current working directory (e.g. C:a\b and \\name). It cannot represent NTFS stream names, Win32 file namespaces, or Win32 device namespaces.

class dlb.fs.PortableWindowsPath

A dlb.fs.WindowsPath that represents a Microsoft Windows-compliant path in its strictest form.

A component cannot end with ' ' or '.' (except '.' and '..') and cannot be longer than 255 characters. The path cannot not be longer than 259 characters.

class dlb.fs.PortablePath

dlb.di — Line-oriented hierarchical diagnostic messages

In contrast to the logging module, this module focuses on hierarchical structure and unambiguity. Absolute time information (date and time of day) is not output in favour of high-resolution relative times. The output is compact, line-oriented, and well readable for humans.

Each message has an associated level, e.g. WARNING, with the same meaning and numerical value as in the logging module. The higher the associated numeric value, the more important the message is considered:

Level Numeric value
CRITICAL 50
ERROR 40
WARNING 30
INFO 20
DEBUG 10

Messages below a global message threshold are not output. The message threshold can be changed any time. It is initialised to INFO if sys.flags.verbose is False and 1 otherwise.

Messages can be nested with message clusters. In the output, the nesting level of a message is expressed by the indentation of its lines.

A precise syntax in enforced to make the output suitable for incremental parsing (e.g. from a named pipe) with the help of simple regular expressions. [1] Any Unicode character except characters from the range U+0000 to U+001F (ASCII control characters) can be used in messages as long as the syntax is not violated.

To output a message, call dlb.di.inform() or enter a context manager instance of dlb.di.Cluster().

All messages are output to the output file assigned at the time the output occurs. When first imported, dlb.di assigns the standard error sys.stderr. It can be changed by calling dlb.di.set_output_file() at any time.

Example

import dlb.di
...

with dlb.di.Cluster(f"analyze memory usage\n    see {logfile.as_string()!r} for details", is_progress=True):
   ram, rom, emmc = ...

   dlb.di.inform(
       f"""
       in use:
           RAM:\t {ram}\b kB
           ROM (NOR flash):\t {rom}\b kB
           eMMC:\t {emmc}\b kB
       """)

   if rom > 0.8 * rom_max:
       dlb.di.inform("more than 80% of ROM used", dlb.di.WARNING)

This will generate the following output:

I analyze memory usage...
  | see 'out/linker.log' for details
  I in use:
    | RAM:              12 kB
    | ROM (NOR flash): 108 kB
    | eMMC:            512 kB
  W more than 80% of ROM used
  I done.

Syntax

Each message starts with a capital letter after indentation according to its nesting level (2 space characters per level) and ends with a '\n' after a non-space character. It can consist of multiple lines: an initial line followed by any number of continuation lines with the same indentation as the initial line. The lines are separated by '␣\n' ('␣' means the character U+0020):

message             ::=  single_line_message | multi_line_message
single_line_message ::=  initial_line '\n'
multi_line_message  ::=  initial_line '␣\n' (continuation_line '␣\n')* continuation_line '\n'

The initial line carries the essential information. Its first letter after the indentation denotes the level of the message: the first letter of the standard names of the standard loglevels of the logging module.

initial_line    ::=  indentation level_indicator '␣' summary summary_suffix
indentation     ::=  '␣␣'*
level_indicator ::=  'C' | 'D' | 'E' | 'I' | 'W'
summary         ::=  summary_first_character [ message_character* summary_last_character ]
summary_suffix  ::=  [ progress_suffix ] [ '␣' relative_time_suffix ]
progress_suffix ::=  '.' | '...'

The timing information is optional and can be enabled per message. It contains the time elapsed in seconds since the first time a message with enabled timing information was output. Later outputs of timing information never show earlier times. The number of decimal places is the same for all output timing information on a given platform and is at most 6.

relative_time_suffix      ::=  '[+' time_since_first_time_use ']'
time_since_first_time_use ::=  decimal_integer [ '.' decimal_digit decimal_digit* ] 's'

continuation_line           ::=  indentation continuation_line_indicator message_character*
continuation_line_indicator ::=  '␣␣|␣'

summary_first_character ::=  any summary_last_character except "'" (U+0027) and '|' (U+007C)
summary_last_character  ::=  any message_character except '␣' (U+0020), '.' (U+002E) and ']' (U+005D)
message_character       ::=  any Unicode character except from the range U+0000 to U+001F
decimal_integer         ::=  nonzero_decimal_digit decimal_digit*
nonzero_decimal_digit   ::=  '1' | ... | '9'
decimal_digit           ::=  '0' | nonzero_decimal_digit

Module content

dlb.di.DEBUG
dlb.di.INFO
dlb.di.WARNING
dlb.di.ERROR
dlb.di.CRITICAL

Positive integer representing a message level of the same name.

In contrast to logging, these attributes are not meant to be changed by the user. Use them to define your own positive integers representing levels like this:

... = dlb.di.INFO + 4  # a level more important than INFO but not yet a WARNING
dlb.di.set_output_file(file)

Set the output file for all future outputs of this module to file and return the previous output file.

file is used by dlb.di exclusively like this: file.write(message). message is a (non-empty) string that can be produced by the production rule message. Its first character is meant to represent the first character of a line.

Parameters:file (an object with a write(string) method) – new output file
Returns:the previous value, an object with a write attribute
Raises:TypeError – if file has no write attribute
dlb.di.set_threshold_level(level)

Set the level threshold for all future messages to level.

Every message with a level below level will be suppressed.

Parameters:

level (int) – new (positive) level threshold
Raises:
dlb.di.is_unsuppressed_level(level)

Is a message of level level unsuppressed be the current level threshold?

Parameters:level (int) – (positive) level
Return type:bool
dlb.di.get_level_indicator(level)

Return a unique capital ASCII letter, representing the lowest standard level not lower than level.

Example:

>>> dlb.di.get_level_indicator(dlb.di.ERROR + 1)
'E'
Parameters:level (int) – level not lower that DEBUG
dlb.di.format_time_ns(time_ns)

Return a string representation for a time in seconds, rounded towards 0 approximately to the resolution of time.monotonic_ns().

The number of decimal places is fixed for all calls. It is a platform-dependent value in the range of 1 to 6.

Parameters:time_ns (int) – time in nanoseconds
Return type:str
dlb.di.format_message(message, level)

Return a formatted message with aligned fields, assuming nesting level level.

First, empty lines are removed from the beginning and the end of message and trailing white space characters is removed from each line. After that, the first line must not start with '␣', "'", "|", '.' or "]". If must not end with "." or "]". Each non-empty line after the first line must start with at least 4 space characters after than the indentation of the first line. Example: If the first line is indented by 8 space characters, each following non-empty line must start with at least 12 space characters.

message can contain fields. A field is declared by appending '\t' or '\b'. A field whose declaration ends with '\t' is left aligned, one whose declaration ends with '\t' is right aligned over all lines of the message. In the return value the '\t' or '\b' are not present but their “positions” are aligned over all lines of the message.

Examples:

>>> dlb.di.format_message('\njust a moment! ', dlb.di.WARNING)
'W just a moment!'

>>> dlb.di.format_message(
...   """
...   summary:
...       detail: blah blah blah...
...       see also here
...   """, dlb.di.INFO)
'I summary: \n  | detail: blah blah blah... \n  | suggestion'

>>> m = ''.join(f"\n    {n}:\t {s} =\b {v}\b{u}" for n, s, v, u in metrics)
>>> print(dlb.di.format_message('Halstead complexity measures:' + m, dlb.di.INFO))
I Halstead complexity measures:
  | volume:               V =   1.7
  | programming required: T = 127.3 s
  | difficulty:           D =  12.8
Parameters:
  • message (str) – message (to be formatted)
  • level (int) – (positive) level
Returns:

formatted message conforming to message after appending a single '\n'
Return type:

str
Raises:

ValueError – if message would violate message or if level is invalid
dlb.di.inform(message, *, level: int = INFO, with_time: bool = False)

If level is not suppressed, output a message to the output file after the title messages of all parent Cluster instances whose output was suppressed so far.

message is formatted by format_message() and indented according the nesting level. If with_time is True, a relative_time_suffix for the current time is included.

class dlb.di.Cluster(message, *, level=INFO, is_progress=False, with_time=False)

A message cluster with message as its title.

When used as a context manager, this defines a inner message cluster with message as its title; entering means an increase of the nesting level by 1.

With is_progress set to False, the output when the context is entered is the same as the output of inform() would be with the same parameters.

With is_progress set to True, a progress_suffix '...' is included in the message when the context is entered. In addition, a message 'done. or 'failed with E.' is output when the context is exited without or with an exception, respectively. See Example.

dlb.cf — Configuration parameters

To change the behaviour of dlb, change the values of the following variables.

dlb.cf.latest_run_summary_max_count

Number of dlb runs to summarize as an integer.

When > 0, a summary of the latest latest_run_summary_max_count dlb runs is output as a diagnostic message when a root context exits.

dlb.cf.max_dependency_age

The maximum age of dependency information in the run-database as a datetime.timedelta object.

Run and dependency information older than max_dependency_age is removed when a root context is entered.

max_dependency_age > datetime.timedelta(0) must be True.

dlb.cf.execute_helper_inherits_files_by_default

Default value for output files of dlb.ex.RedoContext.execute_helper() etc..

False means: Output is suppressed by default. True means: Output file is inherited from the Python process by default.

dlb.cf.level.run_preparation
dlb.cf.level.run_serialization
dlb.cf.level.redo_necessity_check
dlb.cf.level.redo_reason
dlb.cf.level.redo_suspicious_reason
dlb.cf.level.redo_preparation
dlb.cf.level.redo_start
dlb.cf.level.redo_aftermath
dlb.cf.level.helper_execution
dlb.cf.level.output_filesystem_object_replacement
dlb.cf.level.run_summary

Message level (a positive integer like dlb.di.INFO) to be used for all diagnostic messages of the category named by the attribute’s name.

dlb.ex — Execution contexts and dependency-aware tool execution

Context objects

An (execution) context describes how running tool instances shall interact with the execution environment outside the working tree and with each other. E.g:

It also helps with filesystem abstraction (e.g. working tree time or working tree paths).

A context is represented as an instance of dlb.ex.Context used as a context manager. The context is entered with the call of __enter__ and exited with the return of __exit__.

Contexts can be nested:

import dlb.ex

# no active context

with dlb.ex.Context():                # A: root context, outer context of B, C, D
    # A is the active context
    with dlb.ex.Context():            # B: inner context of A, outer context of C
        # B is the active context
        with dlb.ex.Context():        # C: inner context of A, B
           # C is the active context
    with dlb.ex.Context():            # D: inner context of A
        # D is the active context
    # A is the active context

# no active context

Combine contexts with message clusters to describe what happens in the context:

with dlb.di.Cluster('this happens in the context'), dlb.ex.Context():
    ...
class dlb.ex.Context(*, path_cls=dlb.fs.Path, max_parallel_redo_count=1, find_helpers=None)

An instance does nothing unless used as a context manager.

When used as a context manager, it embodies an (execution) context and activates it:

  1. a root context, if dlb is not yet running;
  2. an inner context of the active context, otherwise.

When a root context is entered, the working directory of the Python process must be a working tree’s root whose absolute path does not contain unresolved symbolic link.

When a context (root or not) is entered, the path of the working tree’s root must be representable as as path_cls. This allows you to impose restrictions on the accepted paths.

If find_helpers is None for a root context, True is used instead. If find_helpers is None for an active context that is not the root context, find_helpers of the root context is used.

Parameters:
  • path_cls (dlb.fs.Path) – the subclass of dlb.fs.Path to be used to represent the working tree’s root
  • max_parallel_redo_count (int) – maximum number of redos started in this context than can be pending at the same time
  • find_helpers (None | bool) – are dynamic helpers not defined explicitly to be searched for in executable_search_paths?
Raises:

TypeError – if path_cls is not a subclass of dlb.fs.Path

Entering or exiting a context may raise the following exceptions:

exception meaning when
NoWorkingTreeError the working directory is not a working tree’s root entering root context
ManagementTreeError the management tree cannot be setup inside the working tree
ValueError the working tree’s root path violates the requested restrictions entering (any) context
ContextNestingError the contexts are not properly nested exiting (any) context
WorkingTreeTimeError working tree time behaved unexpectedly exiting root context

Note

You probably want to use the active context only. Use dlb.ex.Context.active to get it.

The Context class supports the following methods and attributes:

active

The active context.

This is a class attribute to be used like this:

with dlb.ex.Context:
    ... = dlb.ex.Context.active.root_path
    ... = dlb.ex.Context.active.find_path_in(...)
Raises:NotRunningError – if dlb is not running).
root_path

The absolute path of the working tree’s root.

It is an instance of Context.active.path_cls and is representable as an instance of path_cls of the active context and every possible outer context.

Raises:NotRunningError – if dlb is not running).
executable_search_paths

A duplicate-free tuple of absolute directory paths where this process should look for executables according to the operating system.

It is compiled from the members os.get_exec_path() when the root context is entered.

Raises:NotRunningError – if dlb is not running).
path_cls

The subclass of dlb.fs.Path as defined in the constructor.

Raises:NotRunningError – if dlb is not running).
max_parallel_redo_count

The maximum number of redos started in this context than can be pending at the same time as defined in the constructor.

Raises:NotRunningError – if dlb is not running).
find_helpers

Find dynamic helpers not defined explicitly in executable_search_paths? As defined in the constructor.

Raises:NotRunningError – if dlb is not running).
env

The environment variable dictionary object with this context as its associated context.

Raises:NotRunningError – if dlb is not running).
helper

The dynamic helper dictionary object with this context as its associated context.

The dynamic helper dictionary object maps dynamic helpers to absolute paths, either explicitly or implicitly with the help of find_path_in().

If the active context and the root context both have find_helpers == False and no paths was explicitly assigned to the dynamic helper p in the active context or one of its outer contexts, a look-up with dlb.ex.Context.active.helper[p] performs a search with dlb.ex.Context.active.find_path_in(p). (Each such search is performed only once for a given path; the result is stored.)

Examples:

>>> dlb.ex.Context.active.helper['gcc']
Path('/usr/bin/gcc')

>>> dlb.ex.Context.active.helper['gcc'] = '/usr/local/bin/my-very-own-very-special-gcc'  # set the path explicitly
>>> dlb.ex.Context.active.helper['gcc']
Path('/usr/local/bin/my-very-own-very-special-gcc')

>>> dlb.ex.Context.active.helper['tmp/'] = 'out/t/'  # relative path: relative to the working tree's root path
>>> dlb.ex.Context.active.helper['tmp/']  # with '/home/schmutzli/projects/esel' as the working tree's root
Path('/home/schmutzli/projects/esel/out/t')
Raises:NotRunningError – if dlb is not running).
find_path_in(path, search_prefixes=None)

Find the first existing and accessible path in search_prefixes and return its absolute path. Returns None if path is not found in search_prefixed.

If path is a dlb.fs.Path with path.is_dir() == True, existing non-directories in search_prefixes are ignored. If path is a dlb.fs.Path with path.is_dir() == False, existing directories in search_prefixes are ignored.

Relative paths in search_prefixes are treated as relative to root_path.

If search_prefixes is None, executable_search_paths is used instead.

Does not raise OSError.

Parameters:
  • path (dlb.fs.Path or anything a dlb.fs.Path can be constructed from) – the relative path to find
  • search_prefixes (an iterable other than str or bytes or None) – paths of directories to search in
Returns:

an absolute path or None.
working_tree_time_ns

The current working tree time in nanoseconds as an integer.

Raises:NotRunningError – if dlb is not running).
temporary(*, suffix='', is_dir=False)

Return a context manager, representing a temporary regular file (for is_dir = False) or a temporary directory (for is_dir = True) in the management tree with a unique path.

Usage example:

with context.temporary(suffix='.o') as p:
    ...  # an empty file with absolute path *p* (as a dlb.fs.Path) exists

... = context.temporary().path  # just get the absolute path, do not create the file

The path attribute of the context manager is an absolute path in the same directory for all calls in the root context as a dlb.fs.Path object. The __enter__ method of the context manager returns path.

The last component of path is unique among all calls in the root context (and is therefore called its unique path component). path.is_dir() is is_dir. The path of the working tree’s root is path[:-3].

The unique path component starts with a lower-case letter and ends with suffix. It contains only lower-case letters and decimal digits between its first characters and the suffix. If suffix is not empty, is must start with a character from string.punctuation and must not contain '/'. The unique path component without the suffix is at most 12 characters long for the first 2**61 calls.

When used as a context manager, an empty regular file or directory with path is created when entered and removed (with its content) on exit. Raises FileExistsError if the regular file or directory exists.

Parameters:

suffix (str) – suffix of the unique path component
Raises:
working_tree_path_of(path, *, is_dir=None, existing=False, collapsable=False, allow_nontemporary_management=False, allow_temporary=False)

Return the managed tree path of the path of a filesystem object in the managed tree.

For path to be considered as the path of a filesystem object in the managed tree, path must either be a relative path or it must have root_path as a prefix.

The arguments existing and collapsable describe the assumptions on the filesystem content that may be used to increase the speed and reduce the number of filesystem accesses.

If existing and collapsable are True and path is relative, the filesystem is never accessed.

If existing is False, is_dir() of the returned path reflects the type of the actual filesystem object. Raises dlb.ex.WorkingTreePathError if path does not exist.

If allow_nontemporary_management is True, the resulting path may denote a filesystem object in the management tree except in .dlbroot/t. If allow_temporary is True, the resulting path may denote a filesystem object in .dlbroot/t of the management tree.

Does not raise OSError.

Parameters:
  • path (dlb.fs.Path or anything a dlb.fs.Path can be constructed from) – a path of a filesystem object in the managed tree
  • is_dir (NoneType | bool) – True if this is a directory path, False if not and None for derivation from path
  • existing (bool) – assume that all involved filesystem objects exist?
  • collapsable (bool) – assume that any relative to the working tree root is collapsable?
  • allow_nontemporary_management (bool) – is the path permitted to denote a filesystem object the :term`management tree` except ones in .dlbroot/t?
  • allow_temporary (bool) – is the path permitted to denote a filesystem object in .dlbroot/t of the management tree?
Returns:

a dlb.fs.Path p with p.is_absolute() == False and p.is_normalized() == True
Return type:

class of path if path is a dlb.fs.Path, or dlb.fs.Path otherwise
Raises:
summary_of_latest_runs(*, max_count=1)

Return a list of information on previous runs of dlb according to the run-database used by the current root context.

If there is information on more than max_count, the max_count runs with the latest UTC datetime of their start are picked.

Note

There is no guaranteed that all the datetimes differ.

Each member of the returned list is a tuple (start_time, duration_ns, tool_inst_run_count, tool_inst, redo_count). start_time is the UTC datetime of the run’s start as a datetime.datetime. duration_ns is the duration of the run in nanoseconds as non-negative integer. tool_inst_run_count is the total number of runs of tool instances in the dlb run as a non-negative integer. redo_count is the total number of redos of tool instances in the dlb run as a non-negative integer not greater than tool_inst_run_count.

Parameters:max_count – maximum number of runs
Return type:list of tuples
Raises:NotRunningError – if dlb is not running).

Environment variable dictionary objects

The environment variable dictionary object env returned by c.env for a context c is a dictionary-like object of all environment variables defined in c. c is called the associated context of env.

In addition, the environment variable dictionary object manages the import of environment variables from environment variables of the outer context (or os.environ for the root context) and validation of imported or assigned values in the form of regular expressions.

Example:

# os.environ usually contains the environment variables in the shell that called the Python interpreter

with dlb.ex.Context():  # takes a snapshot of os.environ

    # import the environment variable 'LANG' into the context
    dlb.ex.Context.active.env.import_from_outer(
        'LANG', pattern=r'[a-z]{2}_[A-Z]{2}', example='sv_SE')

    # now the environment variable is either undefined or matches the regular expression given
    # (in this context and all future inner contexts)

    ... = dlb.ex.Context.active.env['LANG']
        # value in snapshot of os.environ complying to the validation pattern(s) or KeyError

    dlb.ex.Context.active.env['LANG'] = 'de_AT'

    with dlb.ex.Context():

        # further restrict accepted values
        dlb.ex.Context.active.env.import_from_outer('LANG', pattern='(?P<language>de).*', example='de_CH')

        ... = dlb.ex.Context.active.env['LANG']  # 'de_AT'
        del dlb.ex.Context.active.env['LANG']

        dlb.ex.Context.active.env['LANG'] = 'de_CH'
        # dlb.ex.Context.active.env['LANG'] = 'fr_FR'  # would raise ValueError

    ... = dlb.ex.Context.active.env['LANG']  # 'de_AT'

    del dlb.ex.Context.active.env['LANG']  # undefine 'LANG'
    dlb.ex.Context.active.env['LANG'] = 'fr_FR'  # ok

Environment variable dictionary object support the following methods and attributes:

EnvVarDict.import_from_outer(name, *, pattern, value_if_undefined=None, example=None)

Set the value of the environment variable named name from the innermost outer context that defines it or os.environ (value at the time the root context was entered) if there is no such context. If neither an outer context nor os.environ defines it, the environment variable remains undefined.

Also associates a validation pattern for the value with the environment variable; when it is or later becomes defined, the regular expression validation must match its value. Use r'.*' if any value of the environment variable is valid.

The possible imported value and the validation pattern apply to the context and all its future inner contexts.

Parameters:
  • name (str) – (non-empty) name of the environment variable
  • pattern (str | typing.Pattern) – validation pattern for value as a regular expression
  • example (str) – typical value of an environment variable, pattern must match this
Raises:
EnvVarDict.is_imported(name)

Returns True if name is the name of an environment variable imported in the associated context or any of its outer contexts, else False.

Parameters:

name (str) – non-empty name of an environment variable
Raises:
EnvVarDict.get(name, default=None)

Return its value if name is the name of a defined environment variable in the associated context, else default.

Parameters:

name (str) – non-empty name of an environment variable
Raises:
EnvVarDict.items()

Returns a new view of the dictionary’s items (name, value) pairs of all defined environment variables.

name in env

Returns True if there is an environment variable named name defined in env, else False.

name not in env

Equivalent to not name in env

env[name] = value

Defines an imported environment variable named name with value value in the associated context and all its future inner contexts.

Raises KeyError, if name was not imported in the associated context or one of its outer contexts.

Raises ValueError, if name was imported in the associated context or one of its outer contexts but the associated validation pattern in the associated context or in one of its any outer contexts does not match value.

Raises ContextModificationError, if the associated context is not an active context.

del env[name]

Undefines a defined environment variable named name in the associated context and all its future inner contexts.

Raises KeyError, if name is not defined in the context.

Raises ContextModificationError, if the associated context is not an active context.

Tool objects

Every tool is represented by a subclass of Tool that describes its abstract behaviour and the way it is run (e.g. meaning of command line and output, interaction with file system and environment variables).

Tools are parametrized by optional dependency roles (e.g. input files) and execution parameters.

Each tool instance represents a concrete behaviour and can be run in an active context. Running a tool results in an awaitable result object.

Tool instances are immutable and hashable and fast to construct; the heavy lifting takes place while the tool instance is running.

Tools are customized by inheritance and defining class attributes.

class dlb.ex.Tool

A tool declares its dependency roles and execution parameters as class attributes. The name of a dependency role must consist of at least two lower-case words separated by _ (e.g. map_file). The name of an execution parameters must consist of lower-case words separated by _ (e.g. WARNINGS, VERSION_PARAMETERS).

Dependency roles are instances of subclasses of Dependency.

Every tool instance assigns concrete dependencies for the tool’s dependency roles (e.g. a filesystem path './out/hello.map' for a dependency role map_file).

A tool also can define methods. The name of a method must consist of at least two lower-case words separated by _.

A new tool can be defined by inheriting from one or more other tools. When overriding a dependency role, its overriding value must be of the same type as the overridden value and it must be at least as restrictive (e.g. if required dependency must not be overridden by a non-required one). When overriding an execution parameter, its overriding value must be of the same type as the overridden value. When overriding an method, its overriding method must be of the same kind and have the same signature as the overridden method.

Each subclass of Tool must be defined in a source code location unique among all subclasses of Tool. The definition raises DefinitionAmbiguityError if its location cannot be determined or if another subclass of Tool is defined at the same location.

Example:

class Compiler(dlb.ex.Tool):
    WARNINGS = ('all',)
    source_file = dlb.ex.input.RegularFile()
    object_file = dlb.ex.output.RegularFile()
    async def redo(self, result, context): ...

class Linker(dlb.ex.Tool):
    BE_VERBOSE = False
    object_files = dlb.ex.input.RegularFile[1:]()
    linked_file = dlb.ex.output.RegularFile()
    map_file = dlb.ex.output.RegularFile(required=False)
    async def redo(self, result, context): ...

compiler = Compiler(source_file='main.cpp', object_file='main.cpp.o')
linker = Linker(object_files=[compiler.object_file], linked_file='main', BE_VERBOSE=True)

When a tool instance is constructed, the keyword arguments passed to its constructor define its concrete dependencies and the values of its execution parameters. Each keyword argument names a dependency role or an execution parameter.

  • When a keyword argument names a dependency role, its value defines the concrete dependencies of the dependency role. It depends on the dependency role, whether such a definition is permitted or even required. For an explicit dependency role, it is required.
  • When a keyword argument names an execution parameter, its value defines the value of the execution parameter (it overrides the default value defines at declaration of the execution parameter as a class attribute). The same restrictions apply as for overriding the execution parameter by subclassing.

The concrete dependencies passed as keyword arguments to the constructor are validated by the tool’s dependency roles and made accessible (for reading only) as an attribute with the name of the corresponding dependency role and a type determined by the dependency role (e.g. dlb.fs.Path for input.RegularFile):

>>> Compiler.object_file       # dependency role
<dlb.ex.input.RegularFile object at ...>

>>> Compiler(...).object_file  # concrete dependency
Path('main.cpp.o')

The execution parameters passes as keyword arguments to the constructor are made accessible (for reading only) as an attribute:

>>> Linker.BE_VERBOSE
False  # default value defined in tool class
>>> Linker(...).BE_VERBOSE  # ... no keyword argument BE_VERBOSE=
False  # default value defined in tool class

>>> Linker(BE_VERBOSE=True).BE_VERBOSE
True   # value from keyword argument in constructor
start(force_redo=False)

Start running the tool instance in the active context and return a result (proxy) object result.

bool(result) is True if a redo is performed and False otherwise. A redo is performed if force_redo is True or if it is necessary. “Performing a redo” means scheduling the eventual (asynchronous) execution of redo() of this class.

If a redo is performed, this method returns while the redo is pending (not complete). After each of the following actions the redo is guaranteed to be complete, either successfully or by raising an exception:

  • read of a “public” attribute (or method) of the result (proxy) object
  • exit of the context start() was called in
  • enter of an inner context of the context start() was called in
  • modification of env or helper of the context start() was called in
  • call of start() of the same tool instance

The result (proxy) object result contains an attribute for every dependency role of the tool which contains the concrete dependencies. result also has a method complete() which only returns result.

Use t.start(...).complete() for a tool instance t to block until the possible redo is complete.

If bool(result) is True, all attributes for dependency roles have an assigned value. If bool(result) is False, only the attributes for explicit dependency roles have an assigned value; the value of all attributes for non-explicit dependency roles is NotImplemented.

Parameters:force_redo (bool) – perform a redo even if not necessary?
Returns:result (proxy) object
redo(result, context)

Performs the actual work for this tool instance and creates all its output dependencies (output files from input files, for example).

Overwrite this awaitable method to implement a new Tool (do not forget async!).

Assign to attributes of result to define a non-explicit concrete dependency for the dependency role with the same name.

For a redo to be successful, this method must perform the following tasks:

  • Create all explicit output dependencies
  • Assign values to each required non-explicit dependencies

Use await context.execute_helper() to call executable files.

Make sure the mtime of each filesystem object that is an output dependency of this tool instance is updated when it is (potentially) modified, even if the redo fails.

For a filesystem object whose path p is contained in an output dependency, it is recommended to first write to a temporary filesystem object q and then replace it with context.replace_output(p, q). This guarantees that no incomplete output dependency is left behind (like an only half-written object file) when the redo is aborted. A filesystem object that is an output dependency is treated as modified by the redo if it is a non-explicit dependency or if it is an explicit dependency that was replaced with context.replace_output().

Raises RuntimeError on the attempt to enter a new dlb.ex.Context as a context manager or to modify the active context.

Return True if the next run this tool instance should perform a redo, regardless of the necessity according to its dependencies.

Example:

class ATool(dlb.ex.Tool):
    EXECUTABLE = 'atool'

    source_file = dlb.ex.input.RegularFile()
    output_file = dlb.ex.output.RegularFile()
    included_files = dlb.ex.input.RegularFile[:](explicit=False)

    async def redo(self, result, context):
        if ...:
            raise ValueException('invalid ...')
        with context.temporary() as temp_file_
            await context.execute_helper(self.EXECUTABLE,
                                         ['-o', temp_file, result.source_file])
            result.included_files = ...
            context.replace_output(result.output_file, temp_file)
Parameters:
  • result – result proxy object to be returned by start()
  • context (RedoContext) – redo context
Returns:

request a redo for the next run of the tool instance?
Return type:

None | bool
definition_location

The definition location of the class.

It is a tuple of the form (file_path, in_archive_path, lineno) and uniquely identifies the tool among all subclasses of Tool.

in_archive_path is None if the class was defined in an existing Python source file and file_path is the os.path.realpath() of this file.

in_archive_path is the path relative of the source file in the zip archive, if the class was defined in an existing zip archive with a filename ending in .zip (loaded by zipimport) and file_path is the os.path.realpath() of this zip archive.

lineno is the 1-based line number in the source file.

fingerprint

The permanent local tool instance fingerprint of this instance.

This is a bytes object of fixed size, calculated from all its concrete dependencies d with d.explicit == True and all its execution parameters.

If two instances of the same subclass of Tool have equivalent explicit dependencies and equivalent execution parameters, their fingerprints are equal. Otherwise, their fingerprints are different with very high probability.

The explicit dependencies of two instances are equivalent, if they are equal or differ only in a way that does not affect the meaning of the dependencies while the tool instance is running. Execution parameters are equivalent if they are equal after replacing all mappings by dicts and all iterables that are no string, bytes, or sets by tuples.

Redo context

A redo context is a read-only view for a dlb.ex.Context with some additional methods related to dynamic helpers and dependencies.

class dlb.ex.RedoContext

A redo context is constructed automatically by Tool.start(). Redo contexts should not be constructed manually.

execute_helper(helper_file, arguments=(), *, cwd=None, expected_returncodes=frozenset([0]), forced_env={}, stdout_output=None, stderr_output=None)

Execute the helper_file with command-line arguments arguments in a subprocess with cwd as its working directory and wait for it to complete. The execution is considered successful if and only if its returncode is one in expected_returncodes.

If cwd is not None, is must be the path of directory in the managed tree or in .dlbroot/t/ of the management tree. Otherwise the working tree’s root is used as the working directory.

All members of arguments are converted to str objects with prepare_arguments(arguments, cwd=cwd).

env of this object, modified by forced_env, forms the environment for the subprocess.

Note

On Windows, in order to run a side-by-side assembly the specified env must include a valid SYSTEMROOT.

The file descriptors 1 (stdout) and 2 (stderr) are open in the subprocess. Their meaning is specified by stdout_output and stderr_output. No other file descriptor from the Python process is open in the subprocess.

Parameters:
  • helper_file – dynamic helper to be executed as a relative path
  • arguments (iterable of objects that can be converted to str) – command-line arguments
  • cwd (None or a dlb.fs.Path or anything a dlb.fs.Path can be constructed from) – working directory of the subprocess to be started
  • expected_returncodes (collection of integers) – expected return codes of the dynamic helper helper_file
  • forced_env (None | Dict[str, str]) – dictionary of values to override in env or None
  • stdout_output – If None: use dlb.cf.execute_helper_inherits_files_by_default. If True: inherit stdout of the Python process (the file descriptor, not sys.stdout). If False: suppress the output. Otherwise: the path of the output file as a dlb.fs.Path or anything a dlb.fs.Path can be constructed from (opened with mode 'wb').
  • stderr_output – If None: use dlb.cf.execute_helper_inherits_files_by_default. If True: inherit stderr of the Python process (the file descriptor, not sys.stderr). If False: suppress the output. Otherwise: the path of the output file as a dlb.fs.Path or anything a dlb.fs.Path can be constructed from (opened with mode 'wb').
Return type:

int
Returns:

return code of the subprocesses (one of expected_returncodes)
Raises:

HelperExecutionError – if the subprocess exits with a returncode not in expected_returncodes.
execute_helper_with_output(helper_file, arguments=(), *, cwd=None, expected_returncodes=frozenset([0]), forced_env={}, output_to_process=1, other_output=None, chunk_processor=None)

Execute the helper_file with command-line arguments arguments in a subprocess with cwd as its working directory, wait for it to complete, and return its output.

When chunk_processor is None, the entire output of the subprocess to stdout or stderr (as defined by output_to_process) is returned as bytes objects. Example:

_, output = await context.execute_helper_with_output(..., output_to_process=2)  # without *chunk_processor*
# output is a 'bytes' objects containing the output of the subprocess to stderr

When chunk_processor is not None, the output of the subprocess is split into chunks separated by the non-empty bytes object chunk_processor.separator, and chunk_processor.process(chunk, is_last) is called for each of the chunks (without chunk_processor.separator) right after they occur. is_last is False for every call except the last one that was not delimited by chunk_processor.separator but by the end of the stream. This is suitable when a subprocess outputs a lot of data (even unbounded) but only a small part of it is interesting. Example:

class Processor(dlb.ex.ChunkProcessor):
    separator = b'\r\n'
    max_chunk_size = 100  # maximum chunk size (without separator)

    def __init__(self):
        self.result = []

    def process(self, chunk: bytes, is_last: bool):  # len(chunk) is <= 100, chunk does not contain '\r\n'
        if chunk.startswith('hello'):
            self.result.append(chunk)

 _, output = await context.execute_helper_with_output(..., chunk_processor=Processor())
 # output is a list of 'bytes' objects starting with b'hello'

The file descriptors 1 (stdout) and 2 (stderr) are open in the subprocess. Their meaning is specified by output_to_process and other_output. No other file descriptor from the Python process is open in the subprocess.

See execute_helper() for a description of cwd, arguments, expected_returncodes, and forced_env.

Parameters:
  • helper_file – dynamic helper to be executed as a relative path
  • arguments (iterable of objects that can be converted to str) – command-line arguments
  • cwd (None or a dlb.fs.Path or anything a dlb.fs.Path can be constructed from) – working directory of the subprocess to be started
  • expected_returncodes (collection of integers) – expected return codes of the dynamic helper helper_file
  • forced_env (None | Dict[str, str]) – dictionary of values to override in env or None
  • output_to_process (int) – 1 to return stdout and 2 to return stderr of the subprocess
  • other_output – Output for stdout of the subprocess if output_to_process is 2 and for stderr otherwise. If None: use dlb.cf.execute_helper_inherits_files_by_default. If True: inherit stdout/stderr of the Python process (the file descriptor, not sys.stdout/sys.stderr). If False: suppress the output. Otherwise: the path of the output file as a dlb.fs.Path or anything a dlb.fs.Path can be constructed from (opened with mode 'wb').
  • chunk_processor (None | dlb.ex.ChunkProcessor) – If None the entire output is returned. Otherwise, chunk_processor.result after each chunk was fed to chunk_processor as described above.
Raises:
  • HelperExecutionError – if the subprocess exits with a returncode not in expected_returncodes.
  • asyncio.LimitOverrunError – if chunk_processor is not None and the subprocess outputs more than chunk_processor.max_chunk_size bytes without a chunk_processor.separator.

Returns a tuple (returncode, output). returncode is the returncode of the subprocess and output its output to stdout or stderr — processed by chunk_processor if chunk_processor is None — as described above.

prepare_arguments(self, arguments, cwd=None)

Convert all members of arguments to str objects.

If a member of arguments is a dlb.fs.Path object p with p.is_absolute() == ``True, is is replaced by str(p.native). If a member of arguments is a dlb.fs.Path object p with p.is_absolute() == ``False, is is replaced by str(q.native), where q is p expressed relative to the working directory. It must denote a filesystem object in the managed tree or in .dlbroot/t/ of the management tree.

If cwd is not None, is must be the path of directory in the managed tree or in .dlbroot/t/ of the management tree. Otherwise the working tree’s root is used as the working directory.

Parameters:
  • arguments (iterable of objects that can be converted to str) – command-line arguments
  • cwd (None or a dlb.fs.Path or anything a dlb.fs.Path can be constructed from) – working directory of the subprocess to be started
Returns:

List of strings
replace_output(path, source)

Replace the — existing or non-existent — filesystem object path by source. path must be contained in a dependency of the tool instance.

path and source must be different filesystem objects.

After successful completion, path exists and source does not exist. If the parent directory of path does not exist, it is created (with all its parent directories).

The actual operation depends on the corresponding dependency role. If it is dlb.ex.output.RegularFile(replace_by_same_content=False, ...) and path and source both exist with the same content, path is not replaced and treated as unchanged.

If path is replaced, this is always done by an atomic operation. If it fails, path is either source afterwards or it does not exist.

Parameters:
  • path (dlb.fs.Path or anything a dlb.fs.Path can be constructed from) – a path of a future filesystem object in the managed tree
  • source (dlb.fs.Path or anything a dlb.fs.Path can be constructed from) – a path of a filesystem object in the managed tree
Returns:

False if path was not changed by this operation, True otherwise.
Raises:

Dependency classes

A dependency class is a subclass of Dependency. Its instances describe dependency roles (as attributes of a Tool).

The Dependency.validate() methods of dependency classes are used by tool instances to create concrete dependencies from their constructor arguments.

Each dependency role has an multiplicity specification:

  1. An instance d of a dependency class D created with D(...) has a multiplicity of None which means that its concrete dependency must be a single object (its type depends on D only) or None.

  2. An instance d of a dependency class D created with D[m](...) has a multiplicity of m which means that its concrete dependencies are a sequence of objects (their type depends on D only) or None. The accepted number of members is specified by m.

    m can be any non-negative integer or any meaningful proper_slice (of non-negative integers). A number of members is accepted if and only if is either equal to m or contained in range(n + 1)[m].

Example:

class Tool(dlb.ex.Tool):
    # these are dependency roles of the tool 'Tool':
    include_search_directories = dlb.ex.input.Directory[1:]()  # a sequence of at least one dlb.ex.input.Directory
    cache_directory = dlb.ex.input.Directory()  # a single dlb.ex.input.Directory

tool = Tool(include_search_directories=['build/out/Generated/', 'src/Implementation/'])

# these are concrete dependencies of the tool instance 'tool':
tool.include_search_directories  # (Path('build/out/Generated/'), Path('src/Implementation/'))
tool.cache_directory  # (Path('build/out/Generated/'), Path('src/Implementation/'))

Dependency classes are organized in an a hierarchy according to their meaning to a tool by the means of the following abstract classes:

digraph foo { graph [rankdir=BT]; node [height=0.25]; edge [arrowhead=empty]; "dlb.ex.InputDependency" -> "dlb.ex.Dependency"; "dlb.ex.OutputDependency" -> "dlb.ex.Dependency"; }

class dlb.ex.InputDependency

A Dependency that describes an input dependency of a tool.

A running tool instance must perform a redo if it (e.g. the mtime of a file) has changed compared to the state before the last known successful redo of the tool instance.

An redo must not modify it, successful or not (the same object can be an output dependency of the same tool instance though which can be modified, although).

class dlb.ex.OutputDependency

A Dependency that describes an output dependency of a tool.

A successful redo must generate it (e.g. create a regular file).

Dependency classes support the following methods and attributes:

class dlb.ex.Dependency(required=True, explicit=True)

If required is True, a concrete dependency of this dependency role will never be None.

If explicit is True, the concrete dependency can and must be fully defined when the tool instance is created. Otherwise, it cannot and must not be; it is instead automatically assigned by Tool.start().

Each supported constructor argument is available as a property of the same name.

Raises:DependencyError – if the arguments of the constructor do not match the declared dependency roles of the class
class Value

A (potentially abstract) class such that isinstance(v, Value) is True for each validated single value v of each instance t of this class.

This is the type of t.validate() if multiplicity is None and the type of each member of t.validate() otherwise.

validate(value)
Parameters:value (Any type the concrete dependency can convert to T) – The concrete dependency to convert and validate except None
Returns:The validated value of type T
Raises:TypeError – If multiplicity is not None and value is not iterable or is a string
compatible_and_no_less_restrictive(other)

Is this dependency role an instance of the same class as other with a multiplicity and properties no less restrictive than the ones of other?

Parameters:other (Dependency) – reference dependency role
Return type:bool
multiplicity

The multiplicity of the dependency role.

Is None or an multiplicity object that supports test for (in)equality and membership of integers (__eq__ and __contains__, respectively).

tuple_from_value(value)

Return value if multiplicity is None and a tuple of its members otherwise.

Example:

# returns a tuple of t.Value objects or raises an exception:
>>> v = t.tuple_from_value(t.validate(...))

Each non-abstract dependency class is derived from InputDependency or from from InputDependency:

digraph foo { graph [rankdir=BT]; node [height=0.25]; edge [arrowhead=empty]; "dlb.ex.input.RegularFile" -> "dlb.ex.InputDependency"; "dlb.ex.input.NonRegularFile" -> "dlb.ex.InputDependency"; "dlb.ex.input.Directory" -> "dlb.ex.InputDependency"; "dlb.ex.input.EnvVar" -> "dlb.ex.InputDependency"; "dlb.ex.output.RegularFile" -> "dlb.ex.OutputDependency"; "dlb.ex.output.NonRegularFile" -> "dlb.ex.OutputDependency"; "dlb.ex.output.Directory" -> "dlb.ex.OutputDependency"; "dlb.ex.output.Object" -> "dlb.ex.OutputDependency"; "dlb.ex.InputDependency" -> "dlb.ex.Dependency"; "dlb.ex.OutputDependency" -> "dlb.ex.Dependency"; }

Note

dlb identifies filesystem objects by their managed tree path. It assumes that different managed tree paths point to different filesystem objects.

If a filesystem object serves as an output dependency of one tool instance and as an input dependency of another: Make sure both dependencies use the same path. A redo miss could happen otherwise.

You are always safe without hard links, symbolic links and case-insensitive filesystems.

Input dependency classes

Dependency class Keyword arguments of constructor Depended-upon state for redo necessity
Name Default value
input.RegularFile cls dlb.fs.Path summary of status of filesystem object with given managed tree path
input.NonRegularFile
input.Directory
input.EnvVar name   value of enviroment variable name in active context
pattern  
example  

In addition to the keyword arguments of the specific constructors described here, all constructors also accept the keyword arguments of the constructor of Dependency.

class input.RegularFile(cls=dlb.fs.Path, required=True, explicit=True)

A dependency role for regular files, identified by their paths.

If a path is relative, it is treated as relative to dlb.ex.Context.active.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

The validated value of a concrete dependency is the file’s path as an instance of cls if multiplicity is None and a tuple of the file’s paths otherwise.

The depended-upon state for redo necessity is a summary of each file’s status which consists in

  • mtime
  • size
  • UID
  • GID
  • filesystem permissions

For a file outside the managed tree, the state is always assumed to remain unchanged between runs of dlb.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    source_files = dlb.ex.input.RegularFile[1:](cls=dlb.fs.NoSpacePath)
>>> tool = Tool(source_files=['src/main.cpp'])
>>> tool.source_files
(NoSpacePath('src/main.cpp'),)
Parameters:cls (dlb.fs.Path) – class to be used to represent the path
class Value

Is dlb.fs.Path.

class input.NonRegularFile(cls=dlb.fs.Path, required=True, explicit=True)

A dependency role for filesystem objects that are neither directories nor regular files, identified by their paths.

If a path is relative, it is treated as relative to dlb.ex.Context.active.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

The validated value of a concrete dependency is each file’s path as an instance of cls if multiplicity is None and a tuple of the file’s paths otherwise.

The depended-upon state for redo necessity is a summary of each file’s status which consists in

  • mtime
  • size
  • type of filesystem object (e.g. symbolic link, block device, …)
  • target path if a symbolic link
  • UID
  • GID
  • filesystem permissions

For a file outside the managed tree, the state is always assumed to remain unchanged between runs of dlb.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    symbolic_links = dlb.ex.input.NonRegularFile[:](cls=dlb.fs.NoSpacePath)
>>> tool = Tool(symbolic_links=['src/current'])
>>> tool.symbolic_links
(NoSpacePath('src/current'),)
Parameters:cls (dlb.fs.Path) – class to be used to represent the path
class Value

Is dlb.fs.Path.

class input.Directory(cls=dlb.fs.Path, required=True, explicit=True)

A dependency role for directories, identified by their paths.

If a path is relative, it is treated as relative to dlb.ex.Context.active.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

The validated value of a concrete dependency is the directory’s path as an instance of cls if multiplicity is None and a tuple of the directory’s paths otherwise.

The depended-upon state for redo necessity is a summary of each directory’s status which consists in

  • mtime
  • size
  • UID
  • GID
  • filesystem permissions

For a directory outside the managed tree, the state is always assumed to remain unchanged between runs of dlb.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    cache_directory = dlb.ex.input.Directory(required=False)
>>> tool = Tool(cache_directory='/tmp/')
>>> tool.cache_directory
Path('tmp/')
Parameters:cls (dlb.fs.Path) – class to be used to represent the path
class Value

Is dlb.fs.Path.

class input.EnvVar(name, pattern, example, required=True, explicit=True)

A dependency role for an environment variable named name. It must not have a multiplicity (other than None).

If explicit is True, the value assigned in the constructor of the tool instance is used for all future runs of the tool instance. Otherwise, the current value of the active context is used each time Tool.start() is called.

The value of the environment variable is valid if it is a string that matches the regular expression pattern or if it is None and required is False.

The validated value of a concrete dependency is a Value instance with the environment variable’s name and value.

The depended-upon state for redo necessity is the value of environment variable name in the active context.

Example:

>>> class Tool(dlb.ex.Tool):
>>>     language_code = dlb.ex.input.EnvVar(
>>>                         name='LANG',
>>>                         pattern=r'(?P<language>[a-z]{2})_(?P<territory>[A-Z]{2})',
>>>                         example='sv_SE')
>>>     cflags_string = dlb.ex.input.EnvVar(name='CFLAGS', pattern=r'.+', example='-Wall')
>>> tool = Tool(language_code='de_CH')  # use 'de_CH' as value of the environment variable for all
>>> tool.language_code.groups['territory']
'CH'
>>> tool.cflags_string
NotImplemented
>>> tool.start().cflags_string.raw  # assuming dlb.ex.Context.active.env['CFLAGS'] of '-O2'
'-O2'
Parameters:
  • pattern (str | typing.Pattern) – validation pattern for value as a regular expression
  • example (str) – typical value of an environment variable, pattern must match this
class Value

A class decorated with dataclasses.dataclass() and the following fields:

name

The name of the environment variable, as in the corresponding concrete dependency.

raw

The value of the environment variable.

groups

The named groups of pattern of the corresponding concrete dependency when matched against raw.

Output dependency classes

Dependency class Keyword arguments of constructor
Name Default value
output.RegularFile cls dlb.fs.Path
replace_by_same_content True
output.NonRegularFile cls dlb.fs.Path
output.Directory cls dlb.fs.Path
output.Object    

In addition to the keyword arguments of the specific constructors described here, all constructors also accept the keyword arguments of the constructor of Dependency.

class output.RegularFile(cls=dlb.fs.Path, replace_by_same_content=True, required=True, explicit=True)

A dependency role for regular files in the managed tree, identified by their paths.

If a path is relative, it is treated as relative to dlb.ex.Context.active.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

The validated value of a concrete dependency is the file’s path as an instance of cls if multiplicity is None and a tuple of the file’s paths otherwise.

If replace_by_same_content is False for a dependency role containing p, context.replace_output(p, q) in redo(..., context) does not replace p if p and q both exist as accessible regular files and have the same content.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    object_file = dlb.ex.output.RegularFile(cls=dlb.fs.NoSpacePath)
>>> tool = Tool(object_file=['main.cpp.o'])
>>> tool.object_file
(NoSpacePath('main.cpp.o'),)
Parameters:cls (dlb.fs.Path) – class to be used to represent the path
class Value

Is dlb.fs.Path.

class output.NonRegularFile(cls=dlb.fs.Path, required=True, explicit=True)

A dependency role for filesystem objects in the managed tree that are neither directories nor regular files, identified by their paths.

If a path is relative, it is treated as relative to dlb.ex.Context.active.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

The validated value of a concrete dependency is the file’s path as an instance of cls if multiplicity is None and a tuple of the file’s paths otherwise.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    symbolic_links = dlb.ex.output.NonRegularFile[:](cls=dlb.fs.NoSpacePath)
>>> tool = Tool(symbolic_links=['dist'])
>>> tool.symbolic_links
(NoSpacePath('src/current'),)
Parameters:cls (dlb.fs.Path) – class to be used to represent the path
class Value

Is dlb.fs.Path.

class output.Directory(cls=dlb.fs.Path, required=True, explicit=True)

A dependency role for directories in the managed tree, identified by their paths.

If a path is relative, it is treated as relative to dlb.ex.Context.active.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

The validated value of a concrete dependency is the directory’s path as an instance of cls if multiplicity is None and a tuple of the directory’s paths otherwise.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    html_root_directory = dlb.ex.output.Directory(required=False)
>>> tool = Tool(html_root_directory='html/')
>>> tool.html_root_directory
Path('html/')
Parameters:cls (dlb.fs.Path) – class to be used to represent the path
class Value

Is dlb.fs.Path.

class output.Object(required=True, explicit=True)

A dependency role for any Python object other than None and NotImplemented. explicit must be False.

The validated value of a concrete dependency is a deep copy of the value.

class Value

Is typing.Any.

Exceptions

exception dlb.ex.NotRunningError

Raised, when an action requires an active context while dlb was not running.

exception dlb.ex.NoWorkingTreeError

Raised, when the working directory of the calling process is not a working tree’s root.

exception dlb.ex.ManagementTreeError

Raised, when an attempt to prepare or access the management tree failed.

exception dlb.ex.ContextNestingError

Raised, when some contexts were not properly nested. I.e. the calls of __exit__ did not occur in the opposite order of the corresponding calls of __enter__.

exception dlb.ex.WorkingTreeTimeError

Raised, when the working tree time behaved unexpectedly.

exception dlb.ex.ContextModificationError

Raised, when the modification of an environment variable dictionary object or a helper dictionary object has been attempted while its associated context is not the active context.

exception dlb.ex.WorkingTreePathError

Raised, when a path is not a working tree path with certain properties where it should be.

exception dlb.ex.DefinitionAmbiguityError

Raised at the definition of a subclass of Tool, when the location is unknown or another subclass of Tool is defined at the same location.

exception dlb.ex.DependencyError

Raised, when a running tool instance has detected a problem with its dependencies before a redo.

exception dlb.ex.ExecutionParameterError

Raised, when a running tool instance has detected a problem with its execution parameters before a redo.

exception dlb.ex.RedoError

Raised, when a running tool instance has detected a problem with its dependencies during or after a redo.

exception dlb.ex.HelperExecutionError

Raised, when the execution of a dynamic helper file failed.

dlb_contrib – Contributed tools and utilities

dlb (the package) is completely tool agnostic. Everything specific to a certain tool, language or file format is not part of dlb.

The package dlb_contrib contains such tools as well as utilities to generate input understood by the tools and parsers for the output of the tools.

Details

The modules in dlb_contrib are meant as starting points for your own tools and extensions which will reside in your repositories. Therefore, their documentation is maintained only in the source files to make them self-contained. Please see there for details.

module dlb_contrib.backslashescape[source]

Process backslash escape sequences (Python, Bash, Git, …).

License:LGPL-3.0-or-later

Provided objects:

Usage example:

import dlb_contrib.backslashescape

s = b'"tmp/x\\076y"'
... = dlb_contrib.backslashescape.unquote(s)  # b'tmp/x\076y'
module dlb_contrib.iso6429[source]

Output text to a terminal or terminal emulator that supports a Select Graphic Rendition Control Sequence (SGR) compliant with ISO/IEC 6429 (ECMA-48, ANSI X3.64). Of such terminals the DEC VT-100 (although monochrome) is best known.

ISO/IEC 6429:https://www.iso.org/standard/12782.html
ECMA-48:https://www.ecma-international.org/publications-and-standards/standards/ecma-48/
DEC VT-100:https://www.vt100.net/docs/vt100-ug/contents.html
ANSI escape code:
 https://en.wikipedia.org/wiki/ANSI_escape_code#CSI_(Control_Sequence_Introducer)_sequences
Tested with:console of Linux 4.19

tilix 1.8.9

Terminal of PyCharm 2020.1.1 (Community Edition)
License:LGPL-3.0-or-later

Provided objects:

Usage example:

import sys
import dl.di
import dlb_contrib.iso6429

if sys.stderr.isatty():
    dlb.di.set_output_file(dlb_contrib.iso6429.MessageColorator(sys.stderr))
module dlb_contrib.exctrace[source]

Adjust how uncaught exceptions are reported.

License:LGPL-3.0-or-later

Provided objects:

Usage example:

import dlb.ex
import dlb_contrib.exctrace

dlb_contrib.exctrace.enable_compact_with_cwd(traceback_file='build/out/traceback.log')

...
module dlb_contrib.partition[source]

Partition filesystem objects in the managed tree.

License:LGPL-3.0-or-later

Provided objects:

Usage example:

import dlb.fs
import dlb_contrib.partition

paths = dlb.fs.Path(...).list(...)
path_groups = dlb_contrib.partition.by_working_tree_path(paths, number_of_groups=1000)
# path_groups now contains (at most 1000) non-empty lists of elements of *paths*

path_groups = dlb_contrib.partition.split_longer(path_groups, max_length=5)
# path_groups now contains (non-empty) lists with at most 5 members
module dlb_contrib.exportenv[source]

Export environment variables to an UTF-8 encoded JSON file in the current directory. To be used by batch files and shell scripts (see dlb_contrib.msbatch).

JSON:https://tools.ietf.org/html/rfc8259
License:LGPL-3.0-or-later

Provided objects:

Usage example:

# e.g. in a batch file
python3 -m dlb_contrib.exportenv

# 'env.json' now looks like this:
# {"XY": "a line\nand another one"}
module dlb_contrib.python[source]

Manipulate Python module search paths in a portable way.

License:LGPL-3.0-or-later

Provided objects:

Usage example:

import dlb_contrib.python
build_directory = dlb.fs.Path('build/')
dlb_contrib.python.prepend_to_module_search_path(
    build_directory / 'python/',
    build_directory / 'out/gsrc/python/'
)
import my_specific_module  # found in 'build/python/my_specific_module.py'
module dlb_contrib.generic[source]

Group or combine multiple tool instances or tool classes.

License:LGPL-3.0-or-later

Provided tools:

Other objects:

Usage examples:

# Request a redo tool instance based on a dynamic condition with
# dlb_contrib.generic.Check.

import dlb.ex
import dlb_contrib.generic

with dlb.ex.Context():
    # request a redo (no redo miss even if condition cannot be
    # reproduced in the next run)
    problem_detected = ....

    needs_update = dlb_contrib.generic.Check(
        result_file='build/out/result.uptodate'
    ).start(force_redo=problem_detected)
    # performs redo if *result_file* this does not exist; redo removes *result_file*

    with dlb.ex.Context:  # waits for previous redos to complete
        if needs_update:
            # perform actual work
            # *needs_update* will be True next time if this fails with an exception
            ...

    needs_update.result_file.native.raw.touch()  # mark successful completion

# Before checking dependencies in detail, check necessary precondition for redo with
# dlb_contrib.generic.Check:
# Check the source code dependencies of a huge library only if at least one file in
# the source code directory has changed (much faster if no changes).

import dlb.ex
import dlb_contrib.generic

with dlb.ex.Context():
    # contains all source files of a huge library:
    library_source_directory = dlb.fs.Path('src/libx/')
    archive_file = dlb.fs.Path(...)

    # update mtime of directory if content has later mtime
    # (requires monotonic system time to detect all mtime changes)
    mtime = dlb.fs.propagate_mtime(library_source_directory)
    assert mtime is None or mtime <= dlb.ex.Context.active.working_tree_time_ns

    # 1. check fast but course whether a update may be necessary
    needs_update = dlb_contrib.generic.Check(
        input_directories=[library_source_directory],
        output_files=[archive_file]
        result_file=archive_file.with_appended_suffix('.uptodate')
            # redo if this does not exist
    ).start()  # redo removes *result_file*

    with dlb.ex.Context:  # waits for previous redos to complete
        if needs_update:  # need to take a closer look?
            # 2. check in detail and perform actual work if necessary
            # *needs_update* will be True next time if this fails with an exception
            ...

    needs_update.result_file.native.raw.touch()  # mark successful completion

# Query the version of executables used by tool classes with
# dlb_contrib.generic.VersionQuery.

import dlb.ex
import dlb_contrib.generic

with dlb.ex.Context():
    class VersionQuery(dlb_contrib.generic.VersionQuery):
        VERSION_PARAMETERS_BY_EXECUTABLE = {
            tool.EXECUTABLE: tool.VERSION_PARAMETERS
            for tool in [dlb_contrib.gcc.CCompilerGcc, dlb_contrib.doxygen.Doxygen, ...]
        }
    version_by_path = VersionQuery().start().version_by_path
module dlb_contrib.filesystem[source]

Manipulate sets of filesystem objects in an efficient and portable manner.

License:LGPL-3.0-or-later

Provided tools:

Other objects:

Usage example:

# "Atomically" collect a set of files in a directory with
# dlb_contrib.filesystem.FileCollector.

import dlb.ex
import dlb_contrib.filesystem

with dlb.ex.Context():
    dlb_contrib.filesystem.FileCollector(
        input_files=['build/out/application', 'build/out/doxygen/application.html.zip'],
        output_directory='dist/'
    ).start()
    # replaces 'dist/' with a directory that contains the files
    # 'dist/application.html', 'dist/application.html.zip'
module dlb_contrib.url[source]

Map identifier string to URLs bijectively by patterns with placeholders, and build URL regex from typical subexpressions.

URI:https://tools.ietf.org/html/rfc3986
URL vs URL:https://en.wikipedia.org/wiki/Uniform_Resource_Name#URIs,_URNs,_and_URLs
License:LGPL-3.0-or-later

Provided objects:

Usage examples:

import re
import dlb_contrib.url

# HTTPS URL with DNS domain name as host, optional port number and path with only
# non-empty "segments" (components) other than '.' and '..':
url_regex = re.compile(
    '(?i:https)://{host}(:{port}?)?{path}'.format(
        host=dlb_contrib.url.DNS_HOST_DOMAIN_NAME_REGEX.pattern,
        port=dlb_contrib.url.PORT_REGEX.pattern,
        path=dlb_contrib.url.AUTHORITY_NORMALIZED_PATH_REGEX.pattern))

m = url_regex.fullmatch('https://tools.ietf.org/html/rfc3986')
... = m.groupdict()  # {'dns_domain': 'tools.ietf.org', 'port': None, 'authority_path': '/html/rfc3986'}

import dlb.ex
import dlb_contrib.url

with dlb.ex.Context():
    # Bijectively map all repository URLs to identifier strings at a single place and
    # configurable by environment variables.
    #
    # Idea: When the URLs change (e.g. due to a build in a different environment) only
    # the environment variables have to be adapted, not the dlb script.

    dlb.ex.Context.active.env.import_from_outer(
        'URL_PATTERN_EX_1_COMP',
        pattern='https://.+',
        example='https://gitlab.dev.example.org/ex-1-comp-{compid:utf8}.git')

    url_map = dlb_contrib.url.IdToUrlMap({
        # id pattern:         URL pattern
        'ex-1-comp-{compid}': dlb.ex.Context.active.env['URL_PATTERN_EX_1_COMP'],
        'gh:{user}:{repo}':   'https://github/{user:utf8}/{repo:utf8}.git'
    }, {
        # placeholder name:   regex for placeholder's value
        'compid':             r'0|[1-9][0-9]*',
        'user':               r'[^/:]+',
        'repo':               r'[^/:]+'
    })

    # 'git clone' repositories to directories names after their id:
    for repo_id in ['ex-1-comp-42', 'ex-1-comp-747', 'gh:dlu-ch:dlb']:
        # 'git clone' from these URLs to f"build/out/{repo_id}/":
        ... = url_map.get_url_for(repo_id)

    # identify repository from its cloned URLs:
    cloned_url = ...  # 'git config --get remote.origin.url'
    ... = url_map.get_id_for(cloned_url)
module dlb_contrib.zip[source]

Create ZIP archives from directory content.

.ZIP file format:
 https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
License:LGPL-3.0-or-later

Provided tools:

Usage example:

import dlb.ex
import dlb_contrib.zip

with dlb.ex.Context():
    dlb_contrib.zip.ZipDirectory(
        content_directory='build/out/html/',
        archive_file='build/out/application.html.zip'
    ).start()
module dlb_contrib.versioned_interface[source]

Check if the hash in a (line-based) file describing an interface version matches the content hash of the files that define the interface.

License:LGPL-3.0-or-later

Provided objects:

Usage example:

import dlb.fs
import dlb.ex
import dlb_contrib.versioned_interface

# content of file 'api-version.h':
#     ...
#     // last checked for header file hash c58a0b430264752dab68af5efccdd8b4da222995
#     // (use <none> for hash to disable check temporarily)
#     #define LIBX_API_VERSION 2
#     ...

with dlb.ex.Context():
    source_directory = dlb.fs.Path('src/')

    # compare hash of header files with hash in 'api-version.h'
    dlb_contrib.versioned_interface.check_hash(
        # everything that does not end in '.c':
        files_to_hash=source_directory.iterdir(name_filter=r'(?!.+\.c$).+',
                                               recurse_name_filter=''),
        hash_line_file=source_directory / 'api-version.h',
        hash_line_regex=rb'^// last checked for header file hash ([0-9a-f]+|<none>)$',
        warnonly_hash=b'<none>'
    )

# This enforces an update of 'api-version.h' after each change of a .h file
# (which potentially requires an increase of LIBX_API_VERSION)
module dlb_contrib.linux[source]

Query Linux-specific information on hardware, filesystems and operating system as exposed by /proc.

/proc of Linux:https://man7.org/linux/man-pages/man5/proc.5.html
Source code of Linux 4.19:
 https://github.com/torvalds/linux/tree/v4.19
Tested with:Linux 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64 GNU/Linux
License:LGPL-3.0-or-later

Provided objects:

Usage examples:

# Get system information as provided by the Linux kernel.

import dlb_contrib.linux

kernel_info = dlb_contrib.linux.get_kernel_info(['.'])
# e.g. ('Linux', '4.19.0-9-amd64', '#1 SMP Debian 4.19.118-2 (2020-04-29)')

total_usable, estimated_available_without_swapping = \
    dlb_contrib.linux.get_memory_info()
    # e.g. (16694640640, 13603610624)

cache_sizes = dlb_contrib.linux.get_cpu_info().by_key('cache size')
# e.g. {'8192 KB': (0, 1, 2, 3, 4, 5, 6, 7)}

# List filesystems whose mountpoints are a prefix of os.path.realpath('.').

import dlb_contrib.linux

vfstype_name_by_mountpoint = \
    dlb_contrib.backslashescape.get_mounted_filesystems(['.'])
    # e.g. {Path('/'): 'xfs', Path('/tmp/'): 'tmpfs'}
module dlb_contrib.git[source]

Access a working directory managed by Git - the stupid content tracker.

Git:https://git-scm.com/
Tested with:git 2.20.1
Executable:'git'
License:LGPL-3.0-or-later

Provided tools:

Other objects:

Usage examples:

import dlb.di
import dlb.ex
import dlb_contrib.git

with dlb.ex.Context():
    result = dlb_contrib.git.GitDescribeWorkingDirectory().start()

    ... = result.tag_name  # e.g. 'v1.2.3'
    ... = result.branch_refname  # 'refs/heads/master'

    if result.untracked_files:
        s = ','.join(repr(p.as_string()) for p in result.untracked_files)
        dlb.di.inform(f'repository contains {len(result.untracked_files)} '
                      f'untracked file(s): {s}', level=dlb.di.WARNING)

# Check the syntax of all tag names and that local annotated tags match with annoated tags in remote 'origin'.

import dlb.ex
import dlb_contrib.git

with dlb.ex.Context():
    class GitCheckTags(dlb_contrib.git.GitCheckTags):
        ANNOTATED_TAG_NAME_REGEX = r'v(0|[1-9][0-9]*)(\.(0|[1-9][0-9]*)){2}'  # e.g. 'v1.23.0'
    version_tag_names = set(GitCheckTags().start().commit_by_annotated_tag_name)
module dlb_contrib.gnumake[source]

Parse GNU Make rules.

GNU Make:https://www.gnu.org/software/make/
Make:https://pubs.opengroup.org/onlinepubs/009695399/utilities/make.html
Tested with:GNU Make 4.2.1
License:LGPL-3.0-or-later

Provided objects:

Usage example:

import sys
import dlb.fs
import dlb_contrib.gnumake

makefile = dlb.fs.Path(...)

sources = set()
with open(makefile.native, 'r', encoding=sys.getfilesystemencoding()) as f:
    for r in dlb_contrib.gnumake.sources_from_rules(f):
        sources |= set(r)
module dlb_contrib.gnubinutils[source]

Manipulate static libraries and ELF files with GNU Binutils.

GNU Binutils:https://www.gnu.org/software/binutils/
Tested with:GNU Binutils for Debian 2.31.1
Executable:'ar'
License:LGPL-3.0-or-later

Provided tools:

Usage example:

import dlb_contrib.gnubinutils

with dlb.ex.Context():
    ...  # create 'a.o', 'b.o'
    dlb_contrib.gnubinutils.Archive(
        object_files=['a.o', 'b.o'],
        archive_file='libexample.a'
    ).start()
module dlb_contrib.clike[source]

Check elements of the syntax, generate code and build compilers and linkers with a common interface for the C language family.

C:ISO/IEC 9899:1999 (E)
C++:ISO/IEC 14882:1998 (E)
License:LGPL-3.0-or-later

Provided tools:

Other objects:

Usage example:

import dlb.ex
import dlb_contrib.clike

name = ...
assert dlb_contrib.clike.SIMPLE_IDENTIFIER_REGEX.match(name)

... = dlb_contrib.clike.string_literal_from_bytes('Tête-à-tête'.encode())
# '"T\\xC3\\xAAte-\\xC3\\xA0-t\\xC3\\xAAte"'

class GenerateVersionFile(dlb_contrib.clike.GenerateHeaderFile):
    WD_VERSION = ...

    def write_content(self, file):
        wd_version = \
            dlb_contrib.clike.string_literal_from_bytes(self.WD_VERSION.encode())
        file.write(f'\n#define APPLICATION_VERSION {wd_version}\n')

class CCompiler(dlb_contrib.clike.ClikeCompiler):
    EXECUTABLE = 'specific-cc'

    async def redo(self, result, context):
        if len(result.object_files) > len(result.source_files):
            raise ValueError("'object_files' must be of at most the same length as 'source_files'")
        optional_object_files = result.object_files + (None,) * (len(result.source_files) - len(result.object_files))

        included_files = set()
        for source_file, optional_object_file in zip(result.source_files, optional_object_files):
            with context.temporary() as temp_object_file:
                await context.execute_helper(self.EXECUTABLE, ...,
                                             '-o', temp_object_file, source_file)
                included_files |= ...
                if optional_object_file is not None:
                    context.replace_output(optional_object_file, temp_object_file)
        result.included_files = sorted(included_files)

with dlb.ex.Context():
    GenerateVersionFile(file='Version.h').start()
    CCompiler(source_files=['main.c'], object_files=['main.c.o']).start()
module dlb_contrib.gcc[source]

Compile and link languages of the C family with the GNU Compiler Collection (with the help of the linker from the GNU Binutils).

GCC:https://gcc.gnu.org/
GNU Binutils:https://www.gnu.org/software/binutils/
Tested with:gcc 8.3.0
Executable:'gcc'

'g++'
License:LGPL-3.0-or-later

Provided tools:

Other objects:

Usage example:

import dlb.ex
import dlb_contrib.gcc

with dlb.ex.Context():
    source_directory = dlb.fs.Path('src/')
    output_directory = dlb.fs.Path('build/out/')

    compile_results = [
        dlb_contrib.gcc.CCompilerGcc(
            source_files=[p],
            object_files=[output_directory / p.with_appended_suffix('.o')],
            include_search_directories=[source_directory]
        ).start()
        for p in source_directory.iterdir(name_filter=r'.+\.c', is_dir=False)
    ]

    dlb_contrib.gcc.CLinkerGcc(
        object_and_archive_files=[r.object_files[0] for r in compile_results],
        linked_file=output_directory / 'application'
    ).start()
module dlb_contrib.msbatch[source]

Query environment variables after the execution of a Microsoft batch file.

cmd:https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/cmd
Executable:'cmd.exe'
License:LGPL-3.0-or-later

Provided tools:

Usage example:

import dlb.ex
import dlb_contrib.msbatch

# content of 'setup.bat':
#   @call ...
#   @if %errorlevel% neq 0 exit
#   @cd %1
#   @python3 -m dlb_contrib.exportenv

with dlb.ex.Context():
    ... = dlb_contrib.msbatch.RunEnvBatch(batch_file='setup.bat') \
              .exported_environment['LIB']
module dlb_contrib.mscrt[source]

Handle command-lines for programs using the Microsoft C runtime.

License:LGPL-3.0-or-later

Provided objects:

Usage example:

import dlb_contrib.mscrt

cmdline = dlb_contrib.mscrt.list2cmdline(['a', 'b c', '"'])
# *cmdline* is the command line that is interpreted by the Microsoft C runtime as the representation
# of the command-line arguments ['a', 'b c', '"']
module dlb_contrib.msvc[source]

Compile and link the Microsoft dialects of C and C++ with Microsoft Visual Studio platform toolset.

Microsoft Visual Studio Community:
 https://visualstudio.microsoft.com/de/thank-you-downloading-visual-studio/?sku=Community
Visual Studio platform toolset:
 https://docs.microsoft.com/en-us/cpp/build/building-on-the-command-line?view=vs-2019
cl:https://docs.microsoft.com/en-us/cpp/build/reference/compiling-a-c-cpp-program?view=vs-2019
link:https://docs.microsoft.com/en-us/cpp/build/reference/linking?view=vs-2019
Tested with:MSVC v142 (part of Microsoft Visual Studio Community 2019)
Executable:'cl.exe'

'link.exe'
License:LGPL-3.0-or-later

Provided tools:

Usage example:

import dlb.fs
import dlb.ex
import dlb_contrib.msvc

def setup_paths_for_msvc(context):
    # VCINSTALLDIR must be defined, the other environment variables are set by build/setup.bat with the help of
    # %VCINSTALLDIR%\VC\Auxiliary\Build\vcvars*.bat.
    context.env.import_from_outer('VCINSTALLDIR', pattern=r'.+\\',
                                  example='C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\')
    environment = dlb_contrib.msbatch.RunEnvBatch(batch_file='build/setup.bat').start().environment

    install_directory = dlb.fs.Path(dlb.fs.Path.Native(environment['VCTOOLSINSTALLDIR']), is_dir=True)
    binary_directory = install_directory / 'bin/Hostx64/x64/'
    context.helper['cl.exe'] = binary_directory / 'cl.exe'
    context.helper['link.exe'] = binary_directory / 'link.exe'

    context.env.import_from_outer('SYSTEMROOT', pattern=r'.+', example='C:\\WINDOWS')
    context.env.import_from_outer('INCLUDE', pattern=r'[^;]+(;[^;]+)*;?', example='C:\\X;D:\\Y')
    context.env.import_from_outer('LIB', pattern=r'[^;]+(;[^;]+)*;?', example='C:\\X;D:\\Y')
    context.env['INCLUDE'] = environment['INCLUDE']
    context.env['LIB'] = environment['LIB']

with dlb.ex.Context():
    setup_paths_for_msvc(dlb.ex.Context.active)

    source_directory = dlb.fs.Path('src/')
    output_directory = dlb.fs.Path('build/out/')

    compile_results = [
        dlb_contrib.msvc.CCompilerMsvc(
            source_files=[p],
            object_files=[output_directory / p.with_appended_suffix('.o')],
            include_search_directories=[source_directory]
        ).start()
        for p in source_directory.iterdir(name_filter=r'.+\.c', is_dir=False)
    ]

    object_files = [r.object_files[0] for r in compile_results]
    dlb_contrib.msvc.LinkerMsvc(
         linkable_files=object_files,
         linked_file=output_directory / 'application.exe'
    ).start()
module dlb_contrib.pkgconfig[source]

Query paths of installed librares with pkg-config.

pkg-config:https://www.freedesktop.org/wiki/Software/pkg-config/
Tested with:pkg-config 0.29
Executable:'pkg-config'
License:LGPL-3.0-or-later

Provided tools:

Other objects:

Usage example:

import dlb.ex
import dlb_contrib.pkgconfig

class PkgConfig(dlb_contrib.pkgconfig.PkgConfig):
    LIBRARY_NAMES = ('gtk+-3.0',)
    VERSION_CONSTRAINTS_BY_LIBRARY_NAME = {'gtk+-3.0': ['> 3.0.1', '< 4.0']}

with dlb.ex.Context():
    result = PkgConfig().start()
    ... = result.library_filenames
module dlb_contrib.doxygen[source]

Generate documents from source code with Doxygen.

Doxygen:https://www.doxygen.nl/
Tested with:Doxygen 1.8.13
Executable:'doxygen'
License:LGPL-3.0-or-later

Provided tools:

Other objects:

Usage example:

import dlb.ex
import dlb_contrib.doxygen

with dlb.ex.Context():
    dlb_contrib.doxygen.Doxygen(
        # contains '${{source_directories}}', '${{output_directory}}':
        configuration_template_file='Doxyfile',
        source_directories=['src/'],  # replaces '${{source_directories}}' in Doxyfile
        output_directory='build/out/'
    ).start()
module dlb_contrib.doxypress[source]

Generate documents from source code with DoxyPress.

DoxyPress:https://github.com/copperspice/doxypress/
Tested with:DoxyPress 1.3.8
Executable:'doxypress'
License:LGPL-3.0-or-later

Provided tools:

Other objects:

Usage example:

import dlb.ex
import dlb_contrib.doxypress

with dlb.ex.Context():
    dlb_contrib.doxypress.DoxyPress(
        # contains '${{source_directories}}', '${{output_directory}}':
        project_template_file='doxypress.json',
        source_directories=['src/'],  # replaces '${{source_directories}}' in project file
        output_directory='build/out/'
    ).start()
module dlb_contrib.sh[source]

Run commands with the POSIX sh shell - the standard command language interpreter.

sh:https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html
Executable:'sh'
License:LGPL-3.0-or-later

Provided tools:

Other objects:

Usage example:

import dlb.ex
import dlb_contrib.sh

class PrintString(dlb_contrib.sh.ShScriptlet):
    SCRIPTLET = "echo echoed: " + dlb_contrib.sh.quote('a $ is a $')

with dlb.ex.Context():
    ... = PrintString().start().processed_output.decode()  # 'echoed: a $ is a $\n'
module dlb_contrib.strace[source]

Detect read and written filesystem objects of a running process with strace.

strace:https://strace.io/
Tested with:strace 4.26
Executable:'strace'
License:LGPL-3.0-or-later

Provided tools:

Other objects:

Usage example:

from typing import Iterable, List, Tuple, Union
import dlb.ex
import dlb_contrib.strace

class ShowContent(dlb_contrib.strace.RunStraced):
    EXECUTABLE = 'bash'

    def get_command_line(self) -> Iterable[Union[str, dlb.fs.Path, dlb.fs.Path.Native]]:
        return ['-c', '-', 'cat  -- *', 's']

with dlb.ex.Context():
    ... = ShowContent().start().read_files
module dlb_contrib.tex[source]

Typeset documents with TeX and LaTeX implementations based on web2c and kpathsea.

TeX:https://www.tug.org/books/index.html#texbook

https://ctan.org/tex-archive/systems/knuth/dist/tex/texbook.tex
LaTeX:https://www.latex-project.org/
pdflatex:https://www.tug.org/applications/pdftex/
Tested with:pdfTeX 3.14159265-2.6-1.40.19 with kpathsea version 6.3.1/dev
Executable:'tex'

'latex'
License:LGPL-3.0-or-later

Provided tools:

Other objects:

Usage example:

import dlb.ex
import dlb_contrib.tex

with dlb.ex.Context():
    output_directory = dlb.fs.Path('build/out/')

    dlb_contrib.tex.Latex(
        toplevel_file='src/report.tex', output_file=output_directory / 'report.dvi',
        input_search_directories=['src/'],
        state_files=[output_directory / 'report.aux',
        output_directory / 'report.toc']
    ).start()

Footnotes

[1]Possible application: monitoring of the build progress on a build server.
[2]The update of mtime for an mmap’d file conforming to ISO 1003.1-2008 after a write to the mapped memory is only guaranteed via msync(). Therefore such a write operation is not considered complete before the next call of msync() (which may never happen). Actual behaviour of mmap on different operating systems (2018): https://apenwarr.ca/log/20181113.
[3]Especially, mtime is not manually set with touch -t or any tool that uses a coarser time resolution than the effective mtime resolution. See touch and utimensat().
[4]Linux currently (2020) uses ktime_get_coarse_real_ts64() as time source for its (optional) mtime updates, which returns the system-wide realtime clock at the last tick.
[5](1, 2, 3) Some filesystems support mount options to sacrifice this guaranteed for performance. Example: Ext4 with mount option lazytime.
[6]adjtime() is not covered by ISO 1003.1-2008. It originated in 4.3BSD and System V. For many operating systems it states “the clock is always monotonically increasing” (Linux, OpenBSD, FreeBSD).