Reference¶

Terms¶

run of dlb

The execution of a part of a script inside the same root context.

tool

Describes the abstract behaviour and the way a tool is run by a sub class of dlb.ex.Tool.

It is concretized by instantiation.

tool instance

A (concrete) instance of an (abstract) tool with concrete dependencies.

Can be run in an active context with r = t.run() (once or multiple times), which happens sequentially in the following stages:

start of execution: call of t.run().
dynamic helper resolution: if this is the first instance of the tool being run in the active context: find the absolute path of every dynamic helper (e.g. executable) that might be needed during a redo of any instance of this tool and memorize it.
redo check: decide whether a registered dependency required a redo by inspecting its state and memorize it if it does.
if redo necessary:
1. redo: generate all output dependencies.
2. dependency finalization: update registered dependencies and their state (“memo”) in the run-database.
end of execution:
1. successful after awaiting on r has been completed or
2. unsuccessful after exception has been risen.

Between start and end of execution the tool instance is said to be running.

dynamic helper

An external filesystem object in the managed tree or outside the working tree with a path to be determined at run-time.

A dynamic helper is identified by a relative path (non-directory or directory).

Typical example: globally installed executable (e.g. compiler) with an absolute path determined by a search for the first match in the environment variable PATH.

redo

The phase of the execution of a tool instance t that generates all its output dependencies.

A redo is considered necessary if at least one of the following statements is true:

Any of the output dependencies of t does not exist in the expected form.
Any of the input dependencies of t has changed its existence in the expected form since the start of the last known successful execution of t.
Any of the input dependencies of t has changed a depended-upon property (e.g. filesystem attribute, value of environment variable) since the start of the last known successful execution of t.
Any of the execution parameters t has changed since the start of the last known successful execution of t.

redo miss

A redo miss of a tool instance is an undesirable situation that occurs when the redo check considers a redo not necessary while it actually is.

It is caused by bugs of dlb or dynamic helpers and violations of assumptions (e.g. on the behaviour of the filesystem).

context

An (execution) context describes how running tool instances shall interact with the execution environment outside the working tree and with each other.

It is represented as an instance of dlb.ex.Context used as a context manager.

active context

The innermost context, if any.

root context

The outermost context, if any.

script

A Python program that potentially creates an active context at least once.

working tree

The filesystem tree rooted at a directory (directly) containing a directory .dlbroot/ (a symbolic link to directory is not considered a directory here).

management tree

The filesystem tree rooted at .dlbroot/ in the root of the working tree.

Do not modify its content manually unless you are told so by dlb.

managed tree

The filesystem tree rooted at at the root of the working tree without the management tree. Contains the files involved in the build that are to be managed by dlb.

May and (typically will) be manually modified while there is no active context (e.g. by editing source files).

mtime

The time of last data modification of a filesystem object in the sense of ISO 1003.1-2008.

working tree time

The time according to the mtime of an imaginary filesystem object created at a certain instant (assuming a single filesystem).

mtime update

Setting the mtime of a filesystem object to the current working tree time.

working tree’s system time

The system time used as source for every mtime update of every filesystem object in the working tree (assuming there is one).

effective mtime resolution

The effective mtime resolution for a filesystem object p is defined by the following thought experiment:

p is modified at ideal time t, resulting in a mtime m of p.
p is modified at ideal time t + dt, resulting in a mtime m + dm of p.
The effective mtime resolution for p is the minimum dm > 0 for any pair of t and dt > 0.

Nominal resolution of timestamps for some filesystems: XFS: 1 ns, NTFS: 100 ns, ext2: 1 s, FAT32: 2 s. The effective mtime resolution depends also on the filesystem driver and the operating system, but it can (nominally) not be finer that the timestamp resolution of the filesystem.

ideal time

The (strictly increasing) physical time at the place the dlb process is running.

non-upwards path

A relative path that has no prefix denoting its parent directory.

Examples: a/../b is an non-upwards path, a/../../a/b is not.

collapsable path

A path p of an existing filesystem object with the following property: No prefix path of p’ that ends with a component other than .. is the path of a symbolic link, where p’ is p with all . components removed.

Example: a/b/../c/.. is collapsable if and only if neither a/b/ nor a/b/../c is a symbolic link.

canonical-case path

A path whose components are all exactly as listed in their parent directory.

On a case-insensitive filesystem or directory, multiple paths that differ in case or character encoding can point to the same filesystem object. Only one of them is a canonical-case path.

normal path

A path without .. components.

working tree path

The normal path of an existing filesystem object relative to the working tree’s root.

managed tree path

A working tree path of a file system object in the managed tree.

run-database

The database in the management tree that stores information on the current and past runs of dlb, primarily related to dependencies.

Its removal (permitted when dlb is not running) typically leads to unnecessary redos in the following two runs.

true input dependency

A true input dependency of a tool instance t is an input of t that is not known to have been generated by a previous running tool instance (in the current or a previous run of dlb).

redo-safe

An action (e.g. a modification of the managed tree) is said to be redo-safe if it cannot lead to a redo miss for any tool instance in the current run or any future run of dlb.

benign managed tree modification

A modification of the managed tree is benign, if it consist only of an arbitrary number of the following actions in any order:

Remove or create a filesystem object (this includes symbolic links and hard links)
Write to a regular file

Examples of modifications of the managed tree that are no benign managed tree modifications:

Replace a regular file by any other one with mv (does not update mtime of the target)
Swap two directories
Set the mtime of a filesystem object to something different from the current working tree time

Layout of working tree¶

The directory .dlbroot/ is mandatory — it marks its parent directory as the root of a dlb working tree. Everything else is optional. dlb does never touch filesystem objects outside .dlbroot/ unless explicitly directed by a script to do so.

The directory .dlbroot/u/ is optional. dlb does never touch its content. Each regular file or a symbolic links to a regular file in .dlbroot/u/ whose name ends in .zip should be a zip archive loadable by zipimport. You can place dlb as .dlbroot/u/dlb.zip in this directory (under version control). This makes the working tree almost self-contained; only an external Python interpreter is needed.

If you use Git for version control which does not support empty directories, add .dlbroot/o (as created by dlb) or any file in .dlbroot/u/.

The lines marked with * show filesystem objects only given as an example.

Before first run of a dlb script:

.dlbroot/
src/                    *
   a.c                  *
   a.h                  *
   b.c                  *
test/                   *
...

During a run of a dlb script:

.dlbroot/
    o                   empty regular file, used to probe the "current" mtime
    runs-*.sqlite       run-database
    t/                  temporary files
        a.o             *
        b.o             *
 src/                   *
   a.c                  *
   a.h                  *
   b.c                  *
 test/                  *
 out/                   *
   p                    *
 dist/                  *
 ...

After a successful run of a dlb script:

.dlbroot/
    o                   empty regular file
    runs-*.sqlite       run-database
 src/                   *
   a.c                  *
   a.h                  *
   b.c                  *
 test/                  *
 out/                   *
   a.o                  *
   b.o                  *
 dist/                  *
   p                    *
 ...

Top-level specification¶

Assumptions¶

An assumption (A-…) is a requirement for the intended behaviour of dlb that cannot be checked at runtime in a reliable and efficient way.

For every assumption at set of acceptable effects of its violation is given:

repair

The working tree needs manual repair.

obscure-fail

A build may fail in an obscure way. The diagnostic messages do neither clearly indicate the problem nor a way to fix it.

vulnerable

The build becomes vulnerable to attacks. Running tool instances might overwrite any filesystem object the process has permission to, potentially with information generated during the build or stored in the working tree.

redo-miss

A redo miss in the current or a future run of dlb can occur.

graceful-fail

A build may fail in a meaningful way. The diagnostic messages clearly indicate the problem or a way to fix it.

Modification of the working tree¶

A-A1 (access to management tree)

Except for modifications by dlb internals, the management tree is not modified while dlb is running and only as suggested by diagnostic messages of dlb.

Changing the absolute path of the working tree’s root is considered a modification.

Acceptable when violated:

repair (e.g. after the run-database has become corrupted)
obscure-fail (e.g. after intermediate files have been modified)
vulnerable (e.g. by a symlink attack)
redo-miss (e.g. after the run-database has been is tampered with)
graceful-fail

A-A2 (access to managed tree)

While a tool instance in running, the managed tree is modified only by running tool instances.

Changing the absolute path of the working tree’s root is considered a modification.

Acceptable when violated:

obscure-fail (e.g. when a running tool instance “sees” an intermediate state of files that are input dependencies)
vulnerable (e.g. by a symlink attack)
graceful-fail

A-A3 (manual modification of mtime)

Except from modifications requested by a running tool instance, every modification of the mtime of a filesystem object in the working tree is a mtime update. [3]

Acceptable when violated:

redo-miss
graceful-fail

A-A4

No part of the filesystem outside of the working tree is modified while a tool instance t is running, unless it cannot affect the behaviour of t.

Acceptable when violated:

obscure-fail
vulnerable
redo-miss
graceful-fail

Filesystems behaviour of working tree¶

A-F1 (one filesystem)

Every filesystem object w/p, where w is the path of the working tree’s root and p is a relative path without .. components, resides on the same (local or remote) file system.

Acceptable when violated:

redo-miss
graceful-fail

A-F2 (mtime update at creation)

Every creation of a filesystem object in the working tree updates its mtime. [5]

Acceptable when violated:

redo-miss
graceful-fail

A-F3 (mtime update at write to regular file)

Every write to regular file in the working tree updates its mtime as soon as it is completed. [5] [2]

Between start and completion of a write, a reader of the file may observe an intermediate state of the file’s content.

[--------------] content change

^              ^
start          mtime update (write complete)

-------------------> ideal time

Acceptable when violated:

redo-miss
graceful-fail

A-F4 (mtime update for directory)

Every creation, removal, renaming and attribute change of a filesystem object in the working tree updates the mtime of the (directly) containing directory. [5]

Acceptable when violated:

redo-miss
graceful-fail

A-F5 (moving is atomic)

Moving a regular file, a directory or a symbolic link in the working tree to a different directory in the working tree is possible in an reasonably secure, efficient and atomic operation that does not affect the moved object’s filesystem attributes (including mtime in full resolution).

Acceptable when violated:

graceful-fail

A-F6 (moving makes invisible)

Immediately after a regular file, a directory or a symbolic link in the working tree has been successfully moved to a different directory within the same working tree, no other process “sees” it in the original directory.

Acceptable when violated:

obscure-fail
graceful-fail

A-F7 (no corruption)

A filesystem object in working tree is never corrupted (e.g. by failure of software, memory or power).

Acceptable when violated:

repair
obscure-fail
vulnerable
redo-miss
graceful-fail

Timing and concurrency¶

A-T1 (working tree time exists)

The mtime of every filesystem object in the working tree is updated from the same system time (local or remote), the working tree’s system time. Whenever a mtime update occurs for a filesystem object p after one occurs for filesystem object q, where p and q are in the working tree, the mtime of p is not later than the mtime of q. [4]

Acceptable when violated:

redo-miss
graceful-fail

A-T2 (working tree time mostly monotonically increasing)

With the exception of rare backward jumps, the working tree time is monotonically increasing.

The time between consecutive backward jumps is longer than the duration of a run of dlb.

Acceptable when violated:

redo-miss (the finer the effective mtime resolution and the less frequent modification of inputs files are, the less likely is a redo miss due to this violation)
graceful-fail (this is desirable since it forces correction before it can cause redo misses in the future)

A-T3 (effective mtime resolution)

The regular file o in the management tree has an effective mtime resolution no coarser than 100 ms.

Acceptable when violated:

graceful-fail

A-T4 (working tree time of true input dependencies in the past)

The mtime of every filesystem object in the managed tree that is an true input dependency of a tool instance t is earlier than the time t starts running.

Acceptable when violated:

graceful-fail

Dependencies¶

A-D1 (regular files): Most of the filesystem objects in the managed tree that serve as input dependency of tool instances are regular files.

A-D2 (no links to input dependencies)

A filesystem object in the managed tree that serves as an input dependency of a tool instance has no hard link or symbolic link pointing to it in the managed tree.

Acceptable when violated:

redo-miss
graceful-fail

A-D3 (no implicit symbolic links in paths)

A filesystem object in the managed tree that serves as a dependency of a tool instance t does not have a parent directory p in its path that is a symbolic link, unless p is an input dependency of t and in the working tree.

Acceptable when violated:

redo-miss
graceful-fail

Guarantees¶

A guarantee (G-…) is a specification of behaviour observable by the user.

Dependencies¶

G-D1 (no redo miss when working tree time monotonic)

A benign managed tree modification is redo-safe, provided the assumptions A-F1, A-F2, A-F3, A-F4 hold and the working tree time is monotonically increasing (at least since the oldest mtime of all filesystem objects that are true input dependencies of a tool instance).

This is true even when assumption A-A2 is violated.

G-D2 (no redo miss when file size changes): Modifying the content of a regular file in the managed tree while a tool instance is running (in violation of A-A2) is redo-safe if it also changes the size of the regular file.

G-D3 (redo miss unlikely when modification intervals relatively long)

A benign managed tree modification is likely to be redo-safe, provided the assumptions A-F1, A-F2, A-F3, A-F4 hold and the “modification intervals are relatively long” for every filesystem object that is a true input dependency of a tool instance.

Here, a modification interval of a filesystem object p is considered to be relatively long if it is unlikely that the working tree time at the ideal time t is the same as at t + T, where T is the ideal time between two consecutive mtime updates of p.

This is true even when assumption A-A2 is violated.

G-D4: When assumption A-T2 is violated at a certain time at the start of the “redo check” phase of a running tool instance, the build is aborted with a meaningful diagnostic message. [7]

Timing and concurrency¶

G-T1 (active context exit): An active context is not left as long as a tool instance is running in it.

G-T2 (root context exit): A root context is not left other than by a raised exception before there has been a time window with the following property: The mtime of a regular file o in the management tree would have been different from the mtime of the last filesystem object modified by a running tool instance.

G-T3 (multiple dlb processes): When multiple scripts are run by different processes on the same working tree, at most one of them is in an active context at the same time.

G-T4 (threads): dlb does not create threads.

dlb — Command-line utility for convenience¶

The command-line utility dlb calls a dlb script; the real work is done entirely by the dlb script with the help of the package dlb.

dlb can save you some typing, however, with the following features:

Changes the current working directory to the working tree’s root from anywhere in the working tree.
Remembers command-line arguments of the last successful call.
Adds ZIP archives to the module search path.

See dlb --help for details.

`dlb` — Version¶

dlb.__version__¶

The version of dlb as a non-empty string. Example: '1.2.3.

Conforms to PEP 396, PEP 386 and PEP 440.

Contains the string '.dev' if and only if this a development version. For a development version, __version__ looks like this: '1.2.3.dev30+317f'.

dlb.version_info¶

The version of dlb as a non-empty tuple of non-negative integers and strings 'a' … 'z', similar to sys.version_info. Example: (1, 2, 3).

Contains at least 3 members, the first 3 of them integers. Contains more than 3 members if and only if this is an unreleased version. For an unreleased version, version_info looks like this: (1, 2, 3, 'c', 4).

Use it like this to compare (released) versions:

assert (1, 0, 3) <= dlb.version_info  < (2,)

Note: For a development version, version_info carries less information than __version__.

`dlb.fs` — Filesystem paths¶

This module provides classes to represent and access filesystem objects in a safe and platform-independent manner, most prominently dlb.fs.Path and its subclasses.

Path objects¶

class dlb.fs.Path¶

A dlb.fs.Path represents the path of a filesystem object in a platform-independent manner and expresses whether the object is a directory or not. All operations on instances of dlb.fs.Path show the same behaviour on every platform (with the exception of native). Its instances are immutable and hashable (they can be used in sets or as keys in dictionaries).

All represented paths are either absolute or relative to the working directory of a process.

The interface is similar to pathlib and conversion from and to pathlib paths is supported.

If the path represented by a dlb.fs.Path is meaningful as a concrete path on the platform the code is running on, native returns it in the form of a dlb.fs.Path.Native instance, which can then be used to access the filesystem (like a pathlib.Path).

On all platform the following properties hold:

'/' is used as a path component separator.
A path does not contain '\0'.
A path is absolute if and only if it starts with '/'; it is relative if and only if it is not absolute.
A component '..' means the parent directory of the path before it.
A component '.' means the directory of the path before it; a non-empty path with all such components removed is equivalent to the original one.
A sequence of two or more consequent '/' is equivalent to a single '/', except at the beginning of the path.
At the beginning of the path:
- Three or more consequent '/' are equivalent to a single '/'
- Exactly two consequent '/' (followed be a character other than '/') means that the following component is to be interpreted in a implementation-defined manner (e.g. ISO 1003.1-2008 or UNC paths)
If the platform supports symbolic links, it resolves them as specified in ISO 1003.1-2008 (which means that '..' cannot be collapsed without knowing the filesystem’s content).

dlb.fs.Path instances are comparable with each other by =, < etc. They are also comparable with strings and pathlib.PurePath. Comparison is done case-sensitively and component-wise, observing the equivalence relations described below. A non-directory path is smaller than an otherwise identical directory path. If a directory path d is a prefix of another path p, then d < p. Any relative path is smaller than any absolute path.

Usage example:

p = dlb.fs.PortablePath('a/b/c/') / 'x/y'

p.relative_to(...)

... = str(p.native)

with p.native.open() as f:
    f.readline()

The dlb.fs.Path class supports the following methods and attributes:

Path(path[, is_dir=None])¶

Constructs a path from another path or a string.

If path is a string, it is interpreted as the string representation of a path in Posix style with / as a component separator. It must not by empty. If path is a pathlib.Path, it must be either absolute or relative. If path is a sequence, all its members are converted to str and are interpreted as path components; its first component must be '' for a relative path.

If is_dir is None, the ending of path determines whether is considered a directory path or not; it is if it ends with '/' or a '.' or '..' component.

If is_dir is True, the path is considered a directory path irrespective of path.

If is_dir is False, the path is considered a non-directory path irrespective of path However, if path represents '.' or endwith a '..' component, a ValueError exception is raised.

Parameters:	path (str \| `Path` \| `pathlib.PurePath`) – portable string representation or path object is_dir (NoneType \| bool) – `True` if this is a directory path, `False` if not and `None` for derivation from path
Raises:	TypeError – if path is neither a string nor a path ValueError – if path is an empty string ValueError – if path is a `pathlib.PurePath` which is neither absolute nor relative

Examples:

>>> p = Path('a/b/').is_dir()
True

>>> p = Path(pathlib.PureWindowsPath('C:\\Windows'), is_dir=True)
>>> p
Path('/C:/Windows/')
>>> p.is_dir()
True

>>> p = Path('x/y/..', is_dir=False)
Traceback (most recent call last):
...
ValueError: cannot be the path of a non-directory: 'x/y/..'

>>> Path('x/y/z.tar.gz')[:-2]
Path('x/')

>>> Path('x/y/z.tar.gz').parts[-1]
'z.tar.gz'

is_dir()¶

Returns:	`True` if and only if this represents the path of a directory.
Return type:	bool

is_absolute()¶

Returns:	`True` if and only if this represents an absolute path.
Return type:	bool

Note

While Posix considers paths starting with exactly two '/' not as absolute paths, this class does (and so does pathlib).

is_normalized()¶

Returns:	`True` if and only if this represents a normalized path (i.e. it contains no `'..'` components)
Return type:	bool

relative_to(other, collapsable=False):

Returns a version of this path relative to the directory path represented by other.

If collapsable is False, this path must be a prefix of other. If collapsable is True, other is treated as a collapsable path and the minimum of necessary .. components is prepended.

Return type:	class of this object
Raises:	ValueError – if this is a non-directory path ValueError – if other is not a prefix of this

iterdir(name_filter='', recurse_name_filter=None, follow_symlinks=True, cls=None)¶

Yields all path objects of the directory contents denoted by this path and matched by the name filters. The paths are duplicate-free and in a defined and repeatable order, but not necessarily sorted. Their class is the class of this object if cls is None and cls otherwise.

The path of an existing filesystem object is eventually yielded if and only if

its name matches the name filter name_filter and
it is contained in a matched directory.

A directory is a matched directory if and only if it is the directory d denoted by this path or a direct subdirectory of a matched directory whose name matches the name filter recurse_name_filter. If follow_symlinks is True, a symbolic link to an existing directory is considered a direct subdirectory of the director containing the symbolic link. If follow_symlinks is False or the target of the symbolic link does not exist, it is considered a non-directory.

name_filter and recurse_name_filter are name filters. A name filter can be

None — no name matches this filter
a callable c accepting exactly one argument — a name n matches this filter if and only if bool(c(n)) is True
a compiled regular expression r — a name n matches this filter if and only if r.fullmatch(n)) is not None
a non-empty regular expression string s— a name n matches this filter if and only if re.compile(s).fullmatch(n)) is not None
an empty string — every name matches this filter

Example:

for p in dlb.fs.Path('src/').iterdir(name_filter=r'(?i).+\.cpp', recurse_name_filter=lambda n: '.' not in n):
    ...

Return type:	cls \| class of this object
Raises:	TypeError – if cls is neither `None` nor a subclass of `dlb.fs.Path` TypeError – if name_filter or recurse_name_filter are not both name filters ValueError – if this is a non-directory path

iterdir_r(name_filter='', recurse_name_filter=None, follow_symlinks=True, cls=None)¶: Like iterdir(), but all returns paths are relative to this path.

list(name_filter='', recurse_name_filter=None, follow_symlinks=True, cls=None)¶

Returns all paths yielded by iterdir() as a sorted list.

Example:

>>> dlb.fs.NoSpacePath('src/').list(name_filter=r'(?i).+\.cpp')
[NoSpacePath('src/Clock.cpp'), NoSpacePath('src/main.cpp')]

list_r(name_filter='', recurse_name_filter=None, follow_symlinks=True, cls=None)¶

Returns all paths yielded by iterdir_r() as a sorted list.

Example:

>>> dlb.fs.NoSpacePath('src/').list(name_filter=r'(?i).+\.cpp')
[NoSpacePath('Clock.cpp'), NoSpacePath('main.cpp')]

__getitem__(key):

A subpath (a slice of the path).

The resulting path is absolute (with the same anchor) if and only if the slice starts at 0. The resulting path is a non-directory path if and only if it contains the last component and if this path is a non-directory path.

The slice step must be positive.

Examples:

>>> dlb.fs.Path('src/comp/lib/Core.cpp')[:-2]
Path('src/comp/'

>>> dlb.fs.Path('src/comp/..')[:-1]
Path('src/comp/'

Parameters:	key (`slice`) – slice of components (indices into `parts`)
Return type:	class of this object
Returns:	subpath
Raises:	TypeError – if key is not a slice ValueError – if this is an absolute path and key is an empty slice

parts¶

A tuple giving access to the path’s various components:

>>> p = Path('/usr/bin/python3')
>>> p.parts
('/', 'usr', 'bin', 'python3')

Return type:	tuple(str)

native¶

This path as a native path. Use it to access the filesystem:

p = Path('/usr/bin/')
with open(p.native) as f:
   ...

This attribute cannot be written.

Return type:	`dlb.fs.Path.Native`
Raises:	ValueError – if this path is not representable as `Path.Native`

pure_posix¶

This path as a pathlib.PurePosixPath:

>>> p = Path('/usr/bin/')
>>> p.pure_posix
PurePosixPath('/usr/bin')

This attribute cannot be written.

Return type:	`pathlib.PurePosixPath`

pure_windows¶

This path as a pathlib.PureWindowsPath:

>>> p = Path('/C:/Program Files/')
>>> p.pure_windows
PureWindowsPath('C:/Program Files')

This attribute cannot be written.

Return type:	`pathlib.PureWindowsPath`

class Path.Native¶

A native path whose instances can be used much like ones from pathlib.Path and is a os.PathLike.

For each subclass P of dlb.fs.Path there is a corresponding subclass P.Native which imposes the same restrictions on its representable paths as P.

If Q is a subclass of P and P is a subclass of dlb.fs.Path, then Q.Native is a subclass of P.Native.

Example (on a Posix system):

>>> dlb.fs.NoSpacePath.Native('/tmp/x y')
Traceback (most recent call last):
...
ValueError: invalid path for 'NoSpacePath': '/tmp/x y' (must not contain space)

In contrast to pathlib.Path, conversion to string is done in a safe way: relative paths are guaranteed to start with '.'.

Example (on a Posix system):

>>> str(Path.Native('-rf'))
'./-rf'

Instances of dlb.fs.Path.Native and its subclasses should not be constructed directly, but by accessing dlb.fs.Path.native.

Example (on a Posix system):

with open(dlb.fs.NoSpacePath('/tmp/x/a').native) as f:
    ... = f.read()

raw¶

This path as a pathlib.Path. Use it to access the filesystem in an object-oriented manner:

p = Path('/usr/bin/')
... = p.native.raw.is_dir()

This attribute cannot be written.

Constructing a pathlib.Path is an expensive operation. For performance-critical tasks, use p.native and functions for string-like paths instead: e.g. os.path.isdir(p.native) instead of p.native.raw.is_dir().

Restricting paths¶

By subclassing dlb.fs.Path, additional restrictions to the set of value values can be imposed (trying to construct a dlb.fs.Path from an invalid value raises an ValueError exception). A subclass of dlb.fs.Path should implement only check_restriction_to_base().

class dlb.fs.RelativePath¶: A dlb.fs.Path which represents a relative path.

class dlb.fs.AbsolutePath¶: A dlb.fs.Path which represents an absolute path.

class dlb.fs.NormalizedPath¶: A dlb.fs.Path which represents a normalized path (without '..' components).

class dlb.fs.NoSpacePath¶: A dlb.fs.Path whose components do not contain ' '.

class dlb.fs.PosixPath¶

A dlb.fs.Path which represents a POSIX-compliant (ISO 1003.1-2008) paths in its least-constricted form.

Every non-empty string, which does not contain '/' or U+0000 (NUL) is a valid component. Components are separated by '/'.

For every path prefix (in the POSIX sense) {NAME_MAX} and {PATH_MAX} are considered unlimited.

Relevant parts of ISO 1003.1-2008:

section 4.12 Pathname Resolution
section 4.5 File Hierarchy
section 4.6 Filenames
section 4.7 Filename Portability
section 3.267 Pathname
section 3.269 Path Prefix
limits.h

class dlb.fs.PortablePosixPath¶

A dlb.fs.PosixPath which represents a POSIX-compliant (ISO 1003.1-2008) path in its strictest form. Any path whose support is not required by POSIX or is declared as non-portable is considered invalid.

A component cannot be longer than 14 characters, which must all be members of the Portable Filename Character Set.

The length of the string representation of the path is limited to 255 characters.

No absolute path prefix other than '/' is allowed (because implementation-defined).

class dlb.fs.WindowsPath¶

A dlb.fs.Path which represents a Microsoft Windows-compliant file or directory path in its least-constricted form, which is either relative or absolute and is not a reserved non-directory path (e.g. NUL).

It cannot represent incomplete paths which are neither absolute nor relative to the current working directory (e.g. C:a\b and \\name). It cannot represent NTFS stream names, Win32 file namespaces or Win32 device namespaces.

class dlb.fs.PortableWindowsPath¶

A dlb.fs.WindowsPath which represents a Microsoft Windows-compliant path in its strictest form.

A component cannot end with ' ' or '.' (except '.' and '..') and cannot be longer than 255 characters. The path cannot not be longer than 259 characters.

class dlb.fs.PortablePath¶

`dlb.di` — Line-oriented hierarchical diagnostic messages¶

In contrast to the logging module, this module focuses on hierarchical structure and unambiguity. Absolute time information (date and time of day) is not output in favour of high-resolution relative times. The output is compact, line-oriented and well readable for humans.

Each message has an associated level, e.g. WARNING, with the same meaning and numerical value as in the logging module. The higher the associated numeric value, the more important the message is considered:

Level Numeric value

CRITICAL 50

ERROR 40

WARNING 30

INFO 20

DEBUG 10

Messages below a global message threshold are not output. The message threshold can be changed any time. It is initialised to INFO if sys.flags.verbose is False and 1 otherwise.

Messages can be nested with message clusters. In the output, the nesting level of a message is expressed by the indentation of its lines.

An precise syntax in enforced to make the output suitable for incremental parsing (e.g. from a named pipe) with the help of simple regular expressions. [1] Any Unicode character except characters from the range U+0000 to U+001F (ASCII control characters) can be used in messages as long as the syntax is not violated.

To output a message, call dlb.di.inform() or enter a context manager instance of dlb.di.Cluster().

Example¶

import dlb.di
...

with dlb.di.Cluster(f"analyze memory usage\n    note: see {logfile.as_string()!r} for details", is_progress=True):
   ram, rom, emmc = ...

   dlb.di.inform(
       f"""
       in use:
           RAM:\t {ram}\b kB
           ROM (NOR flash):\t {rom}\b kB
           eMMC:\t {emmc}\b kB
       """)

   if rom > 0.8 * rom_max:
       dlb.di.inform("more than 80 % of ROM used", dlb.di.WARNING)

This will generate the following output:

I analyze memory usage...
  | note: see 'out/linker.log' for details
  I in use:
    | RAM:              12 kB
    | ROM (NOR flash): 108 kB
    | eMMC:            512 kB
  W more than 80 % of ROM used
  I done.

Syntax¶

Each message starts with a capital letter after indentation according to its nesting level (2 space characters per level) and ends with a '\n' after a non-space character. It can consist of any number of lines: an initial line followed by any number of continuation lines, separated by '␣\n' and the same indentation as the initial line ('␣' means the character U+0020):

message             ::=  single_line_message | multi_line_message
single_line_message ::=  initial_line '\n'
multi_line_message  ::=  initial_line '␣\n' (continuation_line '␣\n')* continuation_line '\n'
indentation         ::=  '␣␣'*

The initial line carries the essential information. Its first letter after the indentation denotes the level of the message: the first letter of the standard names of the standard loglevels of the logging module. An optional relative file path and 1-based line number of an affected regular file follows.

initial_line    ::=  indentation summary_prefix summary summary_suffix
summary_prefix  ::=  level_indicator '␣' [ file_location '␣' ]
summary_suffix  ::=  [ progress_suffix ] [ '␣' relative_time_suffix ]
level_indicator ::=  'C' | 'D' | 'E' | 'I' | 'W'
file_location   ::=  relative_file_path ':' line_number
summary         ::=  summary_first_character [ message_character* summary_last_character ]
progress_suffix ::=  '.' | '...'

The timing information is optional and can be enabled per message. It contains the time elapsed in seconds since the first time a message with enabled timing information was output. Later outputs of timing information never show earlier times. The number of decimal places is the same for all output timing information on a given platform and is at most 6.

relative_time_suffix      ::=  '[+' time_since_first_time_use ']'
time_since_first_time_use ::=  decimal_integer [ '.' decimal_digit decimal_digit* ] 's'

continuation_line           ::=  indentation continuation_line_indicator message_character*
continuation_line_indicator ::=  '␣␣|␣'

relative_file_path               ::=  "'" path_component [ '/' path_component ] "'"
line_number                      ::=  decimal_integer
path_component                   ::=  path_component_character path_component_character*
path_component_character         ::=  raw_path_component_character | escaped_path_component_character
raw_path_component_character     ::=  any Unicode character except from the range U+0000 to U+001F, '/', '\', ':', "'" and '"'
escaped_path_component_character ::=  '\x' hexadecimal_digit hexadecimal_digit

summary_first_character ::=  any summary_last_character except "'" (U+0027) and '|' (U+007C)
summary_last_character  ::=  any message_character except '␣' (U+0020), '.' (U+002E) and ']' (U+005D)
message_character       ::=  any Unicode character except from the range U+0000 to U+001F
decimal_integer         ::=  nonzero_decimal_digit decimal_digit*
nonzero_decimal_digit   ::=  '1' | ... | '9'
decimal_digit           ::=  '0' | nonzero_decimal_digit
hexadecimal_digit       ::=  decimal_digit | 'a' | ... | 'f'

Module content¶

dlb.di.DEBUG¶

dlb.di.INFO¶

dlb.di.WARNING¶

dlb.di.ERROR¶

dlb.di.CRITICAL¶

Positive integers representing standard logging levels of the same names. See the documentation of logging.

In contrast to logging, these are not meant to be changed by the user. Use them to define your own positive integers representing levels like this:

... = dlb.di.INFO + 4  # a level more important than INFO, but not yet a WARNING

dlb.di.set_output_file(file)¶

Set the output file for all future outputs of this module to file and return the old output file.

Parameters:	file (an object with a `write(string)` method) – new output file
Returns:	the previous value, an object with a `write` attribute

dlb.di.set_threshold_level(level)¶

Set the level threshold for all future messaged to level.

Every message with a level below level will be suppressed.

Parameters:	level (int) – new level threshold (positive)

dlb.di.is_unsuppressed_level(level)¶

Is a message of level level unsuppressed be the current level threshold?

Return type:	bool

dlb.di.get_level_indicator(level)¶

Return a unique capital ASCII letter, representing the lowest standard level not lower than level.

Example:

>>> dlb.di.get_level_indicator(dlb.di.ERROR + 1)
'E'

Parameters:	level (int) – level not lower that `DEBUG`

dlb.di.format_time_ns(time_ns)¶

Return a string representation for a time in seconds, rounded towards 0 approximately to the resolution of time.monotonic_ns(). The time time_ns is given in nanoseconds as an integer.

The number of decimal places is fixed for all calls. It is a platform-dependent value in the range of 1 to 6.

dlb.di.format_message(message, level)¶

Return a formatted message with aligned fields, assuming nesting level 0.

First, empty lines are removed from the beginning and the end of message and trailing white space characters is removed from each line. After that, the first line must not start with '␣', "'", "|", '.' or "]". If must not end with "." or "]". Each non-empty line after the first line must start with at least 4 space characters after than the indentation of the first line. Example: If the first line is indented by 8 space characters, each following non-empty line must start with at least 12 space characters.

message can contain fields. A field is declared by appending '\t' or '\b'. A field whose declaration ends with '\t' is left aligned, one whose declaration ends with '\t' is right aligned over all lines of the message. In the return value, the '\t' or '\b' are not present, but their “positions” are aligned over all lines of the message.

Examples:

>>> dlb.di.format_message('\njust a moment! ', dlb.di.WARNING)
'W just a moment!'

>>> dlb.di.format_message(
...   """
...   summary:
...       detail: blah blah blah...
...       see also here
...   """, dlb.di.INFO)
'I summary: \n  | detail: blah blah blah... \n  | suggestion'

>>> m = ''.join(f"\n    {n}:\t {s} =\b {v}\b{u}" for n, s, v, u in metrics)
>>> print(dlb.di.format_message('Halstead complexity measures:' + m, dlb.di.INFO))
I Halstead complexity measures:
  | volume:               V =   1.7
  | programming required: T = 127.3 s
  | difficulty:           D =  12.8

Returns:	formatted message conforming to `message` after appending a single `'\n'`
Return type:	str
Raises:	ValueError – if message would violate `message` or if level is invalid

dlb.di.inform(message, *, level: int = INFO, with_time: bool = False)¶

If level is not suppressed, output a message to the output file after the title messages of all parent Cluster instances whose output was suppressed so far.

message is formatted by format_message() and indented according the nesting level. If with_time is True, a relative_time_suffix for the current time is included.

class dlb.di.Cluster(message, *, level=INFO, is_progress=False, with_time=False)¶

A message cluster with message as its title.

When used as a context manager, this defines a inner message cluster with message as its title; entering means an increase of the nesting level by 1.

With is_progress set to False, the output when the context is entered is the same as the output of inform() would be with the same parameters.

With is_progress set to True, a progress_suffix '...' is included in the message when the context is entered. In addition, a message 'done. or 'failed with E.' is output when the context is exited without or with an exception, respectively. See Example.

`dlb.cf` — Configuration parameters¶

To change the behaviour of dlb, change the values of the following variables.

dlb.cf.latest_run_summary_max_count¶

Number of dlb runs to summarize as an integer.

When > 0, a summary of the latest latest_run_summary_max_count dlb runs is output when a root context exits.

dlb.cf.max_dependency_age¶

The maximum age of dependency information in the run-database as a datatime.timedelta object.

Run and dependency information older than max_dependency_age is removed when a root context is entered. max_dependency_age > datetime.timedelta(0) must be True.

dlb.cf.level.RUN_PREPARATION¶

dlb.cf.level.RUN_PREPARATION

dlb.cf.level.RUN_SERIALIZATION¶

dlb.cf.level.REDO_NECESSITY_CHECK¶

dlb.cf.level.REDO_REASON¶

dlb.cf.level.REDO_SUSPICIOUS_REASON¶

dlb.cf.level.REDO_PREPARATION¶

dlb.cf.level.REDO_START¶

dlb.cf.level.REDO_AFTERMATH¶

dlb.cf.level.HELPER_EXECUTION¶

dlb.cf.level.RUN_SUMMARY¶: Assign a message level (a positive integer like dlb.di.INFO) to be used for all message of a given category.

`dlb.ex` — Execution contexts and dependency-aware tool execution¶

Context objects¶

An (execution) context describes how running tool instances shall interact with the execution environment outside the working tree and with each other. E.g:

number of asynchronously running tool instances
search paths for dynamic helper files
environment variables to be imported from os.environ for use in tool instances

It also controls how diagnostic messages of tool instances are handled and helps with filesystem abstraction (e.g. working tree time, case sensitivity of names in the working tree).

A context is represented as an instance of dlb.ex.Context used as a context manager. The context is entered with the call of _enter__() and exited with the return of __exit__().

Contexts can be nested:

import dlb.ex

# no active context

with dlb.ex.Context():                # A: root context, outer context of B, C, D
    # A is the active context
    with dlb.ex.Context():            # B: inner context of A, outer context of C
        # B is the active context
        with dlb.ex.Context():        # C: inner context of A, B
           # C is the active context
    with dlb.ex.Context():            # D: inner context of A
        # D is the active context
    # A is the active context

# no active context

Combine contexts with message clusters to describe what happens in the context:

with dlb.di.Cluster('this happens in the context'), dlb.ex.Context():
    ...

class dlb.ex.Context(path_cls=dlb.fs.Path, max_parallel_redo_count=1, find_helpers=None)¶

An instance does nothing unless used as a context manager.

When used as a context manager, it embodies an (execution) context and activates it:

a root context, if dlb is not yet running;
an inner context of the active context, otherwise.

When a root context is entered, the working directory of the Python process must be a working tree’s root whose absolute path does not contain unresolved symbolic link.

When a context (root or not) is entered, the path of the working tree’s root must be representable as as path_cls. This allows you to impose restrictions on the accepted paths.

If find_helpers is None for a root context, True is used instead. If find_helpers is None for an active context that is not the root context, find_helpers of the root context is used.

Parameters:	path_cls (dlb.fs.Path) – the subclass of `dlb.fs.Path` to be used to represent the working tree’s root max_parallel_redo_count (int) – maximum number of redos started in this context than can be pending at the same time find_helpers – are dynamic helpers not defined explicitly to be searched for in `executable_search_paths`? find_helpers – None \| bool
Raises:	TypeError – if path_cls is not a subclass of `dlb.fs.Path`

Entering or exiting a context may raise the following exceptions:

exception	meaning	when
`NoWorkingTreeError`	the working directory is not a working tree’s root	entering root context
`ManagementTreeError`	the management tree cannot be setup inside the working tree	entering root context
`ValueError`	the working tree’s root path violates the requested restrictions	entering (any) context
`ContextNestingError`	the contexts are not properly nested	exiting (any) context
`WorkingTreeTimeError` \| working tree time behaved unexpectedly \| exiting root context

Note

Most attributes and methods are available “on the class” as well as “on the instance”, and refer to the corresponding attribute of the active context:

with dlb.ex.Context:
    with dlb.ex.Context as c:
        ... = dlb.ex.Context.working_tree_time_ns   # preferred
        ... c.active.working_tree_time_ns           # also possible
        ... c.working_tree_time_ns                  # also possible

The Context class supports the following methods and attributes:

active¶

The active context.

Same on class and instance.

Raises:	NotRunningError – if dlb is not running).

path_cls¶

The subclass of dlb.fs.Path defined in the constructor.

When called on class, it refers to the active context.

Raises:	NotRunningError – if dlb is not running).

max_parallel_redo_count¶

The maximum number of redos started in this context than can be pending at the same time, as defined in the constructor.

When called on class, it refers to the active context.

Raises:	NotRunningError – if dlb is not running).

find_helpers¶

Find dynamic helpers not defined explicitly are in executable_search_paths? This is defined in the constructor.

When called on class, it refers to the active context.

Raises:	NotRunningError – if dlb is not running).

root_path¶

The absolute path to the working tree’s root.

It is an instance of Context.active.path_cls and is representable as an instance of path_cls of the active context and every possible outer context.

Same on class and instance.

Raises:	NotRunningError – if dlb is not running).

executable_search_paths¶

A duplicate-free tuple of absolute directory paths where this process should look for executables according to the operating system.

It is compiled from the members os.get_exec_path() when the root context is entered.

Same on class and instance.

Raises:	NotRunningError – if dlb is not running).

find_path_in(path, search_prefixes=None)¶

Find the first existing and accessible path in search_prefixes and return its absolute path. Returns None if path is not found in search_prefixed.

If path is a dlb.fs.Path with path.is_dir() = True, existing non-directories in search_prefixes are ignored. If path is a dlb.fs.Path with path.is_dir() = False, existing directories in search_prefixes are ignored.

Relative paths in search_prefixes are treated as relative to root_path.

If search_prefixes is None, executable_search_paths is used instead.

Does not raise OSError.

Parameters:	path (`dlb.fs.Path` or anything a `dlb.fs.Path` can be constructed from) – the relative path to find search_prefixes (an iterable other than `str` or `bytes` or `None`) – paths of directories to search in
Returns:	an absolute path or `None`.

working_tree_time_ns¶

The current working tree time in nanoseconds as an integer.

Same on class and instance.

Raises:	NotRunningError – if dlb is not running).

temporary(suffix='', is_dir=False)¶

Return a dlb.ex.Temporary object, representing a temporary regular file (for is_dir = False) or a temporary directory (for is_dir = True) in the management tree with a unique path.

Usage example:

with context.temporary(suffix='.o') as p:
    ...  # an empty file with absolute path *p* exists

... = context.temporary().path  # just get the absolute path, do not create the file

The path attribute of the returned object is an absolute path in the same directory for all calls in the root context, as a dlb.fs.Path object. Its last component is unique among all calls in the root context. path.is_dir() is is_dir.

The unique path component starts with a lower-case letter and ends with suffix. It contains only lower-case letters and decimal digits between its first characters and the suffix. If suffix is not empty, is must start with a character from strings.punctuation and must not contain '/'. The the unique path component without the suffix is at most 12 characters long for the first 2**61 calls.

When used as a context manager, an empty regular file or directory with path is created when entered and removed (with its content) on exit. Raises FileExistError if the regular file or directory exists.

Same on class and instance.

Parameters:	suffix (str) – suffix of the unique path component
Raises:	ValueError – if suffix is invalid FileExistsError – if the regular file or directory exists (when used as content manager) NotRunningError – if dlb is not running).

working_tree_path_of(path, *, is_dir=None, existing=False, collapsable=False,

allow_nontemporary_management=False, allow_temporary=False)

Return the managed tree path of the path of a filesystem object in the managed tree.

For path to be considered as the path of a filesystem object in the managed tree, path must either be a relative path or it must have root_path as a prefix.

The arguments existing and collapsable describe the assumptions on the filesystem content that may be used to increase the speed and reduce the number of filesystem accesses.

If existing and collapsable are True and path is relative, the filesystem is never accessed.

If existing is False, is_dir() of the returned path reflects the type of the actual filesystem object. Raises dlb.fs.PathNormalizationError if path does not exist.

If allow_nontemporary_management is True, the resulting path may denote a filesystem object in the management tree except in .dlbroot/t. If allow_temporary is True, the resulting path may denote a filesystem object in .dlbroot/t of the management tree.

Does not raise OSError.

Same on class and instance.

Parameters:	path (`dlb.fs.Path` or anything a `dlb.fs.Path` can be constructed from) – a path of a filesystem object in the managed tree is_dir (NoneType \| bool) – `True` if this is a directory path, `False` if not and `None` for derivation from path existing (bool) – assume that all involved filesystem objects exist? collapsable (bool) – assume that any relative to the working tree root is collapsable? allow_nontemporary_management (bool) – is the path permitted to denote a filesystem object the :term`management tree` except ones in `.dlbroot/t`? allow_temporary (bool) – is the path permitted to denote a filesystem object in `.dlbroot/t` of the management tree?
Returns:	a `dlb.fs.Path` p with `p.is_absolute() == False` and `p.is_normalized() == True`
Return type:	same class as path if path is a `dlb.fs.Path` and `dlb.fs.Path` otherwise
Raises:	dlb.fs.PathNormalizationError – if path does not exist in the parts of the working tree that are described by allow_nontemporary_management and allow_temporary ValueError – if the resulting path is not representable NotRunningError – if dlb is not running).

env¶

The environment variable dictionary object with this context as its associated context.

When called on class, it refers to the active context.

Raises:	NotRunningError – if dlb is not running).

helper¶

The dynamic helper dictionary object with this context as its associated context.

The dynamic helper dictionary object maps dynamic helpers to absolute paths, either explicitly or implicitly with the help of find_path_in().

If the active context and the root context both have find_helpers = False and no paths was explicitly assigned to the dynamic helper p in the active context or one of its outer contexts, a look-up with dlb.ex.Context.helper[p] performs a search with dlb.ex.Context.find_path_in(p). (Each such search is performed only once for a given path; the result is stored.)

Examples:

>>> dlb.ex.Context.helper['gcc']
Path('/usr/bin/gcc')

>>> dlb.ex.Context.helper['gcc'] = '/usr/local/bin/my-very-own-very-special-gcc'  # set the path explicitly
>>> dlb.ex.Context.helper['gcc']
Path('/usr/local/bin/my-very-own-very-special-gcc')

>>> dlb.ex.Context.helper['tmp/'] = 'out/t/'  # relative path: relative to the working tree's root path
>>> dlb.ex.Context.helper['tmp/']
Path('/home/schmutzli/projects/esel/out/t')   # with '/home/schmutzli/projects/esel' as the working tree's root

When called on class, it refers to the active context.

Raises:	NotRunningError – if dlb is not running).

Environment variable dictionary objects¶

The environment variable dictionary object env returned by c.env for a context c is a dictionary-like object of all environment variables defined in this c. c is called the associated context of env.

In addition, the environment variable dictionary object manages the import of environment variables from environment variables of the outer context and restriction of imported or assigned values in the form of regular expressions.

The environment variables of the outer context of the root context is defined by os.environ.

Example:

# os.environ usually contains the environment variables in the shell that called the Python interpreter

with dlb.ex.Context():  # takes a snapshot of os.environ

    # import the environment variable 'LANG' into the context
    dlb.ex.Context.active.env.import_from_outer(
        'LANG', restriction=r'[a-z]{2}_[A-Z]{2}', example='sv_SE')

    # now the environment variable is either undefined or matches the regular expression given
    # (in this context and all future inner contexts)

    ... = dlb.ex.Context.active.env['LANG']
        # value in snapshot of os.environ complying to the restriction or KeyError

    dlb.ex.Context.active.env['LANG'] = 'de_AT'

    with dlb.ex.Context():

        # further restrict the value and make sure it is defined
        dlb.ex.Context.active.env.import_from_outer(
            'LANG', restriction='(?P<language>de).*', example='de_CH')

        ... = dlb.ex.Context.active.env['LANG']  # 'de_AT'
        del dlb.ex.Context.active.env['LANG']

        dlb.ex.Context.active.env['LANG'] = 'de_CH'
        # dlb.ex.Context.active.env['LANG'] = 'fr_FR'  # would raise ValueError

    ... = dlb.ex.Context.active.env['LANG']  # 'de_AT'

    del dlb.ex.Context.active.env['LANG']  # undefine 'LANG'
    dlb.ex.Context.active.env['LANG'] = 'fr_FR'  # ok

Environment variable dictionary object support the following methods and attributes:

EnvVarDict.import_from_outer(name, restriction, value_if_undefined=None, example=None)¶

Sets the value of the environment variable named name from the innermost outer context that defines it. If no outer context defines it, the environment variable remains undefined.

Also sets the importing restriction for the value of the environment variable; when it is or later becomes defined, it regular expression restriction must match its value.

The possible imported value and the importing restriction apply to the context and all its future inner contexts.

When called for a root contest, the environment variables are imported from os.environ at the time is was entered.

Parameters:	name (str) – (non-empty) name of the environment variable restriction (str \| `typing.Pattern`) – regular expression example (str) – typical value of a environment variable, restriction must match this
Raises:	ValueError – if an environment variable named name is defined in the associated or an outer context and restriction does not match its value NonActiveContextAccessError – if the associated context is not an active context

EnvVarDict.is_imported(name)¶

Returns True if name is the name of an environment variable imported in the associated context or any of its outer contexts, else False.

Parameters:	name (str) – non-empty name of an environment variable
Raises:	TypeError – if name is not a string ValueError – if name is an empty string

EnvVarDict.get(name, default=None)¶

Return its value if name is the name of a defined environment variable in the associated context, else default.

Parameters:	name (str) – non-empty name of an environment variable
Raises:	TypeError – if name is not a string ValueError – if name is an empty string

EnvVarDict.items()¶: Returns a new view of the dictionary’s items (name, value) pairs of all defined environment variables.

name in env: Returns True if there is a environment variable named name defined in env, else False.

name not in env: Equivalent to not name in env

env[name] = value

Defines an imported environment variable named name with value value in the associated context and all its future inner contexts.

Raises KeyError, if name was not imported in the associated context or one of its outer contexts.

Raises ValueError, if name was imported in the associated context or one of its outer contexts, but is invalid with respect to the restriction an importing context (can be this context and any outer context).

Raises NonActiveContextAccessError, if the associated context is not an active context.

del env[name]

Undefines a defined environment variable named name in the associated context and all its future inner contexts.

Raises KeyError, if name is not defined in the context.

Raises NonActiveContextAccessError, if the associated context is not an active context.

Tool objects¶

Every tool is represented by a subclass of Tool that describes its abstract behaviour and the way it is run (e.g. meaning of command line and output, interaction with file system and environment variables).

Tools are usually parametrized by dependency roles (e.g. input files) and execution parameters.

Each tool instance represents a concrete behaviour and can be run in an active context. Running a tool results in an awaitable result object.

Tool instances are immutable and hashable and fast to construct; the heavy lifting takes place while the tool instance is running.

Tools are customized by inheritance and defining class attributes.

class dlb.ex.Tool¶

A tool declares its dependency roles (e.g. map_file_dependency) and execution parameters (e.g. DO_INCLUDE_DEBUG_INFO, PAPER_FORMAT) as class attributes.

Every tool instance assigns concrete dependencies for the tool’s dependency roles (e.g. a filesystem path './out/hello.map' for a dependency role map_file_dependency), while the execution parameters are the same of all instances of the some tool.

Dependency roles are instances of subclasses of Tool.Dependency.

A new tool can be defined by inheriting from one or more other tools. When overriding a dependency roles, its overriding value must be of the same type as the overridden value and it must be at least as restrictive (e.g. if required dependency must not be overridden by a non-required one). When overriding an execution parameters, its overriding value must be of the same type as the overridden value.

Each subclass of Tool must be defined in a source code location unique among all subclasses of Tool. The definition raises DefinitionAmbiguityError, if its location is cannot be determined or if another subclass of Tool was defined before at the same location.

Example:

class Compiler(dlb.ex.Tool):
   WARNINGS = ('all',)
   source_file = dlb.ex.Tool.Input.RegularFile()
   object_file = dlb.ex.Tool.Output.RegularFile()

class Linker(dlb.ex.Tool):
   object_files = dlb.ex.Tool.Input.RegularFile[1:]()
   linked_file = dlb.ex.Tool.Output.RegularFile()
   map_file = dlb.ex.Tool.Output.RegularFile(required=False)

compiler = Compiler(source_file='main.cpp', object_file='main.cpp.o')
linker = Linker(object_files=[compiler.object_file], linked_file='main')

At construction of a tool, the dependencies given as keyword arguments to the constructor are validated by the tool’s dependency roles and made accessible (for reading only) as an attribute with the name of the corresponding dependency role and a type determined by the dependency role (e.g. dlb.fs.Path for Tool.Input.RegularFile):

>>> Compiler.object_file  # dependency role
<dlb.ex.Tool.Input.RegularFile object at ...>

>>> compiler.object_file  # dependency
Path('main.cpp.o')

run(force_redo=False)¶

Run the tool instance in the active context and returns a result (proxy) object result.

bool(result) is True if a redo is performed and False otherwise.

A redo is performed if force_redo is True or if it is necessary.

If a redo is performed, this method returns before the (asynchronous) redo is complete. After each of the following actions the redo is guaranteed to be complete (either successfully or by raising an exception):

read of a “public” attribute of the result proxy object
exit of the context run() was called in
enter of an inner context of the context run() was called in
modification of env or helper of the context run() was called in
call of run() of the same tool instance

The result object contains an attribute for every dependency role of the tool which contains the concrete dependencies.

If bool(result) is True, all attributes for dependencies have an assigned value. If bool(result) is False, only the attributes for explicit dependencies have an assigned value; the value of all attributes for non-explicit dependencies is NotImplemented.

redo(result, context)¶

Overwrite this method to implement a new Tool.

result is the result object that will by returned by the calling run(). context is the redo context (see Tool.RedoContext).

Use context.execute_helper() and context.replace_output().

Assign to attributes of result to define a non-explicit concrete dependency for the dependency role with the same name.

For a redo to be successful, this method must perform the following tasks:

Create all explicit output dependencies
Assign values to each required non-explicit dependencies

For a filesystem object whose path p is contained in an output dependency, it is recommended to first write to a temporary filesystem object q and then replace it with context.replace_output(p, q). This guarantees that no incomplete output dependency is left behind (like an only half-written object file) when the redo is aborted.

A filesystem object that is an output dependencies is treated as modified be the redo if it is a non-explicit dependency or if it is a explicit dependency that was replaced with context.replace_output().

Raises RuntimeError on the attempt to enter a new dlb.ex.Context as a context manager or to modify the active context.

Return True if the next run this tool instance should perform a redo, regardless of the necessity according to its dependencies.

Example:

class ATool(dlb.ex.Tool):
   EXECUTABLE = 'atool'

   source_file = dlb.ex.Tool.Input.RegularFile()
   output_file = dlb.ex.Tool.Output.RegularFile()
   included_files = dlb.ex.Tool.Input.RegularFile[:](explicit=False)

   async def redo(self, result, context):
       if ...:
           raise ValueException('invalid ...')
       with context.temporary() as temp_file_
          await context.execute_helper(self.EXECUTABLE, ['-o', temp_file, result.source_file])
          result.included_files = ...
          context.replace_output(result.output_file, temp_file)

definition_location¶

The definition location of the class.

It is a tuple of the form (file_path, in_archive_path, lineno) and uniquely identifies the tool among all subclasses of Tool.

in_archive_path is None, if the class was defined in an existing Python source file, and file_path is the os.path.realpath() of this file.

in_archive_path is the path relative of the source file in the zip archive, if the class was defined in an existing zip archive with a filename ending in .zip (loaded by zipimport) and file_path is the os.path.realpath() of this zip archive.

lineno is the 1-based line number in the source file.

fingerprint¶

The permanent local tool instance fingerprint of this instance.

This is a bytes object of fixed size, calculated from all its concrete dependencies d with d.explicit = True.

If two instances of the same subclass of Tool have “similar” explicit dependencies, their fingerprints are equal. If two instances of the same subclass of Tool have explicit dependencies that are not “similar”, their fingerprints are different with very high probability.

The explicit dependencies of two instances are considered “similar”, if they are equal or differ in a way that does not affect the meaning of the dependencies while the tool instance is running.

Redo context¶

A redo context is a read-only view for a dlb.ex.Context with some additional methods related to dynamic helpers and dependencies.

class Tool.RedoContext¶

A redo context is constructed automatically by Tool.run().

execute_helper(helper_file, arguments=(), *, cwd=None, expected_returncodes=frozenset([0]),

forced_env={}, stdin=None, stdout=None, stderr=None, limit=2**16)

Execute the helper_file with command-line arguments arguments in a subprocess with cwd as its working directory and wait for it to complete. The execution is considered successful if an only if its returncode is one in expected_returncodes.

If cwd is not None, is must be the path of directory in the managed tree or in .dlbroot/t/ of the management tree. Otherwise the working tree’s root is used as the working directory.

All members of arguments are converted to str objects.

If a member of arguments is a dlb.fs.Path object p with p.is_absolute() = True, is is replaced by str(p.native). If a member of arguments is a dlb.fs.Path object p with p.is_absolute() = False, is is replaced by str(q.native), where q is p expressed relative to the working directory. Is must denote a filesystem object in the managed tree or in .dlbroot/t/ of the management tree.

env of this object, modified by forced_env, forms the environment for the subprocess.

Parameters:

helper_file – dynamic helper to be executed as a relative path
arguments (iterable of objects that can be converted to str) – command-line arguments
cwd (None or a dlb.fs.Path or anything a dlb.fs.Path can be constructed from) – working directory of the subprocess to be started
expected_returncodes (collection of integers) – expected return codes of the dynamic helper helper_file
forced_env (None | Dict[str, str]) – dictionary of values to override in env or None
stdin – If not None: either a file-like object representing a pipe to be connected to the subprocess’s standard input stream using asyncio.loop.connect_read_pipe(), or the subprocess.PIPE constant.
stdout – If not None: either a file-like object representing the pipe to be connected to the subprocess’s standard output stream using asyncio.loop.connect_read_pipe(), or the subprocess.PIPE constant.
stderr – If not None: either a file-like object representing the pipe to be connected to the subprocess’s standard error stream using asyncio.loop.connect_read_pipe(), or one of subprocess.PIPE or subprocess.STDOUT constants.
limit – the buffer limit for StreamReader wrappers for Process.stdout and Process.stderr (if subprocess.PIPE is passed to stdout and stderr arguments).

Raises:

HelperExecutionError – if the subprocess exits with a returncode not in expected_returncodes.

Returns the tuple (returncode, stdout_data, stderr_data). returncode is the returncode (contained in expected_returncodes). stdout_data and stderr_data are bytes object with the received data from stdout and stderr, respectively.

replace_output(path, source):

Replace the — existing or non-existent — filesystem object path by source. path must be contained in a dependency of the tool instance.

path and source must be different filesystem objects.

After successful completion, path exists and source does not exist. If the parent directory of path does not exist, it is created (with all its parent directories).

The actual operation depends on the corresponding dependency role. If is it a dlb.ex.Tool.Output.RegularFile with replace_by_same_content = False and path and source both exist with the same content, path is no replaced and treated as unchanged.

If path is replaced, this is always done by an atomic operation. If it fails, path is either source afterwards or it does not exist.

Parameters:	path (`dlb.fs.Path` or anything a `dlb.fs.Path` can be constructed from) – a path of a future filesystem object in the managed tree source (`dlb.fs.Path` or anything a `dlb.fs.Path` can be constructed from) – a path of a filesystem object in the managed tree
Raises:	ValueError – if path is not a managed tree path contained in an explicit output dependency or source is not a working tree path of a filesystem object in the managed tree or in `.dlbroot/t/` of the management tree that is different from path.

Dependency classes¶

A dependency class is a subclass of Tool.Dependency. Its instances describe dependency roles (as attributes of a Tool).

The Tool.Dependency.validate() methods of dependency classes are used by tool instances to create concrete dependencies from their constructor arguments.

Each dependency role has an multiplicity specification:

An instance d of a dependency class D created with D(...) has a multiplicity of None which means that its concrete dependency must be a single object (its type depends on D only) or None.
An instance d of a dependency class D created with D[m](...) has a multiplicity of m which means that its concrete dependencies are a sequence of objects (their type depends on D only) or None. The accepted number of members is specified by m.

m can be any non-negative integer or any meaningful proper_slice (of non-negative integers). A number of members is accepted if and only if is either equal to m or contained in range(n + 1)[m].

Example:

class Tool(dlb.ex.Tool):
    # these are dependency roles of the tool 'Tool':
    include_search_paths = dlb.ex.Tool.Input.Directory[1:]()  # a sequence of at least one dlb.ex.Tool.Input.Directory
    cache_dir_path = dlb.ex.Tool.Input.Directory()  # a single dlb.ex.Tool.Input.Directory

tool = Tool(include_search_paths=['build/out/Generated/', 'src/Implementation/'])

# these are concrete dependencies of the tool instance 'tool':
tool.include_search_paths  # (Path('build/out/Generated/'), Path('src/Implementation/'))
tool.cache_dir_path  # (Path('build/out/Generated/'), Path('src/Implementation/'))

Dependency classes are organized in an a hierarchy according to their meaning to a tool by the means of the following abstract classes:

class Tool.Input¶

A Tool.Dependency that describes an input dependency of a tool.

The tool instance must be redone if it (e.g. the mtime of a file) has changed compared to the state before the last successful redo of the tool instance.

An redo must not modify it, successful or not (the same object can be an output dependency of the same tool instance though which can be modified).

class Tool.Output¶

A Tool.Dependency that describes an output dependency of a tool.

A successful redo must generate it (e.g. create a regular file).

These are all abstract classes and contain inner classes derived from them. Example: Tool.Output.Directory is a non-abstract dependency class derived from Tool.Output.

Note

dlb identifies filesystem objects by their managed tree path. It assumes that different managed tree paths point to different filesystem objects.

If a filesystem object serves as an output dependency of one tool instance and as an input dependency of another: Make sure both dependencies use the same path. A redo miss could happen otherwise.

You are always safe without hard links, symbolic links and case-insensitive filesystems.

Concrete dependency role classes support the following methods and attributes:

class Tool.Dependency(required=True, explicit=True)¶

If required is True, a concrete dependency of this dependency role will never be None.

If explicit is True, the concrete dependency can and must be fully defined when the tool instance is created. Otherwise, it cannot and must not be, but automatically assigned by Tool.run().

Each supported constructor argument is available as a property of the same name.

Raises:	DependencyError – if the arguments of the constructor do not match the declared dependency roles of the class

class Value¶

A (potentially abstract) class such that isinstance(v, Value) is True for each validated single value v of each instance t of this class.

This is the type of t.validate() if multiplicity is None and the type of each member of t.validate() otherwise.

validate(value)¶

Parameters:	value (Any type the concrete dependency can convert to T) – The concrete dependency to convert and validate except `None`
Returns:	The validated value of type T
Raises:	TypeError – If `multiplicity` is not `None` and value is not iterable or is a string

compatible_and_no_less_restrictive(other)¶

Is this dependency role an instance of the same class as other with a multiplicity and properties no less restrictive than the ones of other?

Parameters:	other (Tool.Dependency) – reference dependency role
Return type:	bool

multiplicity¶

The multiplicity of the dependency role.

Is None or a dlb.ex.mult.MultiplicityRange.

tuple_from_value(value)¶

Return value if multiplicity is None and a tuple of its members otherwise.

Example:

# returns a tuple of t.Value objects or raises an exception:
>>> v = t.tuple_from_value(t.validate(...))

Input dependency role classes¶

Dependency role class	Keyword arguments of constructor
Dependency role class	Name	Default value
`Tool.Input.RegularFile`	cls	`dlb.fs.Path`
`Tool.Input.NonRegularFile`	cls	`dlb.fs.Path`
`Tool.Input.Directory`	cls	`dlb.fs.Path`
`Tool.Input.EnvVar`	name
	restriction
	example

In addition to the keyword arguments of the specific constructors described here, all constructors also accept the keyword arguments of the constructor of Tool.Dependency.

class Tool.Input.RegularFile(cls=dlb.fs.Path)¶

Constructs a dependency role for a regular files, identified by their paths.

If a path is relative, is it treated as relative to dlb.ex.Context.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

Files outside the managed tree are assumed to remain unchanged between runs of dlb.

The validated value of a concrete dependency is the file’s path as an instance of cls if multiplicity is None and a tuple of the file’s paths otherwise.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    source_files = dlb.ex.Tool.Input.RegularFile[1:](cls=dlb.fs.NoSpacePath)
>>> tool = Tool(source_files=['src/main.cpp'])
>>> tool.source_files
(NoSpacePath('src/main.cpp'),)

Parameters:	cls (dlb.fs.Path) – class to be used to represent the path

class Value¶: Is dlb.fs.Path.

class Tool.Input.NonRegularFile(cls=dlb.fs.Path)¶

Constructs a dependency role for filesystem objects that are neither directories nor regular files, identified by their paths.

If a path is relative, is it treated as relative to dlb.ex.Context.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

Files outside the managed tree are assumed to remain unchanged between runs of dlb.

The validated value of a concrete dependency is the file’s path as an instance of cls if multiplicity is None and a tuple of the file’s paths otherwise.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    symlinks = dlb.ex.Tool.Input.NonRegularFile[:](cls=dlb.fs.NoSpacePath)
>>> tool = Tool(symlinks=['src/current'])
>>> tool.symlinks
(NoSpacePath('src/current'),)

Parameters:	cls (dlb.fs.Path) – class to be used to represent the path

class Value¶: Is dlb.fs.Path.

class Tool.Input.Directory(cls=dlb.fs.Path)¶

Constructs a dependency role for directories, identified by their paths.

If a path is relative, is it treated as relative to dlb.ex.Context.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

Directories outside the managed tree are assumed to remain unchanged between runs of dlb.

The validated value of a concrete dependency is the directory’s path as an instance of cls if multiplicity is None and a tuple of the directory’s paths otherwise.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    cache_directory = dlb.ex.Tool.Input.Directory(required=False)
>>> tool = Tool(cache_directory='/tmp/')
>>> tool.cache_directory
Path('tmp/')

Parameters:	cls (dlb.fs.Path) – class to be used to represent the path

class Value¶: Is dlb.fs.Path.

class Tool.Input.EnvVar(name, restriction, example)¶

Constructs a dependency role for a environment variable named name. It must not have a multiplicity (other than None).

If explicit is False, the value assign in the constructor of the tool instance is used for all future runs of the tool instance. Otherwise, the current value of the active context is used each time Tool.run() is called.

The value of the environment variable is valid if it a string that matches the regular expression restriction, or if it is None and required is False.

The validated value of a concrete dependency is a Value instance with the environment variable’s name and value.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    language = dlb.ex.Tool.Input.EnvVar(
>>>                   name='LANG',
>>>                   restriction=r'(?P<language>[a-z]{2})_(?P<territory>[A-Z]{2})',
>>>                   example='sv_SE')
>>>     flags = dlb.ex.Tool.Input.EnvVar(name='CFLAGS', restriction=r'.+', example='-Wall')
>>> tool = Tool(language='de_CH')  # use 'de_CH' as value of the environment variable for all
>>> tool.language.value['territory']
'CH'
>>> tool.flags
NotImplemented
>>> tool.run().flags.value  # assuming dlb.ex.Context.env['CFLAGS'] of '-O2'
'-O2'

Parameters:	restriction (str \| `typing.Pattern`) – regular expression example (str) – typical value of a environment variable, restriction must match this

class Value¶

A dataclasses.dataclass object with the following attributes:

name¶: The name of the environment variable, as in the corresponding concrete dependency.

raw¶: The value of the environment variable.

groups¶: The named groups of restriction of the corresponding concrete dependency when matched against raw.

Concrete output dependency role classes¶

Dependency role class	Keyword arguments of constructor
Dependency role class	Name	Default value
`Tool.Output.RegularFile`	cls replace_by_same_content	`dlb.fs.Path` `True`
`Tool.Output.NonRegularFile`	cls	`dlb.fs.Path`
`Tool.Output.Directory`	cls	`dlb.fs.Path`
`Tool.Output.Object`

In addition to the keyword arguments of the specific constructors described here, all constructors also accept the keyword arguments of the constructor of Tool.Dependency.

class Tool.Output.RegularFile(cls=dlb.fs.Path, replace_by_same_content=True)¶

Constructs a dependency role for regular files in the managed tree, identified by their paths.

If a path is relative, is it treated as relative to dlb.ex.Context.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

The validated value of a concrete dependency is the file’s path as an instance of cls if multiplicity is None and a tuple of the file’s paths otherwise.

If replace_by_same_content is False for a dependency role containing p, context.replace_output(p, q) in redo(..., context) does not replace p if p and q both exist as accessible regular files and have the same content.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    object_file = dlb.ex.Tool.Output.RegularFile(cls=dlb.fs.NoSpacePath)
>>> tool = Tool(object_file=['main.cpp.o'])
>>> tool.object_file
(NoSpacePath('main.cpp.o'),)

Parameters:	cls (dlb.fs.Path) – class to be used to represent the path

class Value¶: Is dlb.fs.Path.

class Tool.Output.NonRegularFile(cls=dlb.fs.Path)¶

Constructs a dependency role for filesystem objects in the managed tree that are neither directories nor regular files, identified by their paths.

If a path is relative, is it treated as relative to dlb.ex.Context.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

The validated value of a concrete dependency is the file’s path as an instance of cls if multiplicity is None and a tuple of the file’s paths otherwise.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    symlinks = dlb.ex.Tool.Output.NonRegularFile[:](cls=dlb.fs.NoSpacePath)
>>> tool = Tool(symlinks=['dist'])
>>> tool.symlinks
(NoSpacePath('src/current'),)

Parameters:	cls (dlb.fs.Path) – class to be used to represent the path

class Value¶: Is dlb.fs.Path.

class Tool.Output.Directory(cls=dlb.fs.Path)¶

Constructs a dependency role for directories in the managed tree, identified by their paths.

If a path is relative, is it treated as relative to dlb.ex.Context.root_path, and it must be collapsable and non-upwards (if the path does not contain .. components, these requirements are met).

The validated value of a concrete dependency is the directory’s path as an instance of cls if multiplicity is None and a tuple of the directory’s paths otherwise.

Example:

>>> class Tool(dlb.ex.Tool):
>>>    html_root_directory = dlb.ex.Tool.Output.Directory(required=False)
>>> tool = Tool(html_root_directory='html/')
>>> tool.html_root_directory
Path('html/')

Parameters:	cls (dlb.fs.Path) – class to be used to represent the path

class Value¶: Is dlb.fs.Path.

class Tool.Output.Object¶

Constructs a dependency role for any Python object other than None and NotImplemented. It must not be explicit.

The validated value of a concrete dependency is a deep copy of the value.

class Value¶: Is typing.Any.

Exceptions¶

exception dlb.ex.NotRunningError¶: Raised, when an action requires an active context while dlb was not running.

exception dlb.ex.NoWorkingTreeError¶: Raised, when the working directory of the calling process is not a working tree’s root.

exception dlb.ex.ManagementTreeError¶: Raised, when an attempt to prepare or access the management tree failed.

exception dlb.ex.ContextNestingError¶: Raised, when some contexts were not properly nested. I.e. the calls of __exit__() did not occur in the opposite order of the corresponding calls of __enter__().

exception dlb.ex.WorkingTreeTimeError¶: Raised, when the working tree time behaved unexpectedly.

exception dlb.ex.ContextModificationError¶: Raised, when an environment variable dictionary object or a helper dictionary object is modified while its associated context is not the active context.

exception dlb.ex.WorkingTreePathError¶: Raised, when a path is not a working tree path with certain properties where it should be.

exception dlb.ex.DefinitionAmbiguityError¶: Raised at the definition of a subclass of Tool, when the location is unknown or another subclass of Tool was defined before at the same location.

exception dlb.ex.DependencyError¶: Raised when a running tool instance detects a problem with its dependencies before a redo.

exception dlb.ex.ExecutionParameterError¶: Raised when a running tool instance detects a problem with its execution parameters before a redo.

exception dlb.ex.RedoError¶: Raised when a running tool instance detects a problem with its dependencies during or after a redo.

exception dlb.ex.HelperExecutionError¶: Raised, when the execution of a dynamic helper file failed.

`dlb_contrib` – Contributed tools and utilities¶

dlb (the package) is completely tool agnostic. Everything specific to a certain tool, language or file format is not part of dlb.

The package dlb_contrib contains such tools as well as utilities to generate input understood by the tools and parsers for the output of the tools.

Overview¶

Module	executables (dynamic helpers)	tool classes
`dlb_contrib.generic`		`Check`
`dlb_contrib.backslashescape`
`dlb_contrib.zip`		`ZipDirectory`
`dlb_contrib.make`
`dlb_contrib.git`	`'git'`	`GitDescribeWorkingDirectory`
`dlb_contrib.clike`		`ClikeCompiler` `GenerateHeaderFile`
`dlb_contrib.gcc`	`'gcc'` `'g++'`	`CCompilerGcc` `CplusplusCompilerGcc` `CLinkerGcc` `CplusplusLinkerGcc`
`dlb_contrib.pkgconfig`	`'pkg-config'`	`PkgConfig`
`dlb_contrib.doxygen`	`'doxygen'`	`Doxygen`
`dlb_contrib.sh`	`'sh'`	`ShScriptlet`
`dlb_contrib.strace`	`'strace'`	`RunStraced`
`dlb_contrib.tex`	`'tex'` `'latex'`	`Tex` `Latex`

Details¶

The modules in dlb_contrib are meant as starting points for your own tools and contributions which will reside in your repositories. The documentation is maintained only in the source files to make the modules self-contained. Please see there for details.

module dlb_contrib.generic[source]¶

Add input dependencies to any tool instance.

License:	LGPL-3.0-or-later

Provided tools:

Check

Usage example:

import dlb.ex
import dlb_contrib.generic

...  # define ATool

with dlb.ex.Context():
    check = dlb_contrib.generic.Check(input_files=['logo.png'])
    ATool(...).run(force_redo=check.run())  # performs a redo of ATool(...) when 'logo.png' has changed

module dlb_contrib.backslashescape[source]¶

Backslash escape sequences (Python, Bash, Git, …).

License:	LGPL-3.0-or-later

Provided objects:

Usage example:

import dlb_contrib.backslashescape

s = b'"tmp/x\\076y"'
... = dlb_contrib.backslashescape.unquote(s)  # b'tmp/x\076y'

module dlb_contrib.zip[source]¶

Create Zip archives from directory content.

.ZIP file format:
	https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
License:	LGPL-3.0-or-later

Provided tools:

ZipDirectory

Usage example:

import dlb.ex
import dlb_contrib.zip

with dlb.ex.Context():
    dlb_contrib.zip.ZipDirectory(content_directory='build/out/html/',
                                 archive_file='build/out/application.html.zip').run()

module dlb_contrib.make[source]¶

Parsing of Make rules.

Make:	https://pubs.opengroup.org/onlinepubs/009695399/utilities/make.html
GNU Make:	https://www.gnu.org/software/make/
Tested with:	GNU Make 4.2.1
License:	LGPL-3.0-or-later

Provided objects:

Usage example:

import sys
import dlb.fs
import dlb_contrib.make

makefile = dlb.fs.Path(...)

sources = set()
with open(makefile.native, 'r', encoding=sys.getfilesystemencoding()) as f:
    for r in dlb_contrib.make.sources_from_rules(f):
        sources |= set(r)

module dlb_contrib.git[source]¶

Access a working directory managed by Git - the stupid content tracker.

Git:	https://git-scm.com/
Tested with:	git 2.20.1
Executable:	`'git'`
License:	LGPL-3.0-or-later

Provided tools:

GitDescribeWorkingDirectory

Other objects:

Usage example:

import dlb.di
import dlb.ex
import dlb_contrib.git

with dlb.ex.Context():
    result = dlb_contrib.git.GitDescribeWorkingDirectory().run()

    ... = result.tag  # e.g. 'v1.2.3'
    ... = result.branch_refname  # 'refs/heads/master'

    if result.untracked_files:
        s = ','.join(repr(p.as_string()) for p in result.untracked_files)
        dlb.di.inform(f'repository contains {len(result.untracked_files)} untracked file(s): {s}',
                      level=dlb.di.WARNING)

module dlb_contrib.clike[source]¶

The C language family and its tools.

C:	ISO/IEC 9899:1999 (E)
C++:	ISO/IEC 14882:1998 (E)
License:	LGPL-3.0-or-later

Provided tools:

Other objects:

Usage example:

import dlb.ex
import dlb_contrib.clike

name = ...
assert dlb_contrib.clike.SIMPLE_IDENTIFIER.match(name)

... = dlb_contrib.clike.string_literal_from_bytes('Tête-à-tête'.encode())
# '"T\\xC3\\xAAte-\\xC3\\xA0-t\\xC3\\xAAte"'

class GenerateVersionFile(dlb_contrib.clike.GenerateHeaderFile):
    WD_VERSION = ...

    def write_content(self, file):
        wd_version = dlb_contrib.clike.string_literal_from_bytes(self.WD_VERSION.encode())
        file.write(f'\n#define APPLICATION_VERSION {wd_version}\n')

class CCompiler(dlb_contrib.clike.ClikeCompiler):
    EXECUTABLE = 'specific-cc'  # dynamic helper, looked-up in the context

    async def redo(self, result, context):
        with context.temporary() as object_file:
            await context.execute_helper(self.EXECUTABLE, ..., '-o', object_file, result.source_file)
            included_files = ...
            result.included_files = included_files
            context.replace_output(result.object_file, object_file)

with dlb.ex.Context():
    GenerateVersionFile(file='Version.h').run()
    CCompiler(source_file='main.c', object_file='main.c.o').run()

module dlb_contrib.gcc[source]¶

Compile and link languages of the C family with the GNU Compiler Collection.

GCC:	https://gcc.gnu.org/
Tested with:	gcc 8.3.0
Executable:	`'gcc'` `'g++'`
License:	LGPL-3.0-or-later

Provided tools:

Other objects:

Usage example:

import dlb.ex
import dlb_contrib.gcc

with dlb.ex.Context():
    source_path = dlb.fs.Path('src/')
    output_path = dlb.fs.Path('build/out/')

    compile_results = [
        dlb_contrib.gcc.CCompilerGcc(
            source_file=p,
            object_file=output_path / p.with_appended_suffix('.o'),
            include_search_directories=[source_path]
        ).run()
        for p in source_path.list(name_filter=r'.+\.c') if not p.is_dir()
    ]

    dlb_contrib.gcc.CLinkerGcc(
        object_and_archive_files=[r.object_file for r in compile_results],
        linked_file=output_path / 'application').run()

module dlb_contrib.pkgconfig[source]¶

Query paths of installed librares with pkg-config.

pkg-config:	https://www.freedesktop.org/wiki/Software/pkg-config/
Tested with:	pkg-config 0.29
Executable:	`'pkg-config'`
License:	LGPL-3.0-or-later

Provided tools:

PkgConfig

Other objects:

Usage example:

import dlb.ex
import dlb_contrib.pkgconfig

class PkgConfig(dlb_contrib.pkgconfig.PkgConfig):
    LIBRARY_NAMES = ('gtk+-3.0',)
    VERSION_CONSTRAINTS_BY_LIBRARY_NAME = {'gtk+-3.0': ['> 3.0.1', '< 4.0']}

with dlb.ex.Context():
    result = PkgConfig().run()
    ... = result.library_filenames

module dlb_contrib.doxygen[source]¶

Generate documents from source code with Doxygen.

Doxygen:	http://www.doxygen.nl/
Tested with:	Doxygen 1.8.13
Executable:	`'doxygen'`
License:	LGPL-3.0-or-later

Provided tools:

Doxygen

Other objects:

Usage example:

import dlb.ex
import dlb_contrib.doxygen

with dlb.ex.Context():
    dlb_contrib.doxygen.Doxygen(
        configuration_template_file='Doxyfile',  # contains '${{source_directories}}', '${{output_directory}}'
        source_directories=['src/'],  # replaces '${{source_directories}}' in Doxyfile
        output_directory='build/out/').run()

module dlb_contrib.sh[source]¶

Run commands with the Posix sh shell - the standard command language interpreter.

sh:	https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html
Executable:	`'sh'`
License:	LGPL-3.0-or-later

Provided tools:

ShScriptlet

Other objects:

quote

Usage example:

import dlb.ex
import dlb_contrib.sh

class PrintString(dlb_contrib.sh.ShScriptlet):
    SCRIPTLET = "echo echoed: " + dlb_contrib.sh.quote('a $ is a $')

with dlb.ex.Context():
    ... = PrintString().run().output  # 'echoed: a $ is a $\n'

module dlb_contrib.strace[source]¶

Detect read and written filesystem objects of a running process with strace.

strace:	https://strace.io/
Tested with:	strace 4.26
Executable:	`'strace'`
License:	LGPL-3.0-or-later

Provided tools:

RunStraced

Other objects:

Usage example:

from typing import Union, Tuple, List, Iterable
import dlb.ex
import dlb_contrib.strace

class ShowContent(dlb_contrib.strace.RunStraced):
    EXECUTABLE = 'bash'

    def get_command_line(self) -> Iterable[Union[str, dlb.fs.Path, dlb.fs.Path.Native]]:
        return ['-c', '-', 'cat  -- *', 's']

with dlb.ex.Context():
    ... = ShowContent().run().read_files

module dlb_contrib.tex[source]¶

Typeset documents with TeX and LaTeX implementations based on web2c and kpathsea.

TeX:	https://www.tug.org/books/index.html#texbook https://ctan.org/tex-archive/systems/knuth/dist/tex/texbook.tex
LaTeX:	https://www.latex-project.org/
pdflatex:	http://www.tug.org/applications/pdftex/
Tested with:	pdfTeX 3.14159265-2.6-1.40.19 with kpathsea version 6.3.1/dev
Executable:	`'tex'` `'latex'`
License:	LGPL-3.0-or-later

Provided tools:

Other objects:

Usage example:

import dlb.ex
import dlb_contrib.tex

with dlb.ex.Context():
    output_path = dlb.fs.Path('build/out/')

    dlb_contrib.tex.Latex(
        toplevel_file='src/report.tex', output_file=output_path / 'report.dvi',
        input_search_paths=['src/'],
        state_files=[output_path / 'report.aux', output_path / 'report.toc']).run()

Footnotes

[1]	Possible application: monitoring of the build progress on a build server.

[2] The update of mtime for an mmap’d file conforming to ISO 1003.1-2008 after a write to the mapped memory is only guaranteed via msync(). Therefore such a write operation is not considered complete before the next call of msync() (which may never happen). Actual behaviour of mmap on different operating systems (2018): https://apenwarr.ca/log/20181113.

[3]	Especially, mtime is not manually set with touch -t or any tool that uses a coarser time resolution than the effective mtime resolution. See touch and utimensat().

[4]	Linux currently (2020) uses ktime_get_coarse_real_ts64() as time source for its (optional) mtime updates, which which returns the system-wide realtime clock at the last tick.

[5]	(1, 2, 3) Some filesystems support mount options to sacrifice this guaranteed for performance. Example: Ext4 with mount option lazytime.

[6]	`adjtime()` is not covered by ISO 1003.1-2008. It originated in 4.3BSD and System V. For many operating systems it states “the clock is always monotonically increasing” (Linux, OpenBSD, FreeBSD).

[7]	Resolution for this situation: Try later (delay should be part of the message) or correct the mtime of the affected filesystem objects (list should be part of the message).

Table of Contents

Previous topic

Reference¶

Terms¶

Layout of working tree¶

Top-level specification¶

Assumptions¶

Modification of the working tree¶

Filesystems behaviour of working tree¶

Timing and concurrency¶

Dependencies¶

Guarantees¶

Dependencies¶

Timing and concurrency¶

dlb — Command-line utility for convenience¶

`dlb` — Version¶

`dlb.fs` — Filesystem paths¶

Path objects¶

Restricting paths¶

`dlb.di` — Line-oriented hierarchical diagnostic messages¶

Example¶

Syntax¶

Module content¶

`dlb.cf` — Configuration parameters¶

`dlb.ex` — Execution contexts and dependency-aware tool execution¶

Context objects¶

Environment variable dictionary objects¶

Tool objects¶

Redo context¶

Dependency classes¶

Input dependency role classes¶

Concrete output dependency role classes¶

Exceptions¶

`dlb_contrib` – Contributed tools and utilities¶

Overview¶

Details¶

Reference¶

Terms¶

Layout of working tree¶

Top-level specification¶

Assumptions¶

Modification of the working tree¶

Filesystems behaviour of working tree¶

Timing and concurrency¶

Dependencies¶

Guarantees¶

Dependencies¶

Timing and concurrency¶

dlb — Command-line utility for convenience¶

dlb — Version¶

dlb.fs — Filesystem paths¶

Path objects¶

Restricting paths¶

dlb.di — Line-oriented hierarchical diagnostic messages¶

Example¶

Syntax¶

Module content¶

dlb.cf — Configuration parameters¶

dlb.ex — Execution contexts and dependency-aware tool execution¶

Context objects¶

Environment variable dictionary objects¶

Tool objects¶

Redo context¶

Dependency classes¶

Input dependency role classes¶

Concrete output dependency role classes¶

Exceptions¶

dlb_contrib – Contributed tools and utilities¶

Overview¶

Details¶

`dlb` — Version¶

`dlb.fs` — Filesystem paths¶

`dlb.di` — Line-oriented hierarchical diagnostic messages¶

`dlb.cf` — Configuration parameters¶

`dlb.ex` — Execution contexts and dependency-aware tool execution¶

`dlb_contrib` – Contributed tools and utilities¶