Reference¶
Terms¶
- run of dlb
- The execution of a part of a script inside the same root context.
- tool
Describes the abstract behaviour and the way a tool is run by a sub class of
dlb.ex.Tool
.It is concretized by instantiation.
- tool instance
A (concrete) instance of an (abstract) tool with concrete dependencies.
Can be run in an active context with
r = t.run()
(once or multiple times), which happens sequentially in the following stages:- start of execution: call of
t.run()
. - dynamic helper resolution: if this is the first instance of the tool being run in the active context: find the absolute path of every dynamic helper (e.g. executable) that might be needed during a redo of any instance of this tool and memorize it.
- redo check: decide whether a registered dependency required a redo by inspecting its state and memorize it if it does.
- if redo necessary:
- redo: generate all output dependencies.
- dependency finalization: update registered dependencies and their state (“memo”) in the run-database.
- end of execution:
- successful after awaiting on r has been completed or
- unsuccessful after exception has been risen.
Between start and end of execution the tool instance is said to be running.
- start of execution: call of
- dynamic helper
An external filesystem object in the managed tree or outside the working tree with a path to be determined at run-time.
A dynamic helper is identified by a relative path (non-directory or directory).
Typical example: globally installed executable (e.g. compiler) with an absolute path determined by a search for the first match in the environment variable
PATH
.- redo
The phase of the execution of a tool instance t that generates all its output dependencies.
A redo is considered necessary if at least one of the following statements is true:
- Any of the output dependencies of t does not exist in the expected form.
- Any of the input dependencies of t has changed its existence in the expected form since the start of the last known successful execution of t.
- Any of the input dependencies of t has changed a depended-upon property (e.g. filesystem attribute, value of environment variable) since the start of the last known successful execution of t.
- Any of the execution parameters t has changed since the start of the last known successful execution of t.
- redo miss
A redo miss of a tool instance is an undesirable situation that occurs when the redo check considers a redo not necessary while it actually is.
It is caused by bugs of dlb or external helpers and violations of assumptions (e.g. on the behaviour of the filesystem).
- context
An (execution) context describes how running tool instances shall interact with the execution environment outside the working tree and with each other.
It is represented as an instance of
dlb.ex.Context
used as a context manager.- active context
- The innermost context, if any.
- root context
- The outermost context, if any.
- script
- A Python program that potentially creates an active context at least once.
- working tree
- The filesystem tree rooted at a directory (directly) containing a directory
.dlbroot/
(a symbolic link to directory is not considered a directory here). - management tree
The filesystem tree rooted at
.dlbroot/
in the root of the working tree.Do not modify its content manually unless you are told so by dlb.
- managed tree
The filesystem tree rooted at at the root of the working tree without the management tree. Contains the files involved in the build that are to be managed by dlb.
May and (typically will) be manually modified while there is no active context (e.g. by editing source files).
- mtime
- The time of last data modification of a filesystem object in the sense of ISO 1003.1-2008.
- working tree time
- The time according to the mtime of an imaginary filesystem object created at a certain instant (assuming a single filesystem).
- mtime update
- Setting the mtime of a filesystem object to the current working tree time.
- working tree’s system time
- The system time used a source for every mtime update of every filesystem object in the working tree (assuming there is one).
- effective mtime resolution
The effective mtime resolution for a filesystem object p is defined by the following thought experiment:
- p is modified at ideal time t, resulting in a mtime m of p.
- p is modified at ideal time t + dt, resulting in a mtime m + dm of p.
- The effective mtime resolution for p is the minimum dm > 0 for any pair of t and dt > 0.
Resolution of timestamps for some filesystems: XFS: 1 ns, NTFS: 100 ns, ext2: 1 s, FAT32: 2 s. The effective mtime resolution depends also on the filesystem driver and the operating system, but it cannot be finer that the timestamp resolution of the filesystem.
- ideal time
- The (strictly increasing) physical time at the place the dlb process is running.
- non-upwards path
A relative path that has no prefix denoting its parent directory.
Examples:
a/../b
is an non-upwards path,a/../../a/b
is not.- collapsable path
A path p of an existing filesystem object with the following property: No prefix path of p’ that ends with a component other than
..
is the path of a symbolic link, where p’ is p with all.
components removed.Example:
a/b/../c/..
is collapsable if and only if neithera/b/
nora/b/../c
is a symbolic link.- canonical-case path
A path whose components are all exactly as listed in their parent directory.
On a case-insensitive filesystem or directory, multiple paths that differ in case or character encoding can point to the same filesystem object. Only one of them is a canonical-case path.
- normal path
- A path without
..
components. - working tree path
- The normal path of an existing filesystem object relative to the working tree’s root.
- managed tree path
- A working tree path of file system object in the managed tree.
- run-database
The database in the management tree that stores information on the current and past runs of dlb, primarily related to dependencies.
Its removal (permitted when dlb is not running) typically leads to unnecessary redos in the following two runs.
- true input dependency
- A true input dependency of a tool instance t is an input of t that is not known to have been generated by a previous running tool instance (in the current or a previous run of dlb).
- redo-safe
- An action (e.g. a modification of the managed tree) is said to be redo-safe if it cannot not lead to a redo miss for any tool instance in the current run or any future run of dlb.
- benign managed tree modification
A modification of the managed tree is benign, if it consist only of an arbitrary number of the following actions in any order:
- Remove or create a filesystem object (this includes symbolic links and hard links)
- Write to a regular file
Examples of modifications of the managed tree that are no benign managed tree modifications:
- Replace a regular file by any other one with mv (does not update mtime of the target)
- Swap two directories
- Set the mtime of a filesystem object to something different from the current working tree time
Layout of working tree¶
The directory .dlbroot/
is mandatory — it marks its parent directory as the root of a dlb working tree.
Everything else is optional. dlb does never touch filesystem objects outside .dlbroot/
unless
explicitly directed by a script to do so.
The directory .dlbroot/u/
is optional. dlb does never touch its content.
Each regular file or a symbolic links to a regular file in .dlbroot/u/
whose name ends in .zip
should
be a zip archive loadable by zipimport
.
You can place dlb as .dlbroot/u/dlb.zip
in this directory (under version control). This makes the
working tree almost self-contained; only an external Python interpreter is needed.
If you use Git for version control which does not support empty directories, add .dlbroot/o
(as created by dlb)
or any file in .dlbroot/u/
.
The lines marked with * show filesystem objects only given as an example.
Before first run of a dlb script:
.dlbroot/
src/ *
a.c *
a.h *
b.c *
test/ *
...
During a run of a dlb script:
.dlbroot/
o empty regular file, used to probe the "current" mtime
runs-*.sqlite run-database
t/ temporary files
a.o *
b.o *
src/ *
a.c *
a.h *
b.c *
test/ *
out/ *
p *
dist/ *
...
After a successful run of a dlb script:
.dlbroot/
o empty regular file
runs-*.sqlite run-database
src/ *
a.c *
a.h *
b.c *
test/ *
out/ *
a.o *
b.o *
dist/ *
p *
...
Top-level specification¶
Assumptions¶
An assumption (A-…) is a requirement for the intended behaviour of dlb that cannot be checked at runtime in a reliable and efficient way.
For every assumption at set of acceptable effects of its violation is given:
- repair
- The working tree needs manual repair.
- obscure-fail
- A build may fail in an obscure way. The diagnostic messages do neither clearly indicate the problem nor a way to fix it.
- vulnerable
- The build becomes vulnerable to attacks. Running tool instances might overwrite any filesystem object the process has permission to, potentially with information generated during the build or stored in the working tree.
- redo-miss
- A redo miss in the current or a future run of dlb can occur.
- graceful-fail
- A build may fail in a meaningful way. The diagnostic messages clearly indicate the problem or a way to fix it.
Modification of the working tree¶
- A-A1 (access to management tree)
Except for modifications by dlb internals, the management tree is not modified while dlb is running and only as suggested by diagnostic messages of dlb.
Changing the absolute path of the working tree’s root is considered a modification.
Acceptable when violated:
- repair (e.g. after the run-database has become corrupted)
- obscure-fail (e.g. after intermediate files have been modified)
- vulnerable (e.g. by a symlink attack)
- redo-miss (e.g. after the run-database has been is tampered with)
- graceful-fail
- A-A2 (access to managed tree)
While a tool instance in running, the managed tree is modified only by running tool instances.
Changing the absolute path of the working tree’s root is considered a modification.
Acceptable when violated:
- obscure-fail (e.g. when a running tool instance “sees” an intermediate state of files that are input dependencies)
- vulnerable (e.g. by a symlink attack)
- graceful-fail
- A-A3 (manual modification of mtime)
Except from modifications requested by a running tool instance, every modification of the mtime of a filesystem object in the working tree is a mtime update. [3]
Acceptable when violated:
- A-A4
No part of the filesystem outside of the working tree is modified while a tool instance t is running, unless it cannot affect the behaviour of t.
Acceptable when violated:
Filesystems behaviour of working tree¶
- A-F1 (one filesystem)
Every filesystem object
w/p
, wherew
is the path of the working tree’s root andp
is a relative path without..
components, resides on the same (local or remote) file system.Acceptable when violated:
- A-F2 (mtime update at creation)
Every creation of a filesystem object in the working tree updates its mtime. [5]
Acceptable when violated:
- A-F3 (mtime update at write to regular file)
Every write to regular file in the working tree updates its mtime as soon as it is completed. [5] [2]
Between start and completion of a write, a reader of the file may observe an intermediate state of the file’s content.
[--------------] content change ^ ^ start mtime update (write complete) -------------------> ideal time
Acceptable when violated:
- A-F4 (mtime update for directory)
Every creation, removal, renaming and attribute change of a filesystem object in the working tree updates the mtime of the (directly) containing directory. [5]
Acceptable when violated:
- A-F5 (moving is atomic)
Moving a regular file, a directory or a symbolic link in the working tree to a different directory in the working tree is possible in an reasonably secure, efficient and atomic operation that does not affect the moved object’s filesystem attributes (including mtime in full resolution).
Acceptable when violated:
- A-F6 (moving makes invisible)
Immediately after a regular file, a directory or a symbolic link in the working tree has been successfully moved to a different directory within the same working tree, no other process “sees” it in the original directory.
Acceptable when violated:
- A-F7 (no corruption)
A filesystem object in working tree is never corrupted (e.g. by failure of software, memory or power).
Acceptable when violated:
Timing and concurrency¶
- A-T1 (working tree time exists)
The mtime of every filesystem object in the working tree is updated from the same system time (local or remote), the working tree’s system time. Whenever a mtime update occurs for a filesystem object p after one occurs for filesystem object q, where p and q are in the working tree, the mtime of p is not later than the mtime of q. [4]
Acceptable when violated:
- A-T2 (working tree time mostly monotonically increasing)
With the exception of rare backward jumps, the working tree time is monotonically increasing.
The time between consecutive backward jumps is longer than the duration of a run of dlb.
Acceptable when violated:
- redo-miss (the finer the effective mtime resolution and the less frequent modification of inputs files are, the less likely is a redo miss due to this violation)
- graceful-fail (this is desirable since it forces correction before it can cause redo misses in the future)
- A-T3 (effective mtime resolution)
The regular file
o
in the management tree has an effective mtime resolution no coarser than 100 ms.Acceptable when violated:
- A-T4 (working tree time of true input dependencies in the past)
The mtime of every filesystem object in the managed tree that is an true input dependency of a tool instance t is earlier than the time t starts running.
Acceptable when violated:
Dependencies¶
- A-D1 (regular files)
- Most of the filesystem objects in the managed tree that serve as input dependency of tool instances are regular files.
- A-D2 (no links to input dependencies)
A filesystem object in the managed tree that serves as an input dependency of a tool instance has no hard link or symbolic link pointing to it in the managed tree.
Acceptable when violated:
- A-D3 (no implicit symbolic links in paths)
A filesystem object in the managed tree that serves as a dependency of a tool instance t does not have a parent directory
p
in its path that is a symbolic link, unlessp
is an input dependency of t and in the working tree.Acceptable when violated:
Guarantees¶
A guarantee (G-…) is a specification of behaviour observable by the user.
Dependencies¶
- G-D1 (no redo miss when working tree time monotonic)
A benign managed tree modification is redo-safe, provided the assumptions A-F1, A-F2, A-F3, A-F4 hold and the working tree time is monotonically increasing (at least since the oldest mtime of all filesystem objects that are true input dependencies of a tool instance).
This is true even when assumption A-A2 is violated.
- G-D2 (no redo miss when file size changes)
- Modifying the content of a regular file in the managed tree while a tool instance is running (in violation of A-A2) is redo-safe if it also changes the size of the regular file.
- G-D3 (redo miss unlikely when modification intervals relatively long)
A benign managed tree modification is likely to be redo-safe, provided the assumptions A-F1, A-F2, A-F3, A-F4 hold and the “modification intervals are relatively long” for every filesystem object that is a true input dependency of a tool instance.
Here, a modification interval of a filesystem object p is considered to be relatively long if it is unlikely that the working tree time at the ideal time t is the same as at t + T, where T is the ideal time between two consecutive mtime updates of p.
This is true even when assumption A-A2 is violated.
- G-D4
- When assumption A-T2 is violated at a certain time at the start of the “redo check” phase of a running tool instance, the build is aborted with a meaningful diagnostic message. [7]
Timing and concurrency¶
- G-T1 (active context exit)
- An active context is not left as long as a tool instance is running in it.
- G-T2 (root context exit)
- A root context is not left other than by a raised exception before there has been a time window with the following property: The mtime of a regular file o in the management tree would have been different from the mtime of the last filesystem object modified by a running tool instance.
- G-T3 (multiple dlb processes)
- When multiple scripts are run by different processes on the same working tree, at most one of them is in an active context at the same time.
- G-T4 (threads)
- dlb does not create threads.
dlb — Command-line utility for convenience¶
The command-line utility dlb calls a dlb script;
the real work is done entirely by the dlb script with the help of the package dlb
.
dlb can save you some typing, however, with the following features:
- Changes the current working directory to the working tree’s root from anywhere in the working tree.
- Remembers command-line arguments of the last successful call.
- Adds ZIP archives to the module search path.
See dlb --help
for details.
dlb
— Version¶
-
dlb.
__version__
¶ The version of dlb as a non-empty string. Example:
'1.2.3
.Conforms to PEP 396, PEP 386 and PEP 440.
Contains the string
'.dev'
if and only if this a development version. For a development version, __version__ looks like this:'1.2.3.dev30+317f'
.
-
dlb.
version_info
¶ The version of dlb as a non-empty tuple of non-negative integers and strings
'a'
…'z'
, similar tosys.version_info
. Example:(1, 2, 3)
.Contains at least 3 members, the first 3 of them integers. Contains more than 3 members if and only if this is an unreleased version. For an unreleased version, version_info looks like this:
(1, 2, 3, 'c', 4)
.Use it like this to compare (released) versions:
assert (1, 0, 3) <= dlb.version_info < (2,)
Note: For a development version, version_info carries less information than __version__.
dlb.fs
— Filesystem paths¶
This module provides classes to represent and access filesystem objects in a safe and platform-independent manner,
most prominently dlb.fs.Path
and its subclasses.
Path objects¶
-
class
dlb.fs.
Path
¶ A
dlb.fs.Path
represents the path of a filesystem object in a platform-independent manner and expresses whether the object is a directory or not. All operations on instances ofdlb.fs.Path
show the same behaviour on every platform (with the exception ofnative
). Its instances are immutable and hashable (they can be used in sets or as keys in dictionaries).All represented paths are either absolute or relative to the working directory of a process.
The interface is similar to
pathlib
and conversion from and topathlib
paths is supported.If the path represented by a
dlb.fs.Path
is meaningful as a concrete path on the platform the code is running on,native
returns it in the form of adlb.fs.Path.Native
instance, which can then be used to access the filesystem (like apathlib.Path
).On all platform the following properties hold:
'/'
is used as a path component separator.- A path does not contain
'\0'
. - A path is absolute if and only if it starts with
'/'
; it is relative if and only if it is not absolute. - A component
'..'
means the parent directory of the path before it. - A component
'.'
means the directory of the path before it; a non-empty path with all such components removed is equivalent to the original one. - A sequence of two or more consequent
'/'
is equivalent to a single'/'
, except at the beginning of the path. - At the beginning of the path:
- Three or more consequent
'/'
are equivalent to a single'/'
- Exactly two consequent
'/'
(followed be a character other than'/'
) means that the following component is to be interpreted in a implementation-defined manner (e.g. ISO 1003.1-2008 or UNC paths)
- Three or more consequent
- If the platform supports symbolic links, it resolves them as specified in ISO 1003.1-2008
(which means that
'..'
cannot be collapsed without knowing the filesystem’s content).
dlb.fs.Path
instances are comparable with each other by=
,<
etc. They are also comparable with strings andpathlib.PurePath
. Comparison is done case-sensitively and component-wise, observing the equivalence relations described below. A non-directory path is smaller than an otherwise identical directory path. If a directory path d is a prefix of another path p, then d < p. Any relative path is smaller than any absolute path.Usage example:
p = dlb.fs.PortablePath('a/b/c/') / 'x/y' p.relative_to(...) ... = str(p.native) with p.native.open() as f: f.readline()
The
dlb.fs.Path
class supports the following methods and attributes:-
Path
(path[, is_dir=None])¶ Constructs a path from another path or a string.
If path is a string, it is interpreted as the string representation of a path in Posix style with
/
as a component separator. It must not by empty. If path is apathlib.Path
, it must be either absolute or relative. If path is a sequence, all its members are converted to str and are interpreted as path components; its first component must be''
for a relative path.If is_dir is
None
, the ending of path determines whether is considered a directory path or not; it is if it ends with'/'
or a'.'
or'..'
component.If is_dir is
True
, the path is considered a directory path irrespective ofpath
.If is_dir is
False
, the path is considered a non-directory path irrespective ofpath
However, if path represents'.'
or endwith a'..'
component, aValueError
exception is raised.Parameters: - path (str |
Path
|pathlib.PurePath
) – portable string representation or path object - is_dir (NoneType | bool) –
True
if this is a directory path,False
if not andNone
for derivation from path
Raises: - TypeError – if path is neither a string nor a path
- ValueError – if path is an empty string
- ValueError – if path is a
pathlib.PurePath
which is neither absolute nor relative
Examples:
>>> p = Path('a/b/').is_dir() True >>> p = Path(pathlib.PureWindowsPath('C:\\Windows'), is_dir=True) >>> p Path('/C:/Windows/') >>> p.is_dir() True >>> p = Path('x/y/..', is_dir=False) Traceback (most recent call last): ... ValueError: cannot be the path of a non-directory: 'x/y/..' >>> Path('x/y/z.tar.gz')[:-2] Path('x/') >>> Path('x/y/z.tar.gz').parts[-1] 'z.tar.gz'
- path (str |
-
is_normalized
()¶ Returns: True
if and only if this represents a normalized path (i.e. it contains no'..'
components)Return type: bool
-
relative_to(other, collapsable=False):
Returns a version of this path relative to the directory path represented by other.
If collapsable is
False
, this path must be a prefix of other. If collapsable isTrue
, other is treated as a collapsable path and the minimum of necessary..
components is prepended.Return type: class of this object
Raises: - ValueError – if this is a non-directory path
- ValueError – if other is not a prefix of this
-
iterdir
(name_filter='', recurse_name_filter=None, follow_symlinks=True, cls=None)¶ Yields all path objects of the directory contents denoted by this path and matched by the name filters. The paths are duplicate-free and in a defined and repeatable order, but not necessarily sorted. Their class is the class of this object if cls is
None
and cls otherwise.The path of an existing filesystem object is eventually yielded if and only if
- its name matches the name filter name_filter and
- it is contained in a matched directory.
A directory is a matched directory if and only if it is the directory d denoted by this path or a direct subdirectory of a matched directory whose name matches the name filter recurse_name_filter. If follow_symlinks is
True
, a symbolic link to an existing directory is considered a direct subdirectory of the director containing the symbolic link. If follow_symlinks isFalse
or the target of the symbolic link does not exist, it is considered a non-directory.name_filter and recurse_name_filter are name filters. A name filter can be
None
— no name matches this filter- a callable c accepting exactly one argument — a name n matches this filter if and only if
bool(c(n))
isTrue
- a compiled regular expression r — a name n matches this filter if and only if
r.fullmatch(n))
is notNone
- a non-empty regular expression string
s
— a name n matches this filter if and only ifre.compile(s).fullmatch(n))
is notNone
- an empty string — every name matches this filter
Example:
for p in dlb.fs.Path('src/').iterdir(name_filter=r'(?i).+\.cpp', recurse_name_filter=lambda n: '.' not in n): ...
Return type: cls | class of this object
Raises: - TypeError – if cls is neither
None
nor a subclass ofdlb.fs.Path
- TypeError – if name_filter or recurse_name_filter are not both name filters
- ValueError – if this is a non-directory path
-
iterdir_r
(name_filter='', recurse_name_filter=None, follow_symlinks=True, cls=None)¶ Like
iterdir()
, but all returns paths are relative to this path.
-
list
(name_filter='', recurse_name_filter=None, follow_symlinks=True, cls=None)¶ Returns all paths yielded by
iterdir()
as a sorted list.Example:
>>> dlb.fs.NoSpacePath('src/').list(name_filter=r'(?i).+\.cpp') [NoSpacePath('src/Clock.cpp'), NoSpacePath('src/main.cpp')]
-
list_r
(name_filter='', recurse_name_filter=None, follow_symlinks=True, cls=None)¶ Returns all paths yielded by
iterdir_r()
as a sorted list.Example:
>>> dlb.fs.NoSpacePath('src/').list(name_filter=r'(?i).+\.cpp') [NoSpacePath('Clock.cpp'), NoSpacePath('main.cpp')]
-
__getitem__(key):
A subpath (a slice of the path).
The resulting path is absolute (with the same anchor) if and only if the slice starts at 0. The resulting path is a non-directory path if and only if it contains the last component and if this path is a non-directory path.
The slice step must be positive.
Examples:
>>> dlb.fs.Path('src/comp/lib/Core.cpp')[:-2] Path('src/comp/' >>> dlb.fs.Path('src/comp/..')[:-1] Path('src/comp/'
Parameters: Return type: class of this object
Returns: subpath
Raises: - TypeError – if key is not a slice
- ValueError – if this is an absolute path and key is an empty slice
-
parts
¶ A tuple giving access to the path’s various components:
>>> p = Path('/usr/bin/python3') >>> p.parts ('/', 'usr', 'bin', 'python3')
Return type: tuple(str)
-
native
¶ This path as a native path. Use it to access the filesystem:
p = Path('/usr/bin/') with open(p.native) as f: ...
This attribute cannot be written.
Return type: dlb.fs.Path.Native
Raises: ValueError – if this path is not representable as Path.Native
-
pure_posix
¶ This path as a
pathlib.PurePosixPath
:>>> p = Path('/usr/bin/') >>> p.pure_posix PurePosixPath('/usr/bin')
This attribute cannot be written.
Return type: pathlib.PurePosixPath
-
pure_windows
¶ This path as a
pathlib.PureWindowsPath
:>>> p = Path('/C:/Program Files/') >>> p.pure_windows PureWindowsPath('C:/Program Files')
This attribute cannot be written.
Return type: pathlib.PureWindowsPath
-
class
Path.
Native
¶ A native path whose instances can be used much like ones from
pathlib.Path
and is aos.PathLike
.For each subclass P of
dlb.fs.Path
there is a corresponding subclassP.Native
which imposes the same restrictions on its representable paths as P.If Q is a subclass of P and P is a subclass of
dlb.fs.Path
, thenQ.Native
is a subclass ofP.Native
.Example (on a Posix system):
>>> dlb.fs.NoSpacePath.Native('/tmp/x y') Traceback (most recent call last): ... ValueError: invalid path for 'NoSpacePath': '/tmp/x y' (must not contain space)
In contrast to
pathlib.Path
, conversion to string is done in a safe way: relative paths are guaranteed to start with'.'
.Example (on a Posix system):
>>> str(Path.Native('-rf')) './-rf'
Instances of
dlb.fs.Path.Native
and its subclasses should not be constructed directly, but by accessingdlb.fs.Path.native
.Example (on a Posix system):
with open(dlb.fs.NoSpacePath('/tmp/x/a').native) as f: ... = f.read()
-
raw
¶ This path as a
pathlib.Path
. Use it to access the filesystem in an object-oriented manner:p = Path('/usr/bin/') ... = p.native.raw.is_dir()
This attribute cannot be written.
Constructing a
pathlib.Path
is an expensive operation. For performance-critical tasks, usep.native
and functions for string-like paths instead: e.g.os.path.isdir(p.native)
instead ofp.native.raw.is_dir()
.
-
Restricting paths¶
By subclassing dlb.fs.Path
, additional restrictions to the set of value values can be imposed
(trying to construct a dlb.fs.Path
from an invalid value raises an ValueError
exception).
A subclass of dlb.fs.Path
should implement only check_restriction_to_base()
.
-
class
dlb.fs.
RelativePath
¶ A
dlb.fs.Path
which represents a relative path.
-
class
dlb.fs.
AbsolutePath
¶ A
dlb.fs.Path
which represents an absolute path.
-
class
dlb.fs.
NormalizedPath
¶ A
dlb.fs.Path
which represents a normalized path (without'..'
components).
-
class
dlb.fs.
NoSpacePath
¶ A
dlb.fs.Path
whose components do not contain' '
.
-
class
dlb.fs.
PosixPath
¶ A
dlb.fs.Path
which represents a POSIX-compliant (ISO 1003.1-2008) paths in its least-constricted form.Every non-empty string, which does not contain
'/'
or U+0000 (NUL) is a valid component. Components are separated by'/'
.For every path prefix (in the POSIX sense) {NAME_MAX} and {PATH_MAX} are considered unlimited.
Relevant parts of ISO 1003.1-2008:
- section 4.12 Pathname Resolution
- section 4.5 File Hierarchy
- section 4.6 Filenames
- section 4.7 Filename Portability
- section 3.267 Pathname
- section 3.269 Path Prefix
- limits.h
-
class
dlb.fs.
PortablePosixPath
¶ A
dlb.fs.PosixPath
which represents a POSIX-compliant (ISO 1003.1-2008) path in its strictest form. Any path whose support is not required by POSIX or is declared as non-portable is considered invalid.A component cannot be longer than 14 characters, which must all be members of the Portable Filename Character Set.
The length of the string representation of the path is limited to 255 characters.
No absolute path prefix other than
'/'
is allowed (because implementation-defined).
-
class
dlb.fs.
WindowsPath
¶ A
dlb.fs.Path
which represents a Microsoft Windows-compliant file or directory path in its least-constricted form, which is either relative or absolute and is not a reserved non-directory path (e.g.NUL
).It cannot represent incomplete paths which are neither absolute nor relative to the current working directory (e.g.
C:a\b
and\\name
). It cannot represent NTFS stream names, Win32 file namespaces or Win32 device namespaces.
-
class
dlb.fs.
PortableWindowsPath
¶ A
dlb.fs.WindowsPath
which represents a Microsoft Windows-compliant path in its strictest form.A component cannot end with
' '
or'.'
(except'.'
and'..'
) and cannot be longer than 255 characters. The path cannot not be longer than 259 characters.
-
class
dlb.fs.
PortablePath
¶
dlb.di
— Line-oriented hierarchical diagnostic messages¶
In contrast to the logging
module, this module focuses on hierarchical structure and unambiguity.
Absolute time information (date and time of day) is not output in favour of high-resolution relative times.
The output is compact, line-oriented and well readable for humans.
Each message has an associated level, e.g. WARNING
, with the same meaning and numerical value as in the
logging
module. The higher the associated numeric value, the more important the message is considered:
Messages below a global message threshold are not output. The message threshold can be changed any time.
It is initialised to INFO
if sys.flags.verbose
is False
and 1 otherwise.
Messages can be nested with message clusters. In the output, the nesting level of a message is expressed by the indentation of its lines.
An precise syntax in enforced to make the output suitable for incremental parsing (e.g. from a named pipe) with the help of simple regular expressions. [1] Any Unicode character except characters from the range U+0000 to U+001F (ASCII control characters) can be used in messages as long as the syntax is not violated.
To output a message, call dlb.di.inform()
or enter a context manager instance of dlb.di.Cluster()
.
Example¶
import dlb.di
...
with dlb.di.Cluster(f"analyze memory usage\n note: see {logfile.as_string()!r} for details", is_progress=True):
ram, rom, emmc = ...
dlb.di.inform(
f"""
in use:
RAM:\t {ram}\b kB
ROM (NOR flash):\t {rom}\b kB
eMMC:\t {emmc}\b kB
""")
if rom > 0.8 * rom_max:
dlb.di.inform("more than 80 % of ROM used", dlb.di.WARNING)
This will generate the following output:
I analyze memory usage...
| note: see 'out/linker.log' for details
I in use:
| RAM: 12 kB
| ROM (NOR flash): 108 kB
| eMMC: 512 kB
W more than 80 % of ROM used
I done.
Syntax¶
Each message starts with a capital letter after indentation according to its nesting level (2 space characters per
level) and ends with a '\n'
after a non-space character. It can consist of any number of lines: an initial line
followed by any number of continuation lines, separated by '␣\n'
and the same indentation as the initial line
('␣'
means the character U+0020):
message ::=single_line_message
|multi_line_message
single_line_message ::=initial_line
'\n' multi_line_message ::=initial_line
'␣\n' (continuation_line
'␣\n')*continuation_line
'\n' indentation ::= '␣␣'*
The initial line carries the essential information. Its first letter after the indentation denotes the level of the
message: the first letter of the standard names of the standard loglevels of the logging
module.
An optional relative file path and 1-based line number of an affected regular file follows.
initial_line ::=indentation
summary_prefix
summary
summary_suffix
summary_prefix ::=level_indicator
'␣' [file_location
'␣' ] summary_suffix ::= [progress_suffix
] [ '␣'relative_time_suffix
] level_indicator ::= 'C' | 'D' | 'E' | 'I' | 'W' file_location ::=relative_file_path
':'line_number
summary ::=summary_first_character
[message_character
*summary_last_character
] progress_suffix ::= '.' | '...'
The timing information is optional and can be enabled per message. It contains the time elapsed in seconds since the first time a message with enabled timing information was output. Later outputs of timing information never show earlier times. The number of decimal places is the same for all output timing information on a given platform and is at most 6.
relative_time_suffix ::= '[+'time_since_first_time_use
']' time_since_first_time_use ::=decimal_integer
[ '.'decimal_digit
decimal_digit
* ] 's'
continuation_line ::=indentation
continuation_line_indicator
message_character
* continuation_line_indicator ::= '␣␣|␣'
relative_file_path ::= "'"path_component
[ '/'path_component
] "'" line_number ::=decimal_integer
path_component ::=path_component_character
path_component_character
* path_component_character ::=raw_path_component_character
|escaped_path_component_character
raw_path_component_character ::= any Unicode character except from the range U+0000 to U+001F, '/', '\', ':', "'" and '"' escaped_path_component_character ::= '\x'hexadecimal_digit
hexadecimal_digit
summary_first_character ::= anysummary_last_character
except "'" (U+0027) and '|' (U+007C) summary_last_character ::= anymessage_character
except '␣' (U+0020), '.' (U+002E) and ']' (U+005D) message_character ::= any Unicode character except from the range U+0000 to U+001F decimal_integer ::=nonzero_decimal_digit
decimal_digit
* nonzero_decimal_digit ::= '1' | ... | '9' decimal_digit ::= '0' |nonzero_decimal_digit
hexadecimal_digit ::=decimal_digit
| 'a' | ... | 'f'
Module content¶
-
dlb.di.
DEBUG
¶
-
dlb.di.
INFO
¶
-
dlb.di.
WARNING
¶
-
dlb.di.
ERROR
¶
-
dlb.di.
CRITICAL
¶ Positive integers representing standard logging levels of the same names. See the documentation of logging.
In contrast to
logging
, these are not meant to be changed by the user. Use them to define your own positive integers representing levels like this:... = dlb.di.INFO + 4 # a level more important than INFO, but not yet a WARNING
-
dlb.di.
set_output_file
(file)¶ Set the output file for all future outputs of this module to file and return the old output file.
Parameters: file (an object with a write(string)
method) – new output fileReturns: the previous value, an object with a write
attribute
-
dlb.di.
set_threshold_level
(level)¶ Set the level threshold for all future messaged to level.
Every message with a level below level will be suppressed.
Parameters: level (int) – new level threshold (positive)
-
dlb.di.
is_unsuppressed_level
(level)¶ Is a message of level level unsuppressed be the current level threshold?
Return type: bool
-
dlb.di.
get_level_indicator
(level)¶ Return a unique capital ASCII letter, representing the lowest standard level not lower than level.
Example:
>>> dlb.di.get_level_indicator(dlb.di.ERROR + 1) 'E'
Parameters: level (int) – level not lower that DEBUG
-
dlb.di.
format_time_ns
(time_ns)¶ Return a string representation for a time in seconds, rounded towards 0 approximately to the resolution of
time.monotonic_ns()
. The time time_ns is given in nanoseconds as an integer.The number of decimal places is fixed for all calls. It is a platform-dependent value in the range of 1 to 6.
-
dlb.di.
format_message
(message, level)¶ Return a formatted message with aligned fields, assuming nesting level 0.
First, empty lines are removed from the beginning and the end of message and trailing white space characters is removed from each line. After that, the first line must not start with
'␣'
,"'"
,"|"
,'.'
or"]"
. If must not end with"."
or"]"
. Each non-empty line after the first line must start with at least 4 space characters after than the indentation of the first line. Example: If the first line is indented by 8 space characters, each following non-empty line must start with at least 12 space characters.message can contain fields. A field is declared by appending
'\t'
or'\b'
. A field whose declaration ends with'\t'
is left aligned, one whose declaration ends with'\t'
is right aligned over all lines of the message. In the return value, the'\t'
or'\b'
are not present, but their “positions” are aligned over all lines of the message.Examples:
>>> dlb.di.format_message('\njust a moment! ', dlb.di.WARNING) 'W just a moment!' >>> dlb.di.format_message( ... """ ... summary: ... detail: blah blah blah... ... see also here ... """, dlb.di.INFO) 'I summary: \n | detail: blah blah blah... \n | suggestion' >>> m = ''.join(f"\n {n}:\t {s} =\b {v}\b{u}" for n, s, v, u in metrics) >>> print(dlb.di.format_message('Halstead complexity measures:' + m, dlb.di.INFO)) I Halstead complexity measures: | volume: V = 1.7 | programming required: T = 127.3 s | difficulty: D = 12.8
Returns: formatted message conforming to message
after appending a single'\n'
Return type: str Raises: ValueError – if message would violate message
or if level is invalid
-
dlb.di.
inform
(message, *, level: int = INFO, with_time: bool = False)¶ If level is not suppressed, output a message to the output file after the title messages of all parent
Cluster
instances whose output was suppressed so far.message is formatted by
format_message()
and indented according the nesting level. If with_time isTrue
, arelative_time_suffix
for the current time is included.
-
class
dlb.di.
Cluster
(message, *, level=INFO, is_progress=False, with_time=False)¶ A message cluster with message as its title.
When used as a context manager, this defines a inner message cluster with message as its title; entering means an increase of the nesting level by 1.
With is_progress set to
False
, the output when the context is entered is the same as the output ofinform()
would be with the same parameters.With is_progress set to
True
, aprogress_suffix
'...'
is included in the message when the context is entered. In addition, a message'done.
or'failed with E.'
is output when the context is exited without or with an exception, respectively. See Example.
dlb.cf
— Configuration parameters¶
To change the behaviour of dlb, change the values of the following variables.
-
dlb.cf.
latest_run_summary_max_count
¶ Number of dlb runs to summarize as an integer.
When > 0, a summary of the latest latest_run_summary_max_count dlb runs is output when a root context exits.
-
dlb.cf.
max_dependency_age
¶ The maximum age of dependency information in the run-database as a
datatime.timedelta
object.Run and dependency information older than max_dependency_age is removed when a root context is entered.
max_dependency_age > datetime.timedelta(0)
must beTrue
.
-
dlb.cf.level.
RUN_PREPARATION
¶
-
dlb.cf.level.
RUN_PREPARATION
-
dlb.cf.level.
RUN_SERIALIZATION
¶
-
dlb.cf.level.
REDO_NECESSITY_CHECK
¶
-
dlb.cf.level.
REDO_REASON
¶
-
dlb.cf.level.
REDO_SUSPICIOUS_REASON
¶
-
dlb.cf.level.
REDO_PREPARATION
¶
-
dlb.cf.level.
REDO_START
¶
-
dlb.cf.level.
REDO_AFTERMATH
¶
-
dlb.cf.level.
HELPER_EXECUTION
¶
-
dlb.cf.level.
RUN_SUMMARY
¶ Assign a message level (a positive integer like
dlb.di.INFO
) to be used for all message of a given category.
dlb.ex
— Execution contexts and dependency-aware tool execution¶
Context objects¶
An (execution) context describes how running tool instances shall interact with the execution environment outside the working tree and with each other. E.g:
- number of asynchronously running tool instances
- search paths for dynamic helper files
- environment variables to be imported from
os.environ
for use in tool instances
It also controls how diagnostic messages of tool instances are handled and helps with filesystem abstraction (e.g. working tree time, case sensitivity of names in the working tree).
A context is represented as an instance of dlb.ex.Context
used as a context manager.
The context is entered with the call of _enter__()
and exited with the return of __exit__()
.
Contexts can be nested:
import dlb.ex
# no active context
with dlb.ex.Context(): # A: root context, outer context of B, C, D
# A is the active context
with dlb.ex.Context(): # B: inner context of A, outer context of C
# B is the active context
with dlb.ex.Context(): # C: inner context of A, B
# C is the active context
with dlb.ex.Context(): # D: inner context of A
# D is the active context
# A is the active context
# no active context
Combine contexts with message clusters
to describe what happens in the context:
with dlb.di.Cluster('this happens in the context'), dlb.ex.Context():
...
-
class
dlb.ex.
Context
(path_cls=dlb.fs.Path, max_parallel_redo_count=1, find_helpers=None)¶ An instance does nothing unless used as a context manager.
When used as a context manager, it embodies an (execution) context and activates it:
- a root context, if dlb is not yet running;
- an inner context of the active context, otherwise.
When a root context is entered, the working directory of the Python process must be a working tree’s root whose absolute path does not contain unresolved symbolic link.
When a context (root or not) is entered, the path of the working tree’s root must be representable as as path_cls. This allows you to impose restrictions on the accepted paths.
If find_helpers is
None
for a root context,True
is used instead. If find_helpers isNone
for an active context that is not the root context, find_helpers of the root context is used.Parameters: - path_cls (dlb.fs.Path) – the subclass of
dlb.fs.Path
to be used to represent the working tree’s root - max_parallel_redo_count (int) – maximum number of redos started in this context than can be pending at the same time
- find_helpers – are dynamic helpers not defined explicitly to be searched for in
executable_search_paths
? - find_helpers – None | bool
Raises: TypeError – if path_cls is not a subclass of
dlb.fs.Path
Entering or exiting a context may raise the following exceptions:
exception meaning when NoWorkingTreeError
the working directory is not a working tree’s root entering root context ManagementTreeError
the management tree cannot be setup inside the working tree ValueError
the working tree’s root path violates the requested restrictions entering (any) context ContextNestingError
the contexts are not properly nested exiting (any) context WorkingTreeTimeError
| working tree time behaved unexpectedly | exiting root contextNote
Most attributes and methods are available “on the class” as well as “on the instance”, and refer to the corresponding attribute of the active context:
with dlb.ex.Context: with dlb.ex.Context as c: ... = dlb.ex.Context.working_tree_time_ns # preferred ... c.active.working_tree_time_ns # also possible ... c.working_tree_time_ns # also possible
The
Context
class supports the following methods and attributes:-
active
¶ The active context.
Same on class and instance.
Raises: NotRunningError – if dlb is not running).
-
path_cls
¶ The subclass of
dlb.fs.Path
defined in the constructor.When called on class, it refers to the active context.
Raises: NotRunningError – if dlb is not running).
-
max_parallel_redo_count
¶ The maximum number of redos started in this context than can be pending at the same time, as defined in the constructor.
When called on class, it refers to the active context.
Raises: NotRunningError – if dlb is not running).
-
find_helpers
¶ Find dynamic helpers not defined explicitly are in
executable_search_paths
? This is defined in the constructor.When called on class, it refers to the active context.
Raises: NotRunningError – if dlb is not running).
-
root_path
¶ The absolute path to the working tree’s root.
It is an instance of
Context.active.path_cls
and is representable as an instance ofpath_cls
of the active context and every possible outer context.Same on class and instance.
Raises: NotRunningError – if dlb is not running).
-
executable_search_paths
¶ A duplicate-free tuple of absolute directory paths where this process should look for executables according to the operating system.
It is compiled from the members
os.get_exec_path()
when the root context is entered.Same on class and instance.
Raises: NotRunningError – if dlb is not running).
-
find_path_in
(path, search_prefixes=None)¶ Find the first existing and accessible path in search_prefixes and return its absolute path. Returns
None
if path is not found in search_prefixed.If path is a
dlb.fs.Path
withpath.is_dir()
=True
, existing non-directories in search_prefixes are ignored. If path is adlb.fs.Path
withpath.is_dir()
=False
, existing directories in search_prefixes are ignored.Relative paths in search_prefixes are treated as relative to
root_path
.If search_prefixes is
None
,executable_search_paths
is used instead.Does not raise
OSError
.Parameters: - path (
dlb.fs.Path
or anything adlb.fs.Path
can be constructed from) – the relative path to find - search_prefixes (an iterable other than
str
orbytes
orNone
) – paths of directories to search in
Returns: an absolute path or
None
.- path (
-
working_tree_time_ns
¶ The current working tree time in nanoseconds as an integer.
Same on class and instance.
Raises: NotRunningError – if dlb is not running).
-
temporary
(suffix='', is_dir=False)¶ Return a
dlb.ex.Temporary
object, representing a temporary regular file (for is_dir =False
) or a temporary directory (for is_dir =True
) in the management tree with a unique path.Usage example:
with context.temporary(suffix='.o') as p: ... # an empty file with absolute path *p* exists ... = context.temporary().path # just get the absolute path, do not create the file
The path attribute of the returned object is an absolute path in the same directory for all calls in the root context, as a
dlb.fs.Path
object. Its last component is unique among all calls in the root context.path.is_dir()
is is_dir.The unique path component starts with a lower-case letter and ends with suffix. It contains only lower-case letters and decimal digits between its first characters and the suffix. If suffix is not empty, is must start with a character from strings.punctuation and must not contain
'/'
. The the unique path component without the suffix is at most 12 characters long for the first 2**61 calls.When used as a context manager, an empty regular file or directory with path is created when entered and removed (with its content) on exit. Raises
FileExistError
if the regular file or directory exists.Same on class and instance.
Parameters: suffix (str) – suffix of the unique path component
Raises: - ValueError – if suffix is invalid
- FileExistsError – if the regular file or directory exists (when used as content manager)
- NotRunningError – if dlb is not running).
-
working_tree_path_of(path, *, is_dir=None, existing=False, collapsable=False,
-
allow_nontemporary_management=False, allow_temporary=False)
Return the managed tree path of the path of a filesystem object in the managed tree.
For path to be considered as the path of a filesystem object in the managed tree, path must either be a relative path or it must have
root_path
as a prefix.The arguments existing and collapsable describe the assumptions on the filesystem content that may be used to increase the speed and reduce the number of filesystem accesses.
If existing and collapsable are
True
and path is relative, the filesystem is never accessed.If existing is
False
,is_dir()
of the returned path reflects the type of the actual filesystem object. Raisesdlb.fs.PathNormalizationError
if path does not exist.If allow_nontemporary_management is
True
, the resulting path may denote a filesystem object in the management tree except in.dlbroot/t
. If allow_temporary isTrue
, the resulting path may denote a filesystem object in.dlbroot/t
of the management tree.Does not raise
OSError
.Same on class and instance.
Parameters: - path (
dlb.fs.Path
or anything adlb.fs.Path
can be constructed from) – a path of a filesystem object in the managed tree - is_dir (NoneType | bool) –
True
if this is a directory path,False
if not andNone
for derivation from path - existing (bool) – assume that all involved filesystem objects exist?
- collapsable (bool) – assume that any relative to the working tree root is collapsable?
- allow_nontemporary_management (bool) – is the path permitted to denote a filesystem object the :term`management tree` except ones
in
.dlbroot/t
? - allow_temporary (bool) – is the path permitted to denote a filesystem object in
.dlbroot/t
of the management tree?
Returns: a
dlb.fs.Path
p withp.is_absolute() == False
andp.is_normalized() == True
Return type: same class as path if path is a
dlb.fs.Path
anddlb.fs.Path
otherwiseRaises: - dlb.fs.PathNormalizationError – if path does not exist in the parts of the working tree that are described by allow_nontemporary_management and allow_temporary
- ValueError – if the resulting path is not representable
- NotRunningError – if dlb is not running).
- path (
-
env
¶ The environment variable dictionary object with this context as its associated context.
When called on class, it refers to the active context.
Raises: NotRunningError – if dlb is not running).
-
helper
¶ The dynamic helper dictionary object with this context as its associated context.
The dynamic helper dictionary object maps dynamic helpers to absolute paths, either explicitly or implicitly with the help of
find_path_in()
.If the active context and the root context both have
find_helpers
=False
and no paths was explicitly assigned to the dynamic helper p in the active context or one of its outer contexts, a look-up withdlb.ex.Context.helper[p]
performs a search withdlb.ex.Context.find_path_in(p)
. (Each such search is performed only once for a given path; the result is stored.)Examples:
>>> dlb.ex.Context.helper['gcc'] Path('/usr/bin/gcc') >>> dlb.ex.Context.helper['gcc'] = '/usr/local/bin/my-very-own-very-special-gcc' # set the path explicitly >>> dlb.ex.Context.helper['gcc'] Path('/usr/local/bin/my-very-own-very-special-gcc') >>> dlb.ex.Context.helper['tmp/'] = 'out/t/' # relative path: relative to the working tree's root path >>> dlb.ex.Context.helper['tmp/'] Path('/home/schmutzli/projects/esel/out/t') # with '/home/schmutzli/projects/esel' as the working tree's root
When called on class, it refers to the active context.
Raises: NotRunningError – if dlb is not running).
Environment variable dictionary objects¶
The environment variable dictionary object env returned by c.env
for a context c
is a dictionary-like object of all environment variables defined in this c.
c is called the associated context of env.
In addition, the environment variable dictionary object manages the import of environment variables from environment variables of the outer context and restriction of imported or assigned values in the form of regular expressions.
The environment variables of the outer context of the root context is defined
by os.environ
.
Example:
# os.environ usually contains the environment variables in the shell that called the Python interpreter
with dlb.ex.Context(): # takes a snapshot of os.environ
# import the environment variable 'LANG' into the context
dlb.ex.Context.active.env.import_from_outer(
'LANG', restriction=r'[a-z]{2}_[A-Z]{2}', example='sv_SE')
# now the environment variable is either undefined or matches the regular expression given
# (in this context and all future inner contexts)
... = dlb.ex.Context.active.env['LANG']
# value in snapshot of os.environ complying to the restriction or KeyError
dlb.ex.Context.active.env['LANG'] = 'de_AT'
with dlb.ex.Context():
# further restrict the value and make sure it is defined
dlb.ex.Context.active.env.import_from_outer(
'LANG', restriction='(?P<language>de).*', example='de_CH')
... = dlb.ex.Context.active.env['LANG'] # 'de_AT'
del dlb.ex.Context.active.env['LANG']
dlb.ex.Context.active.env['LANG'] = 'de_CH'
# dlb.ex.Context.active.env['LANG'] = 'fr_FR' # would raise ValueError
... = dlb.ex.Context.active.env['LANG'] # 'de_AT'
del dlb.ex.Context.active.env['LANG'] # undefine 'LANG'
dlb.ex.Context.active.env['LANG'] = 'fr_FR' # ok
Environment variable dictionary object support the following methods and attributes:
-
EnvVarDict.
import_from_outer
(name, restriction, value_if_undefined=None, example=None)¶ Sets the value of the environment variable named name from the innermost outer context that defines it. If no outer context defines it, the environment variable remains undefined.
Also sets the importing restriction for the value of the environment variable; when it is or later becomes defined, it regular expression restriction must match its value.
The possible imported value and the importing restriction apply to the context and all its future inner contexts.
When called for a root contest, the environment variables are imported from
os.environ
at the time is was entered.Parameters: - name (str) – (non-empty) name of the environment variable
- restriction (str |
typing.Pattern
) – regular expression - example (str) – typical value of a environment variable, restriction must match this
Raises: - ValueError – if an environment variable named name is defined in the associated or an outer context and restriction does not match its value
- NonActiveContextAccessError – if the associated context is not an active context
-
EnvVarDict.
is_imported
(name)¶ Returns
True
if name is the name of an environment variable imported in the associated context or any of its outer contexts, else False.Parameters: name (str) – non-empty name of an environment variable
Raises: - TypeError – if name is not a string
- ValueError – if name is an empty string
-
EnvVarDict.
get
(name, default=None)¶ Return its value if name is the name of a defined environment variable in the associated context, else default.
Parameters: name (str) – non-empty name of an environment variable
Raises: - TypeError – if name is not a string
- ValueError – if name is an empty string
-
EnvVarDict.
items
()¶ Returns a new view of the dictionary’s items (name, value) pairs of all defined environment variables.
-
name in env
Returns True if there is a environment variable named name defined in env, else False.
-
name not in env
Equivalent to
not name in env
-
env[name] = value
Defines an imported environment variable named name with value value in the associated context and all its future inner contexts.
Raises
KeyError
, if name was not imported in the associated context or one of its outer contexts.Raises
ValueError
, if name was imported in the associated context or one of its outer contexts, but is invalid with respect to the restriction an importing context (can be this context and any outer context).Raises
NonActiveContextAccessError
, if the associated context is not an active context.
-
del env[name]
Undefines a defined environment variable named name in the associated context and all its future inner contexts.
Raises
KeyError
, if name is not defined in the context.Raises
NonActiveContextAccessError
, if the associated context is not an active context.
Tool objects¶
Every tool is represented by a subclass of Tool
that describes its abstract behaviour and the way it
is run (e.g. meaning of command line and output, interaction with file system and environment variables).
Tools are usually parametrized by dependency roles (e.g. input files) and execution parameters.
Each tool instance represents a concrete behaviour and can be run in an active context. Running a tool results in an awaitable result object.
Tool instances are immutable and hashable and fast to construct; the heavy lifting takes place while the tool instance is running.
Tools are customized by inheritance and defining class attributes.
-
class
dlb.ex.
Tool
¶ A tool declares its dependency roles (e.g.
map_file_dependency
) and execution parameters (e.g.DO_INCLUDE_DEBUG_INFO
,PAPER_FORMAT
) as class attributes.Every tool instance assigns concrete dependencies for the tool’s dependency roles (e.g. a filesystem path
'./out/hello.map'
for a dependency rolemap_file_dependency
), while the execution parameters are the same of all instances of the some tool.Dependency roles are instances of subclasses of
Tool.Dependency
.A new tool can be defined by inheriting from one or more other tools. When overriding a dependency roles, its overriding value must be of the same type as the overridden value and it must be at least as restrictive (e.g. if required dependency must not be overridden by a non-required one). When overriding an execution parameters, its overriding value must be of the same type as the overridden value.
Each subclass of
Tool
must be defined in a source code location unique among all subclasses ofTool
. The definition raisesDefinitionAmbiguityError
, if its location is cannot be determined or if another subclass ofTool
was defined before at the same location.Example:
class Compiler(dlb.ex.Tool): WARNINGS = ('all',) source_file = dlb.ex.Tool.Input.RegularFile() object_file = dlb.ex.Tool.Output.RegularFile() class Linker(dlb.ex.Tool): object_files = dlb.ex.Tool.Input.RegularFile[1:]() linked_file = dlb.ex.Tool.Output.RegularFile() map_file = dlb.ex.Tool.Output.RegularFile(required=False) compiler = Compiler(source_file='main.cpp', object_file='main.cpp.o') linker = Linker(object_files=[compiler.object_file], linked_file='main')
At construction of a tool, the dependencies given as keyword arguments to the constructor are validated by the tool’s dependency roles and made accessible (for reading only) as an attribute with the name of the corresponding dependency role and a type determined by the dependency role (e.g.
dlb.fs.Path
forTool.Input.RegularFile
):>>> Compiler.object_file # dependency role <dlb.ex.Tool.Input.RegularFile object at ...> >>> compiler.object_file # dependency Path('main.cpp.o')
-
run
(force_redo=False)¶ Run the tool instance in the active context and returns a result (proxy) object result.
bool(result)
isTrue
if a redo is performed andFalse
otherwise.A redo is performed if force_redo is
True
or if it is necessary.If a redo is performed, this method returns before the (asynchronous) redo is complete. After each of the following actions the redo is guaranteed to be complete (either successfully or by raising an exception):
- read of a “public” attribute of the result proxy object
- exit of the context
run()
was called in - enter of an inner context of the context
run()
was called in - modification of
env
orhelper
of the contextrun()
was called in - call of
run()
of the same tool instance
The result object contains an attribute for every dependency role of the tool which contains the concrete dependencies.
If
bool(result)
isTrue
, all attributes for dependencies have an assigned value. Ifbool(result)
isFalse
, only the attributes for explicit dependencies have an assigned value; the value of all attributes for non-explicit dependencies isNotImplemented
.
-
redo
(result, context)¶ Overwrite this method to implement a new
Tool
.result is the result object that will by returned by the calling
run()
. context is the redo context (seeTool.RedoContext
).Use
context.execute_helper()
andcontext.replace_output()
.Assign to attributes of result to define a non-explicit concrete dependency for the dependency role with the same name.
For a redo to be successful, this method must perform the following tasks:
- Create all explicit output dependencies
- Assign values to each required non-explicit dependencies
For a filesystem object whose path p is contained in an output dependency, it is recommended to first write to a temporary filesystem object q and then replace it with
context.replace_output(p, q)
. This guarantees that no incomplete output dependency is left behind (like an only half-written object file) when the redo is aborted.A filesystem object that is an output dependencies is treated as modified be the redo if it is a non-explicit dependency or if it is a explicit dependency that was replaced with context.replace_output().
Raises
RuntimeError
on the attempt to enter a newdlb.ex.Context
as a context manager or to modify the active context.Return
True
if the next run this tool instance should perform a redo, regardless of the necessity according to its dependencies.Example:
class ATool(dlb.ex.Tool): EXECUTABLE = 'atool' source_file = dlb.ex.Tool.Input.RegularFile() output_file = dlb.ex.Tool.Output.RegularFile() included_files = dlb.ex.Tool.Input.RegularFile[:](explicit=False) async def redo(self, result, context): if ...: raise ValueException('invalid ...') with context.temporary() as temp_file_ await context.execute_helper(self.EXECUTABLE, ['-o', temp_file, result.source_file]) result.included_files = ... context.replace_output(result.output_file, temp_file)
-
definition_location
¶ The definition location of the class.
It is a tuple of the form
(file_path, in_archive_path, lineno)
and uniquely identifies the tool among all subclasses ofTool
.in_archive_path is
None
, if the class was defined in an existing Python source file, and file_path is theos.path.realpath()
of this file.in_archive_path is the path relative of the source file in the zip archive, if the class was defined in an existing zip archive with a filename ending in
.zip
(loaded byzipimport
) and file_path is theos.path.realpath()
of this zip archive.lineno is the 1-based line number in the source file.
-
fingerprint
¶ The permanent local tool instance fingerprint of this instance.
This is a
bytes
object of fixed size, calculated from all its concrete dependencies d withd.explicit
=True
.If two instances of the same subclass of
Tool
have “similar” explicit dependencies, their fingerprints are equal. If two instances of the same subclass ofTool
have explicit dependencies that are not “similar”, their fingerprints are different with very high probability.The explicit dependencies of two instances are considered “similar”, if they are equal or differ in a way that does not affect the meaning of the dependencies while the tool instance is running.
-
Redo context¶
A redo context is a read-only view for a dlb.ex.Context
with some additional
methods related to dynamic helpers and dependencies.
-
class
Tool.
RedoContext
¶ A redo context is constructed automatically by
Tool.run()
.-
execute_helper(helper_file, arguments=(), *, cwd=None, expected_returncodes=frozenset([0]),
-
forced_env={}, stdin=None, stdout=None, stderr=None, limit=2**16)
Execute the helper_file with command-line arguments arguments in a subprocess with cwd as its working directory and wait for it to complete. The execution is considered successful if an only if its returncode is one in expected_returncodes.
If cwd is not
None
, is must be the path of directory in the managed tree or in.dlbroot/t/
of the management tree. Otherwise the working tree’s root is used as the working directory.All members of arguments are converted to str objects.
If a member of arguments is a
dlb.fs.Path
object p withp.is_absolute()
=True
, is is replaced bystr(p.native)
. If a member of arguments is adlb.fs.Path
object p withp.is_absolute()
=False
, is is replaced bystr(q.native)
, where q is p expressed relative to the working directory. Is must denote a filesystem object in the managed tree or in.dlbroot/t/
of the management tree.env
of this object, modified by forced_env, forms the environment for the subprocess.Parameters: - helper_file – dynamic helper to be executed as a relative path
- arguments (iterable of objects that can be converted to str) – command-line arguments
- cwd (
None
or adlb.fs.Path
or anything adlb.fs.Path
can be constructed from) – working directory of the subprocess to be started - expected_returncodes (collection of integers) – expected return codes of the dynamic helper helper_file
- forced_env (None | Dict[str, str]) – dictionary of values to override in
env
orNone
- stdin – If not
None
: either a file-like object representing a pipe to be connected to the subprocess’s standard input stream usingasyncio.loop.connect_read_pipe()
, or thesubprocess.PIPE
constant. - stdout – If not
None
: either a file-like object representing the pipe to be connected to the subprocess’s standard output stream usingasyncio.loop.connect_read_pipe()
, or thesubprocess.PIPE
constant. - stderr – If not
None
: either a file-like object representing the pipe to be connected to the subprocess’s standard error stream usingasyncio.loop.connect_read_pipe()
, or one ofsubprocess.PIPE
orsubprocess.STDOUT
constants. - limit – the buffer limit for
StreamReader
wrappers forProcess.stdout
andProcess.stderr
(ifsubprocess.PIPE
is passed to stdout and stderr arguments).
Raises: HelperExecutionError – if the subprocess exits with a returncode not in expected_returncodes.
Returns the tuple
(returncode, stdout_data, stderr_data)
. returncode is the returncode (contained in expected_returncodes). stdout_data and stderr_data are bytes object with the received data from stdout and stderr, respectively.
-
replace_output(path, source):
Replace the — existing or non-existent — filesystem object path by source. path must be contained in a dependency of the tool instance.
path and source must be different filesystem objects.
After successful completion, path exists and source does not exist. If the parent directory of path does not exist, it is created (with all its parent directories).
The actual operation depends on the corresponding dependency role. If is it a
dlb.ex.Tool.Output.RegularFile
withreplace_by_same_content
=False
and path and source both exist with the same content, path is no replaced and treated as unchanged.If path is replaced, this is always done by an atomic operation. If it fails, path is either source afterwards or it does not exist.
Parameters: - path (
dlb.fs.Path
or anything adlb.fs.Path
can be constructed from) – a path of a future filesystem object in the managed tree - source (
dlb.fs.Path
or anything adlb.fs.Path
can be constructed from) – a path of a filesystem object in the managed tree
Raises: ValueError – if path is not a managed tree path contained in an explicit output dependency or source is not a working tree path of a filesystem object in the managed tree or in
.dlbroot/t/
of the management tree that is different from path.- path (
-
Dependency classes¶
A dependency class is a subclass of Tool.Dependency
.
Its instances describe dependency roles (as attributes of a Tool
).
The Tool.Dependency.validate()
methods of dependency classes are used by tool instances
to create concrete dependencies from their constructor arguments.
Each dependency role has an multiplicity specification:
An instance d of a dependency class D created with
D(...)
has a multiplicity ofNone
which means that its concrete dependency must be a single object (its type depends on D only) orNone
.An instance d of a dependency class D created with
D[m](...)
has a multiplicity of m which means that its concrete dependencies are a sequence of objects (their type depends on D only) orNone
. The accepted number of members is specified by m.m can be any non-negative integer or any meaningful
proper_slice
(of non-negative integers). A number of members is accepted if and only if is either equal to m or contained inrange(n + 1)[m]
.
Example:
class Tool(dlb.ex.Tool):
# these are dependency roles of the tool 'Tool':
include_search_paths = dlb.ex.Tool.Input.Directory[1:]() # a sequence of at least one dlb.ex.Tool.Input.Directory
cache_dir_path = dlb.ex.Tool.Input.Directory() # a single dlb.ex.Tool.Input.Directory
tool = Tool(include_search_paths=['build/out/Generated/', 'src/Implementation/'])
# these are concrete dependencies of the tool instance 'tool':
tool.include_search_paths # (Path('build/out/Generated/'), Path('src/Implementation/'))
tool.cache_dir_path # (Path('build/out/Generated/'), Path('src/Implementation/'))
Dependency classes are organized in an a hierarchy according to their meaning to a tool by the means of the following abstract classes:
-
class
Tool.
Input
¶ A
Tool.Dependency
that describes an input dependency of a tool.The tool instance must be redone if it (e.g. the mtime of a file) has changed compared to the state before the last successful redo of the tool instance.
An redo must not modify it, successful or not (the same object can be an output dependency of the same tool instance though which can be modified).
-
class
Tool.
Output
¶ A
Tool.Dependency
that describes an output dependency of a tool.A successful redo must generate it (e.g. create a regular file).
These are all abstract classes and contain inner classes derived from them.
Example: Tool.Output.Directory
is a non-abstract dependency class derived
from Tool.Output
.
Note
dlb identifies filesystem objects by their managed tree path. It assumes that different managed tree paths point to different filesystem objects.
If a filesystem object serves as an output dependency of one tool instance and as an input dependency of another: Make sure both dependencies use the same path. A redo miss could happen otherwise.
You are always safe without hard links, symbolic links and case-insensitive filesystems.
Concrete dependency role classes support the following methods and attributes:
-
class
Tool.
Dependency
(required=True, explicit=True)¶ If required is
True
, a concrete dependency of this dependency role will never beNone
.If explicit is
True
, the concrete dependency can and must be fully defined when the tool instance is created. Otherwise, it cannot and must not be, but automatically assigned byTool.run()
.Each supported constructor argument is available as a property of the same name.
Raises: DependencyError – if the arguments of the constructor do not match the declared dependency roles of the class -
class
Value
¶ A (potentially abstract) class such that
isinstance(v, Value)
isTrue
for each validated single value v of each instance t of this class.This is the type of
t.validate()
ifmultiplicity
isNone
and the type of each member oft.validate()
otherwise.
-
validate
(value)¶ Parameters: value (Any type the concrete dependency can convert to T) – The concrete dependency to convert and validate except None
Returns: The validated value of type T Raises: TypeError – If multiplicity
is notNone
and value is not iterable or is a string
-
compatible_and_no_less_restrictive
(other)¶ Is this dependency role an instance of the same class as other with a multiplicity and properties no less restrictive than the ones of other?
Parameters: other (Tool.Dependency) – reference dependency role Return type: bool
-
multiplicity
¶ The multiplicity of the dependency role.
Is
None
or adlb.ex.mult.MultiplicityRange
.
-
tuple_from_value
(value)¶ Return value if
multiplicity
isNone
and a tuple of its members otherwise.Example:
# returns a tuple of t.Value objects or raises an exception: >>> v = t.tuple_from_value(t.validate(...))
-
class
Input dependency role classes¶
Dependency role class | Keyword arguments of constructor | |
---|---|---|
Name | Default value | |
Tool.Input.RegularFile |
cls | dlb.fs.Path |
Tool.Input.NonRegularFile |
cls | dlb.fs.Path |
Tool.Input.Directory |
cls | dlb.fs.Path |
Tool.Input.EnvVar |
name | |
restriction | ||
example |
In addition to the keyword arguments of the specific constructors described here, all constructors also accept the
keyword arguments of the constructor of Tool.Dependency
.
-
class
Tool.Input.
RegularFile
(cls=dlb.fs.Path)¶ Constructs a dependency role for a regular files, identified by their paths.
If a path is relative, is it treated as relative to
dlb.ex.Context.root_path
, and it must be collapsable and non-upwards (if the path does not contain..
components, these requirements are met).Files outside the managed tree are assumed to remain unchanged between runs of dlb.
The
validated value
of a concrete dependency is the file’s path as an instance of cls ifmultiplicity
isNone
and a tuple of the file’s paths otherwise.Example:
>>> class Tool(dlb.ex.Tool): >>> source_files = dlb.ex.Tool.Input.RegularFile[1:](cls=dlb.fs.NoSpacePath) >>> tool = Tool(source_files=['src/main.cpp']) >>> tool.source_files (NoSpacePath('src/main.cpp'),)
Parameters: cls (dlb.fs.Path) – class to be used to represent the path -
class
Value
¶ Is
dlb.fs.Path
.
-
class
-
class
Tool.Input.
NonRegularFile
(cls=dlb.fs.Path)¶ Constructs a dependency role for filesystem objects that are neither directories nor regular files, identified by their paths.
If a path is relative, is it treated as relative to
dlb.ex.Context.root_path
, and it must be collapsable and non-upwards (if the path does not contain..
components, these requirements are met).Files outside the managed tree are assumed to remain unchanged between runs of dlb.
The
validated value
of a concrete dependency is the file’s path as an instance of cls ifmultiplicity
isNone
and a tuple of the file’s paths otherwise.Example:
>>> class Tool(dlb.ex.Tool): >>> symlinks = dlb.ex.Tool.Input.NonRegularFile[:](cls=dlb.fs.NoSpacePath) >>> tool = Tool(symlinks=['src/current']) >>> tool.symlinks (NoSpacePath('src/current'),)
Parameters: cls (dlb.fs.Path) – class to be used to represent the path -
class
Value
¶ Is
dlb.fs.Path
.
-
class
-
class
Tool.Input.
Directory
(cls=dlb.fs.Path)¶ Constructs a dependency role for directories, identified by their paths.
If a path is relative, is it treated as relative to
dlb.ex.Context.root_path
, and it must be collapsable and non-upwards (if the path does not contain..
components, these requirements are met).Directories outside the managed tree are assumed to remain unchanged between runs of dlb.
The
validated value
of a concrete dependency is the directory’s path as an instance of cls ifmultiplicity
isNone
and a tuple of the directory’s paths otherwise.Example:
>>> class Tool(dlb.ex.Tool): >>> cache_directory = dlb.ex.Tool.Input.Directory(required=False) >>> tool = Tool(cache_directory='/tmp/') >>> tool.cache_directory Path('tmp/')
Parameters: cls (dlb.fs.Path) – class to be used to represent the path -
class
Value
¶ Is
dlb.fs.Path
.
-
class
-
class
Tool.Input.
EnvVar
(name, restriction, example)¶ Constructs a dependency role for a environment variable named name. It must not have a multiplicity (other than
None
).If explicit is
False
, the value assign in the constructor of the tool instance is used for all future runs of the tool instance. Otherwise, the current value of the active context is used each timeTool.run()
is called.The value of the environment variable is valid if it a string that matches the regular expression restriction, or if it is
None
and required isFalse
.The
validated value
of a concrete dependency is aValue
instance with the environment variable’s name and value.Example:
>>> class Tool(dlb.ex.Tool): >>> language = dlb.ex.Tool.Input.EnvVar( >>> name='LANG', >>> restriction=r'(?P<language>[a-z]{2})_(?P<territory>[A-Z]{2})', >>> example='sv_SE') >>> flags = dlb.ex.Tool.Input.EnvVar(name='CFLAGS', restriction=r'.+', example='-Wall') >>> tool = Tool(language='de_CH') # use 'de_CH' as value of the environment variable for all >>> tool.language.value['territory'] 'CH' >>> tool.flags NotImplemented >>> tool.run().flags.value # assuming dlb.ex.Context.env['CFLAGS'] of '-O2' '-O2'
Parameters: - restriction (str |
typing.Pattern
) – regular expression - example (str) – typical value of a environment variable, restriction must match this
-
class
Value
¶ A
dataclasses.dataclass
object with the following attributes:-
name
¶ The name of the environment variable, as in the corresponding concrete dependency.
-
raw
¶ The value of the environment variable.
-
groups
¶ The named groups of restriction of the corresponding concrete dependency when matched against raw.
-
- restriction (str |
Concrete output dependency role classes¶
Dependency role class | Keyword arguments of constructor | |
---|---|---|
Name | Default value | |
Tool.Output.RegularFile |
cls replace_by_same_content | dlb.fs.Path
True |
Tool.Output.NonRegularFile |
cls | dlb.fs.Path |
Tool.Output.Directory |
cls | dlb.fs.Path |
Tool.Output.Object |
In addition to the keyword arguments of the specific constructors described here, all constructors also accept the
keyword arguments of the constructor of Tool.Dependency
.
-
class
Tool.Output.
RegularFile
(cls=dlb.fs.Path, replace_by_same_content=True)¶ Constructs a dependency role for regular files in the managed tree, identified by their paths.
If a path is relative, is it treated as relative to
dlb.ex.Context.root_path
, and it must be collapsable and non-upwards (if the path does not contain..
components, these requirements are met).The
validated value
of a concrete dependency is the file’s path as an instance of cls ifmultiplicity
isNone
and a tuple of the file’s paths otherwise.If replace_by_same_content is
False
for a dependency role containing p,context.replace_output(p, q)
inredo(..., context)
does not replace p if p and q both exist as accessible regular files and have the same content.Example:
>>> class Tool(dlb.ex.Tool): >>> object_file = dlb.ex.Tool.Output.RegularFile(cls=dlb.fs.NoSpacePath) >>> tool = Tool(object_file=['main.cpp.o']) >>> tool.object_file (NoSpacePath('main.cpp.o'),)
Parameters: cls (dlb.fs.Path) – class to be used to represent the path -
class
Value
¶ Is
dlb.fs.Path
.
-
class
-
class
Tool.Output.
NonRegularFile
(cls=dlb.fs.Path)¶ Constructs a dependency role for filesystem objects in the managed tree that are neither directories nor regular files, identified by their paths.
If a path is relative, is it treated as relative to
dlb.ex.Context.root_path
, and it must be collapsable and non-upwards (if the path does not contain..
components, these requirements are met).The
validated value
of a concrete dependency is the file’s path as an instance of cls ifmultiplicity
isNone
and a tuple of the file’s paths otherwise.Example:
>>> class Tool(dlb.ex.Tool): >>> symlinks = dlb.ex.Tool.Output.NonRegularFile[:](cls=dlb.fs.NoSpacePath) >>> tool = Tool(symlinks=['dist']) >>> tool.symlinks (NoSpacePath('src/current'),)
Parameters: cls (dlb.fs.Path) – class to be used to represent the path -
class
Value
¶ Is
dlb.fs.Path
.
-
class
-
class
Tool.Output.
Directory
(cls=dlb.fs.Path)¶ Constructs a dependency role for directories in the managed tree, identified by their paths.
If a path is relative, is it treated as relative to
dlb.ex.Context.root_path
, and it must be collapsable and non-upwards (if the path does not contain..
components, these requirements are met).The
validated value
of a concrete dependency is the directory’s path as an instance of cls ifmultiplicity
isNone
and a tuple of the directory’s paths otherwise.Example:
>>> class Tool(dlb.ex.Tool): >>> html_root_directory = dlb.ex.Tool.Output.Directory(required=False) >>> tool = Tool(html_root_directory='html/') >>> tool.html_root_directory Path('html/')
Parameters: cls (dlb.fs.Path) – class to be used to represent the path -
class
Value
¶ Is
dlb.fs.Path
.
-
class
-
class
Tool.Output.
Object
¶ Constructs a dependency role for any Python object other than
None
andNotImplemented
. It must not be explicit.The
validated value
of a concrete dependency is adeep copy
of the value.-
class
Value
¶ Is
typing.Any
.
-
class
Exceptions¶
-
exception
dlb.ex.
NotRunningError
¶ Raised, when an action requires an active context while dlb was not running.
-
exception
dlb.ex.
NoWorkingTreeError
¶ Raised, when the working directory of the calling process is not a working tree’s root.
-
exception
dlb.ex.
ManagementTreeError
¶ Raised, when an attempt to prepare or access the management tree failed.
-
exception
dlb.ex.
ContextNestingError
¶ Raised, when some contexts were not properly nested. I.e. the calls of
__exit__()
did not occur in the opposite order of the corresponding calls of__enter__()
.
-
exception
dlb.ex.
WorkingTreeTimeError
¶ Raised, when the working tree time behaved unexpectedly.
-
exception
dlb.ex.
ContextModificationError
¶ Raised, when an environment variable dictionary object or a helper dictionary object is modified while its associated context is not the active context.
-
exception
dlb.ex.
WorkingTreePathError
¶ Raised, when a path is not a working tree path with certain properties where it should be.
-
exception
dlb.ex.
DefinitionAmbiguityError
¶ Raised at the definition of a subclass of
Tool
, when the location is unknown or another subclass ofTool
was defined before at the same location.
-
exception
dlb.ex.
DependencyError
¶ Raised when a running tool instance detects a problem with its dependencies before a redo.
-
exception
dlb.ex.
ExecutionParameterError
¶ Raised when a running tool instance detects a problem with its execution parameters before a redo.
-
exception
dlb.ex.
RedoError
¶ Raised when a running tool instance detects a problem with its dependencies during or after a redo.
-
exception
dlb.ex.
HelperExecutionError
¶ Raised, when the execution of a dynamic helper file failed.
Footnotes
[1] | Possible application: monitoring of the build progress on a build server. |
[2] | The update of mtime for an mmap ’d file
conforming to ISO 1003.1-2008
after a write to the mapped memory is only guaranteed via msync() .
Therefore such a write operation is not considered complete before the next call of msync()
(which may never happen).
Actual behaviour of mmap on different operating systems (2018): https://apenwarr.ca/log/20181113. |
[3] | Especially, mtime is not manually set with touch -t or any tool that uses a coarser time resolution than the effective mtime resolution. See touch and utimensat(). |
[4] | Linux currently (2020) uses ktime_get_coarse_real_ts64() as time source for its (optional) mtime updates, which which returns the system-wide realtime clock at the last tick. |
[5] | (1, 2, 3) Some filesystems support mount options to sacrifice this guaranteed for performance. Example: Ext4 with mount option lazytime. |
[6] | adjtime() is not covered by ISO 1003.1-2008.
It originated in 4.3BSD and System V.
For many operating systems it states “the clock is always monotonically increasing”
(Linux,
OpenBSD,
FreeBSD). |
[7] | Resolution for this situation: Try later (delay should be part of the message) or correct the mtime of the affected filesystem objects (list should be part of the message). |