Usage overview¶

Recommendations¶

These recommendation describe the typical use case. Use them as a starting point for most efficient and reliable operation. [1]

Setup a working tree¶

Place the entire working tree on the same file system with a decently fine effective mtime resolution (no courser than 100 ms). XFS or Ext4 are fine. Avoid FAT32. [2]

Make sure the filesystem is mounted with “normal” (immediate) update of mtime (e.g. without lazytime for Ext4). [3]

Place all input files (that are only read by tool instances) in a filesystem tree in the working tree that is not modified by tool instances.

This is not required but good practice. It also enables you to use operating system specific features to protect the build against accidental changes of input files. For example: Protect the input files from change by a transparent read-only filesystem mounted on top of it during the build.

Do not use symbolic links in the managed tree to filesystem objects not in the same managed tree.

Run dlb¶

Do not modify the management tree unless told so by dlb. [4]

Do not modify the mtime of filesystem objects in the working tree manually while dlb is running. [11]

Do not modify the content of filesystem objects in the managed tree on purpose while dlb is running, if they are used as input dependencies or output dependencies of a tool instance.

Yes, I know: it happens a lot by mistake when editing source files.

dlb itself is designed to be relatively robust to such modifications. As long as the size of modified regular file changes or the working tree time is monotonic, there is no redo miss in the current or in any future run of dlb. [5] [6]

However, many external tools cannot guarantee proper behaviour if some of their input files are changed while they are being executed (e.g. a compiler working on multiple input files).

Avoid mv to replace regular files; is does not update its target’s mtime.

Use cp instead.

Be careful when you modify a file via mmap that is an input dependency of a tool instance. [12]

Do not put the system time used as working tree’s system time back on purpose while dlb is running or while you are modifying the managed tree. [9]

Write scripts and tools¶

Do not modify the managed tree in a script inside a root context, e.g. by calling shutil.rmtree() directly. [5]

Use tool instances instead.

It is safe to modify the managed tree immediately after a run of dlb is completed (e.g. in the same script, without risking a redo miss [7]

Do not use (explicit) multithreading. Use asyncio instead.

Do not use multiple hierarchical scripts (where one calls another). This would be error-prone an inefficient. Use scripts only on the top-level.

Split large scripts into small modules that are imported by the script. You can place these modules in the directory they control.
Use only one root context and nest all other contexts inside (even in modules imported inside this context). [8]

Do:
import dlb.ex
...
with dlb.ex.Context():
    with dlb.ex.Context():
        ...
    with dlb.ex.Context():
        ...
    import build_subsystem_a  # contains 'with dlb.ex.Context(): ... '
Don’t:
import dlb.ex
...

with dlb.ex.Context():
   ...  # context manager exit is artificially delayed as necessary according to the
        # filesystem's effective mtime resolution

with dlb.ex.Context():
   ...  # context manager exit is artificially delayed as necessary according to the
        # filesystem's effective mtime resolution (again)
Use context to serialize groups of running tool instances, even when running in parallel [10]:
with dlb.ex.Context(max_parallel_redo_count=4):
    ...

...  #  all running tool instances are completed here

with dlb.ex.Context():
    ...

Layout of working tree¶

The directory .dlbroot/ is mandatory - it marks its parent directory as the root of a dlb working tree. Everything else is optional. dlb does never touch filesystem objects outside .dlbroot/ unless explicitly directed by a script to do so.

The directory .dlbroot/u/ is optional. dlb does never touch its content. Each regular file or a symbolic links to a regular file in .dlbroot/u/ whose name ends in .zip should be a zip archive loadable by zipimport. You can place dlb as .dlbroot/u/dlb.zip in this directory (under version control). This makes the working tree almost self-contained; only an external Python interpreter is needed.

If you use Git for version control which does not support empty directories, add an empty regular file .dlbroot/o or any file in .dlbroot/u/.

The lines marked with * show filesystem objects only given as an example.

Before first run of a dlb script:

.dlbroot/
src/                    *
   a.c                  *
   a.h                  *
   b.c                  *
test/                   *
...

During a run of a dlb script:

.dlbroot/
    o                   empty regular file, used to probe the "current" mtime
    runs-*.sqlite       run-database
    t/                  temporary files
        a.o             *
        b.o             *
 src/                   *
   a.c                  *
   a.h                  *
   b.c                  *
 test/                  *
 out/                   *
   p                    *
 dist/                  *
 ...

After a successful run of a dlb script:

.dlbroot/
    o                   empty regular file
    runs-*.sqlite       run-database
 src/                   *
   a.c                  *
   a.h                  *
   b.c                  *
 test/                  *
 out/                   *
   a.o                  *
   b.o                  *
 dist/                  *
   p                    *
 ...

[1]	Although they are not formally specified, Make has by design much stricter requirements and much looser guarantees.

[2]	A-F1, A-T3

[3]	A-F2, A-F3, A-F4

[4]

A-A1

[5]	(1, 2) A-A2, G-D1, G-D2, G-D3

[6]

Make is very vulnerable to this. Even with a monotonically increasing working tree time, the inputs (sources of a rule) must not be changed from the moment its recipe’s execution is started until the next increase of the working tree time after the recipe’s execution is completed. Otherwise, there is a redo miss in every future run - until the working tree time a an input is changed again in a way that does not cause a redo miss.