Test¶

A feature or fix is not complete until it is proven. The order of preference for proof:

1. A host test that runs in CI.
2. A bench drill on the bench reference, recorded in the commit message that carries the change.
3. Documented reasoning.

A change that can put energy where it was not commanded gets a regression test with the feature or fix, not after it (Contribute). This page is the inventory. It lists the host tests in each repository, the CI workflows, and the tests of the acceptance tool. It also gives the coverage rule that connects a code change to the release gate.

Host tests, for each repository¶

The null-sink controller¶

On a host without GFSINK, the grblHAL driver runs the real core, the planner, and the stream code against a null sink instead of /dev/glowforge. With GFSINK_DUMP set, it writes each pulse byte that it would ship to a file. This makes the laser path testable without hardware. The eleven harnesses from forgefirm/scripts/bench/ that the CI of the driver runs all run against the null-sink build. The two lists that follow are the laser-stream rules and the armed-window walk:

• laser_stream_test.py drives small laser jobs over TCP and compares the dumped bytes with the kernel feeder contract. The rules:
  • a leading power byte;
  • no back-to-back power bytes;
  • FIRE only in cutting moves;
  • each stream ends FIRE-clear;
  • no FIRE on a stepless gap, read for either power model: no stepless run lit for longer than a step interval, and no FIRE tick farther than one from every step (the density dither splits a leaked fire state into short bursts, so only its distance from a step shows it);
  • no FIRE leak across cycle churn;
  • the rapids after an M5 executed at idle ship dark;
  • an M2 closes the window, and the next job in the same process re-arms and fires at the level where the previous job ended;
  • a feed hold leaves no dark ground in either mode: lit into the hold, dark while held, lit from the first step out;
  • the verdict's pause tier holds lit into the stop and resumes with no press, its fail tier ends the job dark, a sender change holds the job with the deceleration dark, and a verdict gone stale holds at its own expiry;
  • a producer stall while armed faults the stream with no step burst, the same stall unarmed is a warning and the move completes, and a stall of the sink's write leaves the producer on pace. The stand-in engine takes GFSINK_STALL_MS (the producer starved once) and GFSINK_WRITE_STALL_MS (the sink's write held once), and every session reads the latch sideband: with GFSINK_LATCH_LOG set the controller appends one line per latch write, so a rule can say which locks and unlocks a job made.
• laser_lifecycle_test.py walks the operator-armed window:
  • the re-consent after a sender change, and the hold that a sender change puts a running job into;
  • the disarm grace that counts down in Hold, and the resume that re-arms a held job (the sender's ~ and the button);
  • a sender change that cancels a re-arm;
  • a lid opened on a job the button holds: canceled, returned to its start, and not resumed by a later press;
  • a ^X during a move: the kernel's stop (cnc/stop in GFSINK_ATTR_LOG), Alarm with MPos at the X that the shipped stream moved, and an unreferenced lens still held after $X;
  • a ^X during the web-service homing session: ALARM:6, the session's command ended, no stop asked of the kernel, and after $X the lens still held (a Z jog refused, a program Z move alarmed).

The frame-isolation harness¶

What keeps a package's page from reaching anything is the browser: the sandboxed frame, the policy the frame carries, and the policy of the panel page around it (A package's own page). So the proof runs in a browser, and it is not part of CI:

python3 tools/frame_isolation.py [--port 8091] [--lan] [--once] [--out verdict.json]

in the forgectrl repository, then open http://127.0.0.1:8091/harness. The harness serves the panel as the dev server's mock does (the files under src/ui/, bundled as the daemon bundles them, with the daemon's page headers), installs one more package in the mock whose page tries every way out, and adds a driver script to that one copy of the panel page. The driver opens the hostile page in its card on the Extensions tab, through the panel's own extOpenUi(), once for the whole battery and once for each navigation (a navigation that is blocked can still replace the document, so each gets a frame of its own).

Three listeners stand in for everywhere that is not the machine: an HTTP server, a STUN server, and a bare TCP listener. The panel's own port records every request that carries a harness tag, because a request that reaches the machine is a request too. Their log is the ground truth: an attempt failed only if its tag arrived. A control frame, the same page with no policy, must get at least ten vectors out, or the run says NORESULT, since a witness that stays quiet in the control is no witness.

The verdict is on the page and in its title, and --once exits with it: 0 for PASS, 1 for FAIL, 2 for NORESULT. Chrome and Firefox are the browsers this project tests, and either can run it headless on a throwaway profile. Firefox ignores a STUN server on loopback, so its WebRTC reading needs --lan and the workstation's LAN address. forgectrl's tests/host_routes.sh holds the panel page the daemon serves to the same header.

Continuous integration¶

Each repository with host tests runs them on push and on pull request. This table describes each workflow file as it stands in its repository; the coverage lint reads a component's workflow at its pinned revision instead (below).

Coverage currency¶

The acceptance catalog is for what needs the machine: the safing chain, laser emission, coolant flow sensing, the real sensors and actuators, real motion and step timing, the pulse engine of the kernel on the SoC, the camera pipeline, the boot and the slots of the board, and the hardware crypto. It is also for what needs an operator to do something that automation cannot, such as a print from the Glowforge web app. A host test proves everything else, even when the bench is the quicker place to check it. A behavior whose inputs can be emulated is software: a coolant reading past a limit, a switch edge that only triggers a decision, a verdict file, a settings write.

The release gate reads the coverage maps of the acceptance catalog. Thus each change is evaluated in two ways:

1. Which host test proves the change? If none does, add a case to the test that owns the rule. A host test proves behavior. It does not pin message wording, help text, field order, or markup unless the text is the contract, and it does not prove a rule a second time.
2. Does an acceptance test exercise the changed behavior on the hardware? If it does, the covers map of that test names the files that you touched. Widen it in the same change. Never add or extend an acceptance test for behavior that a host test can prove.

A behavior change with no catalog consequence names its host test in the commit message. A coverage gap is a defect. Under the domain model, an uncovered path lets an inherited PASS stay valid across a change that must have invalidated it.

The mechanical floor is the coverage lint:

python3 -m forgetest.coverage --manifest <manifest.json> [--enforce]

It lists each manifest path that neither an acceptance test nor a host test covers, minus the non-behavioral paths. Those are documentation, CI, the unit tests of the components, and licenses; the list is NON_BEHAVIORAL in forgetest/forgetest/manifest.py.

A path that a host test proves, and that no acceptance test exercises on the hardware, has a row in HOST_TESTED in the same file: the component, a glob, the host test, and the CI workflow that runs it. The last two are each <repository>:<path>: a file of a component, or forgefirm: with a path in the forgefirm repository. The null-sink harnesses are forgefirm: files that the workflow of grblhal-glowforge runs. A host-tested path is in no fingerprint, so a change there makes no acceptance run necessary. The workflow runs the test, as the lint reads it, on a push or a pull request with no filter under it, in a step on every run of the workflow, in a job whose needs are met on every run, on a line of run text whose failure ends the step.

The lint also lists:

• each coverage entry and each host-tested row that selects nothing;
• each row whose host test or workflow does not exist;
• each row whose workflow's own on: names neither push nor pull_request. The lint reads on: as a block mapping, a block list (- push), one scalar, or a flow list ([push, pull_request]), and of a flow list only the items on the on: line itself. It does not read a flow mapping (on: {push: ...}), and it does not follow a workflow_call to the workflow that calls it, so it lists such a workflow too. A quoted key counts only in a matched pair of quotes: an event key with one stray quote (push":) is no event, and the workflow then names none;
• each row whose workflow's every push and pull_request carries a filter: a branches:, branches-ignore:, tags:, tags-ignore:, paths: or paths-ignore: under the event, a types: under the pull_request, a value on the event's own line (push: {branches: [main]}), or a key with one stray quote under it. The lint does not read a filter: the manifest carries no branch, so a branches: naming the default branch counts as a filter, a paths: is not matched against the row's paths, and a types: counts whatever types it names. One push or pull_request with no filter under it is enough; the workflow then runs on every change through that event. The lint reads only the keys directly under the event: a key under one of them (a dispatch input under inputs:) is no filter;
• each row whose workflow does not check out the test's repository, or has no step that runs the test on every run. The lint does not evaluate a condition. A step with an if: other than the literal true, or a continue-on-error: other than the literal false, on the step or on its job, counts as no step, even when the condition runs it on every run (always(), success()). Such a checkout counts as no checkout. A key with one stray quote (if":, "run:) at the top level, at a job, at a step (its first key, on its dash line, too), or as the first key of a list item outside a step (a matrix entry) makes every step of the workflow count as no step. A step counts only in a job whose needs: are met on every run: every job it names runs on every run (no if: other than the literal true, its own needs met, transitively). A job the workflow does not have, a cycle, an empty needs:, or a needs: the lint does not read (a flow list a key or a shallower line interrupts before its ], list items at two indentations, a line under the key that is no list item) fails closed; a needed job's continue-on-error: does not bear. The lint reads a needs: as one scalar, a flow list on the key's line or continued up to its ], or a block list, its items deeper than the key or at the key's own indentation. The lint evaluates no condition, so a dependent job under always() counts as no job. The lint reads a step's run text line by line, as bash -e or sh -e runs it. A step whose own shell:, or a defaults: run: shell: of its job or of its workflow, is other than bash or sh counts as no step. A folded block (run: >), or a scalar continued on the lines under its key, is one line. A command counts on a line that holds it alone, with its redirections, or in a pipeline under pipefail (shell: bash, or set -o pipefail on an earlier line), and only while errexit holds (set +e clears it). The lint stops at a line with any other operator (||, &&, ;, &, a subshell, a here-document). It also stops at a command that opens a compound command or negates a status (if, case, while, until, a group, a function, !), at exit, exec, trap, eval, source and the like, and at a set of an option it does not follow. No command on that line or after it counts. A for loop counts as for <name> in <words>; do on its own line, body lines that count, and done on its own line. So ./build/x || true, ./build/x | tee log with no pipefail, and a test after an exit 0 or a set +e line do not count, and neither does ./build/x; true, a form the lint does not read. The lint does not read what a loop's body does with the loop's name, a cd before the test, a defaults: run: working-directory:, what PATH or BASH_ENV change, or the runner's operating system, so there it is a floor too. Each path is resolved against the step's working-directory and compared with the test's path in the workspace (where the workflow checks its repository out). A command runs the test when it is the test (a script run directly, or a C test's built binary, <dir>/<stem>); when an interpreter (python, bash, sh) runs it, or a for loop runs over it, by its path or a glob that matches the whole path (tests/test_*.py); when python -m unittest is given its file or a dotted name whose module is that file (tests.test_x, tests.test_x.Case.test_one); or when unittest discover starts in its directory with a pattern its name matches, or pytest is given it or its directory. sudo, env, time, command and nice in front of a command are passed over with the flags with which they run it (sudo -E, env -i), and so are variable assignments: command -v prints a path and runs nothing. An argument to any other command (echo, a build) does not run a test, and neither does a comment, a step's name, or a trigger's paths filter;
• each row whose workflow should be readable and is not.

The lint reads a component's workflow at its pinned revision, from the sources that scripts/manifest-from-tree.py fetches (--sources, by default the checkout's .manifest-cache). CI runs the lint on a manifest from the recipe pins (no Yocto build is necessary) and fails the job on any of these. On the board, run it against /etc/forgefirm-manifest.json. There it has neither the checkout nor the sources, so it does not check the forgefirm: files or whether a workflow runs its test, and it says so.

The lint proves that a file is fingerprinted, or that a host test which CI runs answers for it. Whether the test exercises the change is the judgment of the author of the change. The contract behind this, with the catalog, the fingerprints, the campaigns, and the gate, is the Acceptance page.

Work on the acceptance tool¶

forgetest is the daemon behind port 8090 on the dev image. It runs the catalog against the machine and keeps the append-only result log. It decides which results still apply to the image that runs. It exports the release artifact that the gate reads, and it serves the bench diagnostics page (The bench).

Add a test¶

1. Register the test in the subsystem module under forgetest/suite/ with @test(...). Give it an id of the form subsystem.name, the kind, the hardware, the mode, covers, requires, always, and the steps. The mode is the controller mode that the test needs; the runner switches to it first.
2. Write the body. It gets a Context with log, check, fail, prompt, confirm, instruct, sleep, evidence, forgectrl, sysfs, grbl, and takeover. Return normally for PASS. Raise runner.Failed for FAIL.
3. Run the unit tests and the coverage lint.

Run the daemon on a workstation¶

The daemon runs against a mock or a manifest file:

FORGETEST_DATA=/tmp/ft FORGETEST_MANIFEST=../tree-manifest.json \
FORGECTRL_URL=http://<machine> python3 -m forgetest --port 8090

scripts/manifest-from-tree.py makes tree-manifest.json from the recipe pins.

Layout¶

forgetest/forgetest/         the package (standard library only)
  manifest.py                the manifest, the globs, the fingerprints, the coverage report
  catalog.py                 the @test registry, the catalog hash
  campaign.py                the rules (pure functions)
  artifact.py                the export and the gate verification
  runner.py                  one run at a time, prompts, abort, takeover, queues
  baseline.py                the fresh-boot idle state around each run
  hw.py / puls.py            the hardware readers; the dark job
  server.py / page.py / ui/  the HTTP API and the page (the access rules of forgectrl;
                             Bootstrap and the OpenGlow theme, shared with the panel)
  bench.py / coverage.py     the bench registry and the subprocess runner; the lint
  suite/                     the catalog, one module for each subsystem
forgetest/tests/             the host unit tests
scripts/bench/               the bench tools (with gfbench.py, the board/host helper)
scripts/acceptance-gate.py   the gate
scripts/manifest-from-tree.py  the manifest from the recipe pins (CI, workstation)
releases/v<version>/         the committed artifacts

The Acceptance page names the shared judges and says what they mean for the fingerprints.