Test¶

A feature or fix is not complete until it is proven. The order of preference for proof:

1. A host test that runs in CI.
2. A bench drill on the bench reference, recorded in the commit message that carries the change.
3. Documented reasoning.

A change that can put energy where it was not commanded gets a regression test with the feature or fix, not after it (Contribute). This page is the inventory. It lists the host tests in each repository, the CI workflows, and the tests of the acceptance tool. It also gives the coverage rule that connects a code change to the release gate.

Host tests, for each repository¶

The null-sink controller¶

On a host without GFSINK, the grblHAL driver runs the real core, the planner, and the stream code against a null sink instead of /dev/glowforge. With GFSINK_DUMP set, it writes each pulse byte that it would ship to a file. This makes the laser path testable without hardware. Two harnesses in forgefirm/scripts/bench/ use it, and both run in the CI of the driver:

• laser_stream_test.py drives small laser jobs over TCP and compares the dumped bytes with the kernel feeder contract. The rules:
  • a leading power byte;
  • no back-to-back power bytes;
  • FIRE only in cutting moves;
  • each stream ends FIRE-clear;
  • no FIRE on a stepless gap;
  • no FIRE leak across cycle churn;
  • the rapids after an M5 executed at idle ship dark;
  • the next job in the same process fires at the level where the previous job ended;
  • a feed hold leaves no dark ground in either mode: lit into the hold, dark while held, lit from the first step out.
• laser_lifecycle_test.py walks the operator-armed window:
  • one arm for each job, with M5/M3 persistence;
  • the M2 close;
  • the re-consent after a sender change, and the hold that a sender change puts a running job into;
  • the disarm grace that counts down in Hold, and the resume that re-arms a held job (the sender's ~ and the button);
  • the arm refusal under a cooling verdict that blocks.

Continuous integration¶

Each repository with host tests runs them on push and on pull request.

Coverage currency¶

The release gate reads the coverage maps of the acceptance catalog. Thus each change is evaluated against the catalog, in addition to its unit tests:

1. Does an acceptance test exercise the changed behavior? If not, add one or extend one in the same change.
2. Does the covers map of that test name the files that you touched? If not, widen it in the same change.

Exercise a gate or a limit through the settings API. Set a value that a healthy machine cannot meet. The engine reads it again at the next run start, and the teardown of the test restores it. Never use the GFCOOL_* environment overrides for this. They need a daemon restart, and they stay bench-only.

A behavior change with no catalog consequence needs a sentence of justification in the commit message. A coverage gap is a defect. Under the domain model, an uncovered path lets an inherited PASS stay valid across a change that must have invalidated it.

The mechanical floor is the coverage lint:

python3 -m forgetest.coverage --manifest <manifest.json> [--enforce]

It lists each manifest path that no test covers, minus the non-behavioral paths. Those are documentation, CI, the unit tests of the components, and licenses; the list is NON_BEHAVIORAL in forgetest/forgetest/manifest.py. It also lists each coverage entry that selects nothing. CI runs the lint on a manifest from the recipe pins (scripts/manifest-from-tree.py; no Yocto build is necessary) and fails the job on an uncovered path. On the board, run it against /etc/forgefirm-manifest.json. The lint proves that a file is fingerprinted. Whether the test exercises the change is the judgment of the author of the change (rule 1). The contract behind this, with the catalog, the fingerprints, the campaigns, and the gate, is the Acceptance page.

Work on the acceptance tool¶

forgetest is the daemon behind port 8090 on the dev image. It runs the catalog against the machine and keeps the append-only result log. It decides which results still apply to the image that runs. It exports the release artifact that the gate reads, and it serves the bench diagnostics page (The bench).

Add a test¶

1. Register the test in the subsystem module under forgetest/suite/ with @test(...). Give it an id of the form subsystem.name, the kind, the hardware, the mode, covers, requires, always, and the steps. The mode is the controller mode that the test needs; the runner switches to it first.
2. Write the body. It gets a Context with log, check, fail, prompt, confirm, instruct, sleep, evidence, forgectrl, sysfs, grbl, and takeover. Return normally for PASS. Raise runner.Failed for FAIL.
3. Run the unit tests and the coverage lint.

Run the daemon on a workstation¶

The daemon runs against a mock or a manifest file:

FORGETEST_DATA=/tmp/ft FORGETEST_MANIFEST=../tree-manifest.json \
FORGECTRL_URL=http://<machine> python3 -m forgetest --port 8090

scripts/manifest-from-tree.py makes tree-manifest.json from the recipe pins.

Layout¶

forgetest/forgetest/         the package (standard library only)
  manifest.py                the manifest, the globs, the fingerprints, the coverage report
  catalog.py                 the @test registry, the catalog hash
  campaign.py                the rules (pure functions)
  artifact.py                the export and the gate verification
  runner.py                  one run at a time, prompts, abort, takeover, queues
  baseline.py                the fresh-boot idle state around each run
  server.py / page.py / ui/  the HTTP API and the page (the access rules of forgectrl;
                             Bootstrap and the OpenGlow theme, shared with the panel)
  bench.py / coverage.py     the bench registry and the subprocess runner; the lint
  suite/                     the catalog, one module for each subsystem
forgetest/tests/             the host unit tests
scripts/bench/               the bench tools (with gfbench.py, the board/host helper)
scripts/acceptance-gate.py   the gate
scripts/manifest-from-tree.py  the manifest from the recipe pins (CI, workstation)
releases/v<version>/         the committed artifacts