Skip to content

Using TrustSight in CI

The default terminal output is the findings list, not a score: the score exists and is deterministic, but the tool leads with evidence. Automation therefore runs on the JSON contract below, the CLI machine-readable surface, and the one this guide uses throughout.

Exit codes

Code Meaning
0 The analysis completed. Says nothing about what was found.
2 The analysis could not run or could not complete.
130 The process was interrupted with Ctrl+C.

Do not gate on the exit code. A flagged package still exits 0; see exit codes. Gate on the JSON, which is the contract this guide uses throughout.

The JSON contract

trustsight review --json writes a list, one object per package, to stdout. Progress events go to stderr, so a pipeline only has to read stdout.

Every object carries the report fields below. The list can contain successful rows and rows with failed: true: a failed row is an explicit NOT VETTED result, not an omitted package.

Field Meaning
package Package name.
old_version, new_version Installed version and the version the AUR declares.
findings Each with rule_id, file, line, description.
verdict The rendered sentence.
first_seen true when there is no prior history for this package.
is_trivial true when only pkgver and checksums moved.
coverage_gaps What the run could not examine. Always present.
failed true when this package could not be vetted. Do not treat an empty findings list on such a row as clean.
required_by Under --deps, the packages that declare this one. Empty otherwise, and always present, so a gate never has to test for the key.

score, risk and risk_label are added when --score or --risk is passed. risk is the bare band; risk_label is the same band qualified when the run was incomplete, for tools that display it to a person.

A minimal CI step:

trustsight review --score --json > report.json
python3 - <<'EOF'
import json, sys

reports = json.load(open("report.json"))
flagged = [r for r in reports if (r.get("score") or 0) > 20]
partial = [r for r in reports if r.get("coverage_gaps")]

for r in flagged:
    print(f"HIGH/CRITICAL {r['package']}: {r['verdict']}")
for r in partial:
    print(f"NOT FULLY VETTED {r['package']}: {', '.join(r['coverage_gaps'])}")

sys.exit(1 if flagged or partial else 0)
EOF

Treat coverage_gaps as blocking, not informational. A non-empty list means the score describes part of the change, not all of it. Two separate bypasses ride on ignoring it:

  • Pad a diff past the size cap, append the payload, and the visible score drops.
  • Do the same but include one cheap deliberate HIGH in the visible prefix. The score does not drop, risk reads "High", and a pipeline gating on risk alone sees a plausible verdict computed from a fraction of the change.

Gating on risk without coverage_gaps closes the first and leaves the second. See the security model.

Policy gating

Decouple the score from your pass/fail decision. TrustSight's verdict threshold (20) is a sensible default but your team's tolerance may differ.

export THRESHOLD=40
trustsight review --score --json > report.json
python3 - <<'EOF'
import json, os, sys

threshold = int(os.environ.get("THRESHOLD", "20"))
reports = json.load(open("report.json"))
over = [r["package"] for r in reports if (r.get("score") or 0) > threshold]
if over:
    print("Packages above threshold:", ", ".join(over))
    sys.exit(1)
EOF

Raising the threshold does not raise the coverage bar: coverage_gaps is independent of the score and should be checked whatever threshold you pick.

Deeper output

trustsight inspect <pkg> --json --score --verbose exposes a detailed report body for one package: the score breakdown, bucket classifications, novelty context and evidence. Field by field, it is documented in the report schema.

Per-class CI regression

For teams that want a statistical gate, TrustSight publishes benchmark distributions for each severity class on the benchmarks page:

Class Metric Value
CRITICAL p5 (5th percentile) 60
Benign p95 (95th percentile) 30
Zero-rate (benign scored 0) percentage 72.3%
Test count total Run uv run --with pytest python -m pytest for the current checkout

The gate: the published figures separate the two tails by 30 points: benign p95 = 30, CRITICAL p5 = 60, measured on the locked 3,739-diff corpus. The calibration gates re-measure both percentiles on every push and fail the build if the benign p95 stops being strictly below the malicious p5 (the separation gate). This is a claim about two percentiles, not about the distributions as a whole: 7.1% of benign diffs score above the 20-point threshold and some score above 30, which is why the operational threshold is 20 and not 30.

To set up your own gate:

  1. Run a baseline against your package set after initial configuration: trustsight review --record, because runs are read-only by default and only --record writes the history later runs compare against. See the re-baselining guide.
  2. Choose a threshold: typically 30-40, depending on your tolerance for benign novelty signals.
  3. Add a CI check that compares regression scores against the baseline. Any package whose score moves from Low to Medium or higher without a corresponding PKGBUILD change is a regression.

The CRITICAL recall of 100% means every labelled malice sample in the corpus fires the rules it is labelled for. A gate at 40 catches all known CRITICAL patterns and passes benign bumps that score below the 95th percentile.

Nightly vs per-commit

  • Per-commit: run trustsight review on every PR that touches a PKGBUILD or a rules.toml change. Gate on the JSON, as above.
  • Nightly: run a full review of all installed AUR packages and diff the output against the previous night. Detects drift over time.

Config in CI

Provision ~/.config/trustsight in CI, for example by setting HOME to a controlled workspace-local home. Check in the configuration files and initialize the database there before running trustsight review.

See also: