Quick start: after Installation, run the Demo — it works immediately with no data setup. config.example.yaml, covered later under Configuration, is a template for your own findings/artifacts and will not run until you edit it.

Installation

Set the API key:

For an organization LiteLLM proxy, use config.litellm.example.yaml instead — it's commented inline. Credentials and custom header values are read from environment variables and are never stored in SAFE configuration or result files.

Demo

demo/ contains three small, fully synthetic example artifacts — none derived from or corresponding to any real published research artifact — one per taxonomy label, so reviewers can exercise the full pipeline without any external data:

• demo-contextual-risk/ — a toy federated-learning checkpoint aggregator that deserializes a checkpoint downloaded from a caller-supplied URL with torch.load. Untrusted, network-sourced input reaches an unsafe deserialization sink, which SAFE is expected to classify CONTEXTUAL_RISK.
• demo-hardening-recommendation/ — a toy benchmark harness that runs subprocess.run(..., shell=True) against command lines that are all hardcoded Python literals, with no caller-controlled input. SAFE is expected to classify this HARDENING_RECOMMENDATION: the shell pattern is real and worth flagging, but nothing external can reach or influence it.
• demo-false-positive/ — a test-fixture generator pinned to an older Pillow version with a hypothetical decompression-bomb advisory. The code only creates new in-memory images and never opens external data, so the advisory's actual code path is never reached. SAFE is expected to classify this FALSE_POSITIVE.

demo/findings.csv holds one finding per artifact, and demo/demo-zero-shot.yaml / demo/demo-agentic.yaml are ready-to-run configs (artifact_root: . resolves relative to the config file, so run from inside demo/):

Results land in demo/runs/demo-zero-shot/ and demo/runs/demo-agentic/ respectively (see Output).

Three labels

• CONTEXTUAL_RISK
• HARDENING_RECOMMENDATION
• FALSE_POSITIVE

No additional category and no deterministic label-changing rule is used. A documented, isolated research/security mechanism in an artifact's own code is classified HARDENING_RECOMMENDATION, since the underlying practice is still real even when isolation limits realistic exploitability.

Binary classes

• SECURITY_RELEVANT: a valid contextual risk or hardening concern, including intentional, isolated security-research behavior.
• NON_SECURITY: a false, mismatched, non-applicable, absent, or demonstrably unused affected-feature finding.

The evaluator also derives a binary view from multi-class predictions: FALSE_POSITIVE becomes NON_SECURITY; every other multi-class label becomes SECURITY_RELEVANT. Direct and derived binary results remain explicitly separate.

Input structure

The mapping is exact: artifact_id = artifact_001 resolves to artifacts/artifact_001/.

Required CSV columns:

Optional columns:

An initial unnamed index column is ignored. Additional columns are preserved by the input model.

Example:

Generate findings from a new codebase

scripts/run_scanners.py and scripts/build_findings_csv.py produce the findings.csv and artifact layout described above directly from your own code, using Semgrep and Trivy.

Install Semgrep (works the same on any OS, including Linux):

Install Trivy on Linux — either the apt repository (Debian/Ubuntu):

or the official install script, which works on any Linux distribution and installs a binary release into /usr/local/bin (no root packages required beyond sudo for that directory):

Verify both are on PATH before continuing:

Then lay out one directory per artifact under an artifact_root/ and run:

The first command runs Semgrep and Trivy (vulnerability and secret scanning) against each artifact directory and saves the raw scanner JSON. The second parses that JSON into a SAFE-compatible findings.csv (columns match Input structure; file is reported relative to each artifact directory). Pass --skip-semgrep/--skip-trivy to either script to run only one tool. --config on run_scanners.py pins a specific Semgrep ruleset instead of the default auto, which is convenient but not reproducibly pinned.

Configuration

This section is for running SAFE against your own findings CSV and artifact folders (see Input structure above). If you just want to see SAFE run, use Demo instead — config.example.yaml below is a template and will not run as-is.

Copy config.example.yaml:

Then edit input_csv and artifact_root (and optionally paper_root) to point at your own data before running.

Key settings:

• model / provider: exact OpenAI model identifier (or LiteLLM alias), and openai or litellm with proxy URL and credential environment-variable name.
• analysis_mode: zero_shot or agentic.
• classification_task: binary or multiclass; independent of analysis_mode.
• max_agent_steps: required only in an agentic configuration.
• max_workers / max_output_tokens / max_schema_retries: concurrency, per-response output ceiling, and model-call retry budget for schema-invalid responses.
• resume / resume_policy: incomplete retries failures, missing artifacts, and unattempted findings; failed_only retries only failures while retaining recorded successes.
• cost: optional live cost accounting and max_run_cost_usd termination.

The default model is gpt-5.6-sol. Change it explicitly if availability, cost, or latency requirements differ.

Run analysis

Or without installing the console command:

For a runnable matched comparison against the included synthetic data, see Demo (demo/demo-zero-shot.yaml and demo/demo-agentic.yaml). They differ only in analysis_mode and run_name. Zero-shot makes one model call over the base evidence. Agentic mode starts from the same evidence and may call bounded read-only repository tools before returning the same structured result.

Output

results.csv is intended for analysis. results.jsonl preserves the complete structured records. Evidence and raw model outputs support auditing and error analysis. Both are canonical: they contain only the latest record for each finding, while logs/result_attempts.jsonl is append-only and preserves every historical outcome.

On resume, SAFE first re-parses every failed finding's saved raw responses with the current strict parser; a uniquely valid classification is recovered without an API call. Only unrecoverable failures are scheduled for model inference. For a failed-only continuation of a partially completed run, keep the same output_root and run_name and set:

To evaluate predictions against a labeled gold CSV (with a security_label or security_class column):

Testing

The test suite uses a fake provider and therefore does not require an API key.