SAST False Positive Reduction Prototype

This repo is a small research sandbox for reducing false positives in SAST by running the default Semgrep rules on PyGoat and then applying tree-sitter-based heuristics before scoring against a labeled PyGoat manifest. SAST assets live under sast/, SCA lives under sca/, and the shared benchmark project lives under benchmarks/.

How it works

benchmarks/pygoat
        |
        v
  Semgrep scan (default rules)
        |
        v
  sast/results/semgrep-*.json (baseline)
        |
        v
sast/tools/filter_semgrep.py
        |
        v
  sast/results/semgrep-*-filtered.json
        |
        v
sast/tools/evaluate_semgrep.py -> precision/recall/FP-rate (PyGoat manifest)

The filtering step uses tree-sitter parsing to drop findings that are clearly safe (sanitized shell commands, parameterized SQL, SafeLoader YAML, allowlisted SSRF, sanitized paths).

Generated scan outputs live under sast/results/ and sca/results/. The SAST wrapper caches the Semgrep auto config at sast/rules/semgrep-auto.yml so default runs work without re-downloading rules.

Dependencies

• Semgrep (run via .venv)
• tree_sitter + tree_sitter_python for parsing in sast/tools/filter_semgrep.py
• Tooling is fixed to Semgrep (default rules) and pip-audit for this prototype.

Setup

python -m venv .venv
.venv/bin/pip install -r requirements.txt

Quick start (SAST with default Semgrep rules)

sast/tools/run_semgrep.sh --config auto --json --output sast/results/semgrep-pygoat-default.json --no-git-ignore benchmarks/pygoat
.venv/bin/python sast/tools/filter_semgrep.py sast/results/semgrep-pygoat-default.json sast/results/semgrep-pygoat-default-filtered.json
.venv/bin/python sast/tools/evaluate_semgrep.py sast/results/semgrep-pygoat-default.json --scope benchmarks/pygoat
.venv/bin/python sast/tools/evaluate_semgrep.py sast/results/semgrep-pygoat-default-filtered.json --scope benchmarks/pygoat
.venv/bin/python sast/tools/compare_metrics.py sast/results/semgrep-pygoat-default.json sast/results/semgrep-pygoat-default-filtered.json --scope benchmarks/pygoat

Run the same flow on the Pretix codebase (real-world Django app):

sast/tools/run_semgrep.sh --config auto --json --output sast/results/semgrep-pretix-default.json --no-git-ignore --include '*.py' benchmarks/pretix/src/pretix
.venv/bin/python sast/tools/filter_semgrep.py sast/results/semgrep-pretix-default.json sast/results/semgrep-pretix-default-filtered.json
.venv/bin/python sast/tools/evaluate_semgrep.py sast/results/semgrep-pretix-default.json --scope benchmarks/pretix
.venv/bin/python sast/tools/evaluate_semgrep.py sast/results/semgrep-pretix-default-filtered.json --scope benchmarks/pretix
.venv/bin/python sast/tools/compare_metrics.py sast/results/semgrep-pretix-default.json sast/results/semgrep-pretix-default-filtered.json --scope benchmarks/pretix

Quick start (SCA with pip-audit)

sca/tools/run_pip_audit.sh -r benchmarks/pygoat/requirements.txt -f json -o sca/results/sca-baseline.json
.venv/bin/python sca/tools/collect_python_imports.py --root benchmarks/pygoat --output sca/results/sca-imports.json
.venv/bin/python sca/tools/filter_pip_audit.py sca/results/sca-baseline.json sca/results/sca-filtered.json --imports sca/results/sca-imports.json --import-map sca/tools/sca_import_map.json
.venv/bin/python sca/tools/evaluate_pip_audit.py sca/results/sca-filtered.json --manifest sca/dataset/sca_manifest.json
.venv/bin/python sca/tools/compare_pip_audit.py sca/results/sca-baseline.json sca/results/sca-filtered.json --manifest sca/dataset/sca_manifest.json

Pretix:

sca/tools/run_pip_audit.sh benchmarks/pretix -f json -o sca/results/sca-pretix-baseline.json
.venv/bin/python sca/tools/collect_python_imports.py --root benchmarks/pretix/src/pretix --output sca/results/sca-pretix-imports.json
.venv/bin/python sca/tools/filter_pip_audit.py sca/results/sca-pretix-baseline.json sca/results/sca-pretix-filtered.json --imports sca/results/sca-pretix-imports.json --import-map sca/tools/sca_import_map.json
.venv/bin/python sca/tools/evaluate_pip_audit.py sca/results/sca-pretix-filtered.json --manifest sca/dataset/sca_manifest_pretix.json
.venv/bin/python sca/tools/compare_pip_audit.py sca/results/sca-pretix-baseline.json sca/results/sca-pretix-filtered.json --manifest sca/dataset/sca_manifest_pretix.json

sca/tools/run_pip_audit.sh bootstraps sca/.venv with a compatible Python runtime (default 3.10) using uv. Override with SCA_PYTHON_VERSION.

Configuration

sast/tools/filter_config.json controls:

• Which rules map to each heuristic category (rule IDs, CWE, keywords).
• Optional suppress category to drop out-of-scope rule IDs.
• The allowlist names/lookback windows for SSRF and path traversal.
• Safe YAML loaders and shell sanitizers.

Edit that file to extend coverage without touching the Python code.

sca/tools/sca_filter_config.json controls:

• Allowlisted packages and vulnerability IDs to suppress known false positives.

See sca/SCA.md for the full SCA workflow and notes.

CI

GitHub Actions workflow metrics reruns SAST/SCA scans for PyGoat and Pretix and prints metric comparisons on each push or manual dispatch.