Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
7 changes: 7 additions & 0 deletions .github/ct.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
chart-dirs:
- deploy/helm
# The chart version comes from the git tag at publish time, not from a bump
# in each commit, so ct must not require a version increment here.
check-version-increment: false
validate-maintainers: false
target-branch: main
106 changes: 85 additions & 21 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@stable
with:
components: clippy, rustfmt
Expand All @@ -23,50 +23,56 @@ jobs:
# examples and the `bytecode_gating` integration test) is excluded from the
# default build, so lint it in a dedicated pass.
- run: cargo clippy -p exav-core --all-targets --features unstable-internals -- -D warnings
# The full native test matrix (default + http + checksums + no-decrypt +
# unstable-internals feature passes). Defined once in the Makefile so this
# and a local `make test` can't drift.
- run: cargo clippy -p exav-render --all-targets --features dwg -- -D warnings
- run: cargo clippy -p exav-render --all-targets --no-default-features --features ifc,stl -- -D warnings
# The full native test matrix, every feature pass. Defined once in the
# Makefile so this and a local `make test` can't drift.
- run: make test-native

# The promised floor is `rust-version` in Cargo.toml (1.91, imposed by
# yara-x-parser). `make msrv` reads it there and installs that toolchain on
# demand, so a bump flows through with no edit here — and a contributor on
# The promised floor is `rust-version` in Cargo.toml (1.92, imposed by the
# hayro image decoders). `make msrv` reads it there and installs that toolchain on
# demand, so a bump flows through with no edit here, and a contributor on
# stable can't land a newer feature without this going red.
msrv:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- run: make msrv

# The yara-x A/B differential harness. Deliberately NOT part of `make test`
# (see the Makefile) — a differential test can go red because the other engine
# (see the Makefile): a differential test can go red because the other engine
# changed, so it is not a merge gate. It runs here anyway, unlike the clamav
# harness, only because it is self-contained: no signature database, no second
# engine to install. Its own job so the optional `yara-x` dependency — 83 extra
# crates, 21 of them cranelift/wasmtime — never lands in the `test` job, and so
# it costs no wall clock.
# harness, only because it is self-contained: no signature database, just
# yara-x's `yr` binary. The tests skip when `yr` is missing, so it is installed
# and checked first; without it this job would pass having compared nothing.
yara-diff:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- name: Install yr (yara-x CLI)
run: |
YR=v1.21.0
curl -sSL "https://github.com/VirusTotal/yara-x/releases/download/$YR/yara-x-$YR-x86_64-unknown-linux-gnu.tar.gz" | tar xz -C /tmp
echo "/tmp" >> "$GITHUB_PATH"
- run: yr --version
- run: make test-yara-diff

wasm:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@stable
with:
targets: wasm32-wasip1, wasm32-unknown-unknown
- uses: Swatinem/rust-cache@v2
- name: Install binaryen (wasm-opt)
run: sudo apt-get update && sudo apt-get install -y binaryen
# The scanning core compiled to WASI, and the archive-extractor WASM
# bindings (excluded from the workspace, built on their own target) — both
# bindings (excluded from the workspace, built on their own target), both
# the default (all-formats + decrypt) build and a minimal single-format,
# no-decrypt build, so the feature-gated paths can't silently rot.
- run: cargo build -p exav-core --features wasi-bin --target wasm32-wasip1
Expand Down Expand Up @@ -97,11 +103,28 @@ jobs:
with:
node-version: 22
- run: make test-js
# @exav/viewer: its wasm modules and its archive plugin's peer
# (exav-unpack-wasm's pkg/, which test-js built with test-only features)
# are rebuilt from source, with the wasm-bindgen-cli its Cargo.toml pins,
# and its demo is driven in Chromium, its sandboxed frame in Chromium,
# Firefox and WebKit. See scripts/test-viewer.sh.
- uses: taiki-e/install-action@v2
with:
tool: wasm-bindgen@0.2.129
- run: make test-viewer
# What the browser saw, for the failures that do not happen on a
# developer's machine.
- uses: actions/upload-artifact@v4
if: failure()
with:
name: viewer-e2e-results
path: crates/exav-viewer/test-results
if-no-files-found: ignore

docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v7
- uses: actions/setup-node@v4
with:
node-version: 22
Expand All @@ -116,7 +139,7 @@ jobs:
audit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@stable
- name: cargo-audit
run: |
Expand All @@ -137,16 +160,57 @@ jobs:
run: |
cargo deny --manifest-path crates/exav-unpack-wasm/Cargo.toml \
--config deny.toml check advisories bans sources licenses
# The viewer's wasm modules, also its own workspace. Every feature: the
# DWG module is not a default.
- name: cargo-deny (viewer)
run: |
cargo deny --manifest-path crates/exav-viewer/Cargo.toml --all-features \
--config deny.toml check advisories bans sources licenses

fuzz-smoke:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@nightly
- uses: Swatinem/rust-cache@v2
with:
workspaces: |
. -> target
fuzz -> target
# What the targets found interesting last time, so a 60 s run starts
# where the last one stopped. A new key each run, the newest restored.
- uses: actions/cache@v4
with:
path: fuzz/corpus
key: fuzz-corpus-${{ github.run_id }}
restore-keys: fuzz-corpus-
- name: The fuzz crate's own tests (the mutator and the input framing)
run: cargo test --manifest-path fuzz/Cargo.toml -p exav-fuzz-support
- name: cargo-fuzz smoke run
run: |
cargo install cargo-fuzz --locked
for t in filetype sigs cvd pe unpack analyze bytecode full_pipeline ndb_compile rar3_ppmd; do
cargo +nightly fuzz run "$t" -- -max_total_time=60 -rss_limit_mb=2048 -timeout=10
# Seeded from the test fixtures where they match a target's input:
# random bytes rarely get past a header. Outside the checkout, as
# the masked fixtures are unmasked.
seeds="$RUNNER_TEMP/fuzz-seeds"
scripts/fuzz-seeds.sh "$seeds"
for t in filetype sigs cvd pe unpack analyze bytecode full_pipeline seekable yara pcre clamd icap volumes authenticode database ndb_compile rar3_ppmd pe_emulator x86_decode parser_recursion imagehash drawing cad_dxf cad_dwg ifc; do
mkdir -p "fuzz/corpus/$t"
extra=""
[ -d "$seeds/$t" ] && extra="$seeds/$t"
# Strings for the targets whose input is text or a protocol.
dict=""
case "$t" in
yara|pcre|icap|clamd) dict="-dict=fuzz/dicts/$t.dict" ;;
full_pipeline|ndb_compile|sigs|cvd) dict="-dict=fuzz/dicts/signatures.dict" ;;
esac
# iced-x86, the oracle x86_decode compares against, builds its
# decoder tables in a `lazy_static` it never frees: the newest
# nightly's LeakSanitizer reports them at exit (552 bytes, whatever
# the input), which is not a leak of exav's. The other targets keep
# the check.
asan=""
[ "$t" = x86_decode ] && asan="detect_leaks=0"
ASAN_OPTIONS="$asan" cargo +nightly fuzz run "$t" "fuzz/corpus/$t" $extra \
-- $dict -max_total_time=60 -rss_limit_mb=2048 -timeout=10
done
123 changes: 119 additions & 4 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
# binaries natively on amd64 (via `cross`, NOT under QEMU) and `publish`
# assembles the multi-arch image around them with `COPY` — compiling the
# Rust+ring tree emulated for arm64 is what used to make publishing slow.
# `publish` then re-checks the pushed tag, because the image must carry Docker
# (not OCI) mediatypes for podman to honour its HEALTHCHECK; see that step.
# Pull with:
# docker run -d -p 3310:3310 -v exav-db:/var/lib/exav ghcr.io/sylvinus/exav
name: docker
Expand All @@ -23,7 +25,7 @@ jobs:
build-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v7
- uses: docker/setup-buildx-action@v3
- name: Build image (amd64, load into docker)
uses: docker/build-push-action@v6
Expand All @@ -46,6 +48,12 @@ jobs:
exit 1
fi
docker logs exav
# The image's HEALTHCHECK is on the container, and its command works.
# Not `docker inspect .State.Health`: the first probe is 30s away.
HC=$(docker inspect --format '{{json .Config.Healthcheck.Test}}' exav)
echo "healthcheck: $HC"
[ "$HC" = '["CMD","/exav","--ping"]' ]
docker exec exav /exav --ping
docker rm -f exav

# Static musl binaries for the published image, cross-compiled natively on
Expand All @@ -64,7 +72,7 @@ jobs:
arch: arm64
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@stable
- name: Install cross
run: cargo install cross --locked
Expand Down Expand Up @@ -97,7 +105,7 @@ jobs:
contents: read
packages: write # push to ghcr.io
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v7
# No QEMU anywhere in this job: each platform build only COPYs its
# prebuilt binary (plus mkdir/chown, pinned to the build machine in the
# Dockerfile), so there is nothing to emulate.
Expand Down Expand Up @@ -125,9 +133,116 @@ jobs:
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
# Take bins/exav-$TARGETARCH from the `binaries` job instead of
# compiling in-image (Dockerfile documents both branches).
build-args: BIN_SOURCE=prebuilt
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
# Publish Docker manifest-list/schema2 mediatypes rather than OCI
# ones, so that podman honours the image's HEALTHCHECK.
#
# `Healthcheck` is a Docker config extension with no equivalent in
# the OCI image spec. Nothing strips it: flipping this exporter
# option leaves the config blob byte-identical (same sha256), and
# only the declared mediatype changes. But podman reads the field
# back solely from a schema2 config — containers/common fills
# `HealthCheck` in the `DockerV2Schema2MediaType` branch of
# libimage/inspect.go and in no other branch — so under OCI
# mediatypes it reports "has no defined healthcheck" however healthy
# the daemon actually is. That leaves a rootless podman deployment
# with no readiness signal at all: the image is distroless, so a
# compose-level `test:` has no shell to run in either. Docker reads
# the field from both configs, so this is a no-op for `docker run`.
#
# `provenance: false` is load-bearing, not tidying: attestations can
# only be referenced from an OCI index, so buildx's default
# provenance attestation would force OCI mediatypes straight back.
# Attaching SLSA provenance as a separate referrer instead (e.g.
# actions/attest-build-provenance with push-to-registry) leaves the
# image's own mediatype alone.
#
# `push` moves into `outputs` so there is a single image exporter
# making a single mediatype decision.
provenance: false
outputs: type=image,oci-mediatypes=false,push=true

# The `build-test` job above smoke-tests a locally `load`ed image, which
# cannot catch a *push-time* mediatype change — the exporter is what
# differs. These two steps therefore run against the pushed tag.
- name: Assert the published manifest is a Docker manifest list
env:
# Via env: rather than inlined into `run:`, so a tag reaches bash as
# data and not as code.
TAGS: ${{ steps.meta.outputs.tags }}
EXPECTED: application/vnd.docker.distribution.manifest.list.v2+json
run: |
set -euo pipefail
# The tags are already public when this runs: a failure here is an
# alarm to fix and re-publish, not a gate.
bad=0
while read -r TAG; do
[ -n "$TAG" ] || continue
GOT=$(docker buildx imagetools inspect "$TAG" --raw | jq -r '.mediaType')
echo "$TAG -> $GOT"
if [ "$GOT" != "$EXPECTED" ]; then
echo "::error::$TAG is already published as $GOT, expected $EXPECTED -- podman will ignore the image HEALTHCHECK"
bad=1
fi
done <<<"$TAGS"
exit "$bad"

# The assertion above pins the wire format; this one tests the consumer
# that the format exists for. podman is preinstalled on ubuntu-latest.
- name: Verify podman sees the HEALTHCHECK on the published image
env:
TAGS: ${{ steps.meta.outputs.tags }}
REGISTRY_USER: ${{ github.actor }}
REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
run: |
set -euo pipefail
TAG=$(printf '%s\n' "$TAGS" | head -1)
podman --version
# Log in rather than relying on an anonymous pull: a GHCR package
# starts out private, so the very first publish of a fork would
# otherwise fail here for the wrong reason.
podman login ghcr.io -u "$REGISTRY_USER" --password-stdin <<<"$REGISTRY_PASSWORD"
podman pull -q "$TAG"

# This coming back empty *is* the regression being guarded against:
# podman populates it only from a Docker schema2 config.
IMAGE_HC=$(podman image inspect \
--format '{{if .HealthCheck}}{{.HealthCheck.Test}}{{end}}' "$TAG")
echo "image healthcheck: ${IMAGE_HC:-<absent>}"
if [ -z "$IMAGE_HC" ]; then
echo "::error::podman reports no healthcheck on $TAG"
exit 1
fi

# Same env as the `build-test` smoke test: no mirror configured, so
# opt into the built-in EICAR-only baseline and skip the wait for a
# sidecar that is not coming.
podman run -d --name exav-podman \
-e EXAV_STARTUP_WAIT_SECS=0 -e EXAV_ALLOW_NO_DB=1 "$TAG"
CONTAINER_HC=$(podman inspect \
--format '{{if .Config.Healthcheck}}{{.Config.Healthcheck.Test}}{{end}}' exav-podman)
echo "container healthcheck: ${CONTAINER_HC:-<absent>}"
if [ -z "$CONTAINER_HC" ]; then
echo "::error::healthcheck did not reach the container config"
podman rm -f exav-podman
exit 1
fi

# `/exav --ping` answers only once the database is loaded, so poll
# instead of asserting on the first probe.
for _ in $(seq 60); do
if podman healthcheck run exav-podman; then
echo "podman reports: $(podman inspect --format '{{.State.Health.Status}}' exav-podman)"
podman rm -f exav-podman
exit 0
fi
sleep 1
done
echo "::error::podman healthcheck never passed"
podman logs exav-podman
podman rm -f exav-podman
exit 1
Loading
Loading