Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,8 @@ jobs:
# (#956). A PR that touches the sidecar runs the suite again against the locked dependency
# set, with a missing fastembed a failure rather than a skip. About 550 MB and a minute, and
# the embedding test downloads the 67 MB English model, so only when sidecar/ changed.
# Installed exactly as the app installs it, `--no-deps` included (sidecar.rs `provision`
# says why: the lock overrides fastembed's pillow cap).
- name: Sidecar unit tests (the locked dependencies, sidecar changes only)
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
Expand All @@ -219,7 +221,7 @@ jobs:
fi
python -m venv "$RUNNER_TEMP/sidecar-venv"
"$RUNNER_TEMP/sidecar-venv/bin/python" -m pip install --disable-pip-version-check \
--no-cache-dir --require-hashes -r sidecar/requirements.lock
--no-cache-dir --require-hashes --no-deps -r sidecar/requirements.lock
"$RUNNER_TEMP/sidecar-venv/bin/python" sidecar/test_pm_sidecar.py

# --- secrets: gitleaks over the working tree AND git history ------------
Expand Down
13 changes: 8 additions & 5 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -339,10 +339,13 @@ jobs:
# interpreter is a prebuilt binary, so the build runners still need no Python
# toolchain to compile.)
#
# Installs exactly as the app does — `--require-hashes -r sidecar/requirements.lock`
# — so this is also the proof that the universal lock's markers select correctly on
# all three platforms, and that every artifact each one needs is covered by a hash.
# It used to install requirements.txt, which left the shipped resolution untested.
# Installs exactly as the app does — `--require-hashes --no-deps -r
# sidecar/requirements.lock` — so this is also the proof that the universal lock's
# markers select correctly on all three platforms, and that every artifact each one
# needs is covered by a hash. It used to install requirements.txt, which left the
# shipped resolution untested. `--no-deps` because the lock overrides fastembed's
# pillow cap (sidecar.rs `provision` says why); it also means pip no longer reports a
# dependency the markers failed to select, so the import line is what catches one.
python-smoke:
needs: guards
strategy:
Expand Down Expand Up @@ -370,7 +373,7 @@ jobs:
source .smoke-venv/bin/activate
fi
python -m pip install --upgrade pip
pip install --require-hashes -r sidecar/requirements.lock
pip install --require-hashes --no-deps -r sidecar/requirements.lock
python -c "import markitdown, fastembed, faster_whisper; print('sidecar deps import OK')"

publish:
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ prior exposure.
## 3. Set up your environment

Full prerequisites are in the [README](README.md#prerequisites). In short: **Node 24+**,
**Rust (stable)**, **Python 3.10+** on your PATH, plus your platform's Tauri build
**Rust (stable)**, **Python 3.11+** on your PATH, plus your platform's Tauri build
prerequisites.

```bash
Expand Down
13 changes: 7 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,12 +128,13 @@ For a full tour of the architecture and the rules for working in this repo, see
## Prerequisites

- **Node.js** 24+ (LTS) and **Rust** (stable)
- **Python** 3.10+ — needed for development (`tauri dev`) and as the base for the
- **Python** 3.11+ — needed for development (`tauri dev`) and as the base for the
document sidecar's managed virtual environment, built on first use. PM finds a suitable
interpreter automatically: `PM_PYTHON`, then versioned names (`python3.12` … `python3.10`),
`python3`/`python`, and common macOS locations (Homebrew, python.org) — so even a
Finder-launched app finds one off its stripped PATH. It also rebuilds the venv if that
interpreter is older than 3.10. Set `PM_PYTHON` only for an unusual setup. (Packaged
interpreter automatically: `PM_PYTHON`, then versioned names newest-first (`python3.16` …
`python3.11`), `python3`/`python`, and common macOS locations (Homebrew, python.org) — so even
a Finder-launched app finds one off its stripped PATH. It also rebuilds the venv if that
interpreter is older than 3.11 (the floor was 3.10 until Pillow 13, which needs 3.11). Set
`PM_PYTHON` only for an unusual setup. (Packaged
**Windows and Linux** release builds **bundle** a standalone interpreter at build time —
`npm run tauri build` fetches it via [`scripts/fetch-python.mjs`](scripts/fetch-python.mjs)
— so end users need no Python install. On **macOS**, packaged apps instead **download** a
Expand All @@ -143,7 +144,7 @@ For a full tour of the architecture and the rules for working in this repo, see
build-time bundling into the signed `.app` is gated on the Apple-signing pipeline, whereas a
runtime download picks the right per-arch build and isn't gated on signing — see
[`docs/MACOS-SIGNING.md`](docs/MACOS-SIGNING.md). Development from source still needs Python
3.10+ on your machine on every platform.)
3.11+ on your machine on every platform.)
- **Tauri OS prerequisites:** see <https://tauri.app/start/prerequisites/>
- Windows: Microsoft C++ Build Tools + WebView2 (preinstalled on Win 11)
- macOS: Xcode Command Line Tools
Expand Down
9 changes: 8 additions & 1 deletion justfile
Original file line number Diff line number Diff line change
Expand Up @@ -136,8 +136,15 @@ deny:
# transitive ones included, on the base venv AND both optional components (L-6). It used to scan
# requirements.txt, i.e. the six top-level pins only, so a CVE in (say) pdfminer.six or lxml — both
# of which parse untrusted input — was never looked at.
#
# `--disable-pip`: audit the pinned versions as written instead of pip-installing the locks into a
# throwaway venv first. pip-audit allows it only for hashed requirements, which every lock line is,
# and it does not resolve anything, so it audits exactly the lock. Installing them was also how the
# audit broke: the base lock overrides fastembed's `pillow<13` (sidecar/requirements-override.txt),
# which pip's resolver refuses. Like the install it replaces, it reads the lines whose markers
# match the auditing interpreter (3.12 in CI).
pip-audit:
pip-audit -r sidecar/requirements.lock -r sidecar/requirements-ocr.lock -r sidecar/requirements-tsne.lock
pip-audit --disable-pip -r sidecar/requirements.lock -r sidecar/requirements-ocr.lock -r sidecar/requirements-tsne.lock

# JS dependency CVE audit against the npm lockfile. `moderate` (not `high`) so a moderate-rated
# sanitizer / proto-pollution advisory in the render path can't pass green (L-7).
Expand Down
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "pm",
"private": true,
"version": "3.148.6-alpha",
"version": "3.148.7-alpha",
"license": "AGPL-3.0-or-later",
"type": "module",
"engines": {
Expand Down
62 changes: 59 additions & 3 deletions scripts/check-requirements-lock.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
// What it proves:
// * every lock is stamped for the CURRENT sidecar/requirements.txt, requirements-optional.txt
// and (for the optional components) the current base lock;
// * the base lock was resolved WITH the current sidecar/requirements-override.txt, and that file
// only overrides packages requirements.txt pins at the same version;
// * the resolution targets the MIN_PYTHON floor Rust actually enforces;
// * every entry is pinned with `==` — no range can slip in;
// * every entry carries at least one sha256, so `--require-hashes` has something to check;
Expand All @@ -32,6 +34,7 @@ import { fileURLToPath, pathToFileURL } from "node:url";

const SIDECAR_RS = "src-tauri/src/sidecar.rs";
const BASE_INPUT = "sidecar/requirements.txt";
const BASE_OVERRIDE = "sidecar/requirements-override.txt";
const BASE_LOCK = "sidecar/requirements.lock";
const OPTIONAL_INPUT = "sidecar/requirements-optional.txt";

Expand Down Expand Up @@ -81,16 +84,59 @@ export function parseHeader(text) {
const rawConstraint = stamp("constraint");
const constraint = rawConstraint ? source(rawConstraint) : null;
if (rawConstraint && !constraint) problems.push("malformed `# pm-constraint:` stamp");
const rawOverride = stamp("override");
const override = rawOverride ? source(rawOverride) : null;
if (rawOverride && !override) problems.push("malformed `# pm-override:` stamp");
const rawPins = stamp("pins");
return {
floor: stamp("python-floor"),
input,
constraint,
override,
pins: rawPins ? rawPins.split(/\s+/).filter(Boolean) : null,
problems,
};
}

/**
* The base lock's override file: stamped, and narrow. Each line of it REPLACES every requirement on
* that package in the resolution, a dependency's declared bounds included — that is its whole job
* (fastembed's `pillow<13`). So a base lock resolved without it must fail, and the file may only
* name packages requirements.txt pins at the same version: a range, or a package nobody pins by
* name, would be a silent hole in bounds that nobody reviews.
*/
export function overrideProblems({ header, overrideText, requirementsText }) {
const problems = [];
if (!header.override) {
problems.push(
`no \`# pm-override:\` stamp — this lock was resolved without ${BASE_OVERRIDE}; run \`just lock-regen\``,
);
} else if (header.override.path !== BASE_OVERRIDE) {
problems.push(
`\`# pm-override:\` names ${header.override.path}, but the base lock's override is ${BASE_OVERRIDE}`,
);
}
const pinned = new Map(
parseRequirements(requirementsText).map((p) => [normalise(p.name), p.version]),
);
for (const line of overrideText.replace(/\r\n/g, "\n").split("\n")) {
const trimmed = line.trim();
if (!trimmed || trimmed.startsWith("#")) continue;
const [pin] = parseRequirements(trimmed);
if (!pin) {
problems.push(
`${BASE_OVERRIDE}: \`${trimmed}\` is not an exact \`==\` pin — an override may only pin`,
);
} else if (pinned.get(normalise(pin.name)) !== pin.version) {
problems.push(
`${BASE_OVERRIDE}: \`${pin.raw}\` is not pinned at ${pin.version} in ${BASE_INPUT} — an override ` +
`lifts every declared bound on a package, so it may only name one PM pins at that version`,
);
}
}
return problems;
}

/** Every pinned entry in a lock, with the hashes attached to it. */
export function parseEntries(text) {
const lines = text.replace(/\r\n/g, "\n").split("\n");
Expand Down Expand Up @@ -162,7 +208,7 @@ export function problemsForLock({ path, text, floor, inputs, expectedPins }) {
);
}

for (const key of ["input", "constraint"]) {
for (const key of ["input", "constraint", "override"]) {
const stamped = header[key];
if (!stamped) continue;
const actual = inputs[stamped.path];
Expand Down Expand Up @@ -226,14 +272,16 @@ export function problemsForLock({ path, text, floor, inputs, expectedPins }) {
return problems;
}

export function scan(root) {
const read = (rel) => readFileSync(join(root, rel), "utf8");
/** `read` is injectable so a test can run the whole scan over the real tree with one file altered. */
export function scan(root, read = (rel) => readFileSync(join(root, rel), "utf8")) {
const { floor, pins } = readRustFacts(read(SIDECAR_RS));
const baseInput = read(BASE_INPUT);
const baseOverride = read(BASE_OVERRIDE);
const optionalInput = read(OPTIONAL_INPUT);
const baseLock = read(BASE_LOCK);
const inputs = {
[BASE_INPUT]: baseInput,
[BASE_OVERRIDE]: baseOverride,
[OPTIONAL_INPUT]: optionalInput,
[BASE_LOCK]: baseLock,
};
Expand All @@ -253,6 +301,14 @@ export function scan(root) {
expectedPins: parseRequirements(baseInput),
}),
);
record(
BASE_LOCK,
overrideProblems({
header: parseHeader(baseLock),
overrideText: baseOverride,
requirementsText: baseInput,
}),
);

const optionalPins = parseRequirements(optionalInput);
let entryCount = parseEntries(baseLock).length;
Expand Down
Loading
Loading