From dbbace47f8c2983dff41d91ffb3c45abf7d0cb9b Mon Sep 17 00:00:00 2001 From: neohiro Date: Sun, 4 Oct 2026 04:11:24 +0200 Subject: [PATCH 01/17] feat: force HTTPS for all package repos before any download A plaintext http:// package mirror lets anyone on the path swap the .deb/.rpm/.pkgz you just fetched for one of theirs. Signature checks catch forged packages but not downgrades to an older, genuinely-signed, vulnerable build -- only TLS on the transport closes that gap. Adds lib/apt-https.sh as a repository transport guard that runs BEFORE anything is fetched: - apt: installs /etc/apt/apt.conf.d/99neohiro-force-https (the key line is Acquire::http::AllowRedirect "false", so an https:// mirror cannot silently downgrade a fetch to plaintext), and rewrites http:// -> https:// on active lines in sources.list, sources.list.d/*.list and DEB822 *.sources. Comments and DEB822 structural fields are left untouched. - Every file is backed up to /var/backups/neohiro-apt-https/ before the first edit and logged to the rollback log. - The rewrite is verified with a real `apt-get update`; if a mirror does not speak TLS the rewrite is rolled back automatically, so a working package manager is never left broken. - dnf/yum/zypper/pacman/flatpak are audited and reported but not blindly rewritten (many mirrors lack TLS on the same paths); opt in with NEOHIRO_APT_HTTPS_REWRITE=1. - Optional belt-and-braces `ufw deny out 80/tcp` behind NEOHIRO_APT_BLOCK_PORT80=1, which declines while plaintext repos remain. Wired into every surface that downloads packages: pkg_update, pkg_install, pkg_upgrade, pkg_autoremove, update_system, update_kernel, updates_only_mode, all _update_* in lib/updater.sh plus the _run_all_updates dispatcher, restore_ssh.sh (before openssh-server), DeepClean.sh (before apt-get autoremove --purge), the Maintenance submenu (option 1, now 21 tools), --step apt_https, the progress checklist, and the --apt-https / --apt-https-audit / --apt-https-off CLI flags on both linuxinstall.sh and the standalone scripts. It is the first workflow step on every profile, including Custom, and it is idempotent and latched to once per process so AUTO_MODE re-runs and the hot-path guards cost nothing. Also fixes two latent bugs found while wiring this up: - helpers that returned their result via command substitution had their own progress messages folded into the return value, which then landed inside an arithmetic expansion; results now go through globals. - _step_apt_https used `if ! cmd; then rc=$?`, which captures the negation's status, so a failed enforcement reported success. test_color_gate.sh now anchors its restore_ssh.sh slice on function names instead of hardcoded line numbers, and README.md had a stray unresolved merge-conflict marker removed. Tests: tests/test_apt_https.sh, 92 hermetic assertions (fake /etc, fake backup dir, fake ufw, NEOHIRO_APT_FAMILY pin) covering the rewrite, the DEB822 path, comment preservation, idempotency, the backup-before-edit guarantee, refusal-without-backup, revert, exit codes, every env kill switch, and that all ten package entry points guard before their first package command. Full suite: 522 passed, 0 failures. --- DeepClean.sh | 17 + README.md | 151 ++++++- SECURITY.md | 28 ++ lib/README.md | 23 +- lib/apt-https.sh | 754 +++++++++++++++++++++++++++++++++++ lib/updater.sh | 45 +++ linuxinstall.sh | 539 +++++++++++++++++++++++-- restore_ssh.sh | 32 ++ tests/run-all.sh | 1 + tests/test_apt_https.sh | 785 +++++++++++++++++++++++++++++++++++++ tests/test_color_gate.sh | 19 +- tests/test_linuxinstall.sh | 2 +- 12 files changed, 2343 insertions(+), 53 deletions(-) create mode 100644 lib/apt-https.sh create mode 100644 tests/test_apt_https.sh diff --git a/DeepClean.sh b/DeepClean.sh index 90f8c12..f5ff522 100644 --- a/DeepClean.sh +++ b/DeepClean.sh @@ -17,6 +17,14 @@ if [ -r "$(dirname "$(readlink -f "${BASH_SOURCE[0]:-$0}")")/lib/color.sh" ]; th source "$(dirname "$(readlink -f "${BASH_SOURCE[0]:-$0}")")/lib/color.sh" fi +# Repository transport guard. DeepClean's apt branch runs +# `apt-get autoremove --purge`, which resolves dependency chains and can +# pull packages from a mirror, so HTTPS is enforced before it. +# shellcheck disable=SC1091 +if [ -r "$(dirname "$(readlink -f "${BASH_SOURCE[0]:-$0}")")/lib/apt-https.sh" ]; then + source "$(dirname "$(readlink -f "${BASH_SOURCE[0]:-$0}")")/lib/apt-https.sh" +fi + # If lib/color.sh was not sourced or did not set USE_COLOR, run the canonical # gate inline so _c is safe to call in all execution paths. if [ -z "${USE_COLOR:-}" ]; then @@ -66,6 +74,15 @@ PM=$(pkg_mgr) USED_BEFORE_KB=$(df -kP / | tail -1 | awk '{print $3}') msg "Detected package manager: ${PM:-none}" + +# Precaution: `apt-get autoremove --purge` below resolves dependencies and +# can fetch from a mirror, so force repository traffic over HTTPS first. +# No-op once per process, and a silent no-op when the lib is unavailable +# (curl|bash of DeepClean.sh on its own). +if declare -F apt_https_guard >/dev/null 2>&1 && [ "$PM" = "apt" ]; then + apt_https_guard "DeepClean" || true +fi + msg "Starting DeepClean..." # 1. Systemd Journal Logs diff --git a/README.md b/README.md index 9217630..6be08de 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,14 @@ the package name right per distro. This script: - **Rollback log per file.** Every config it edits is backed up to a timestamped copy; the index lives at `/var/log/linux-install-rollback.log` and is one `cp` away from a full undo. -- **Three security profiles + a 20-tool maintenance suite.** From +- **HTTPS-only package transport, enforced before every download.** A + plaintext `http://` mirror lets anyone on the path swap a `.deb`/`.rpm` + for their own. The guard rewrites every repo URL to `https://`, installs + an apt policy drop-in that refuses HTTPS→HTTP downgrade redirects, backs + the originals up, and auto-reverts if a mirror turns out not to speak + TLS. Runs first on every profile — see + [Package transport security](#package-transport-security-https). +- **Three security profiles + a 21-tool maintenance suite.** From "Recommended" (firewall + updates, 6 steps, no SSH risk) to "Full" (Tor + IPv6 disable + ASR + deep clean, 12 steps). Maintenance menu re-runs any step on a live box without re-hardening. @@ -45,9 +52,10 @@ the package name right per distro. This script: | SSH hardening | `PasswordAuthentication no` gated on validated pubkey; port never changed | | Fail2ban, sysctl profile, AppArmor/SELinux check | per-distro package names | | Tor, dnscrypt-proxy, unattended-upgrades, DeepClean | optional per profile | +| **HTTPS-only package transport** | enforced before every `apt`/`dnf`/`yum`/`zypper`/`pacman` download — `--apt-https` | | **Rollback log** | `/var/log/linux-install-rollback.log` — `original\tbackup` per file | | **SSH self-heal** | `--install-self-heal` — systemd timer or cron, every 60s | -| **20-tool maintenance suite** | Re-runs any step, lists keys, tails logs, dumps config | +| **21-tool maintenance suite** | Re-runs any step, lists keys, tails logs, dumps config | ## Quick start @@ -82,8 +90,6 @@ the in-script `restore_ssh` routine or Tailscale SSH gets you back in. > → yum → apt`, so Arch derivatives pick `pacman`, SUSE picks `zypper`, > RHEL/Fedora pick `dnf`, Debian/Ubuntu pick `apt`. No manual flag required. -<<<<<<< HEAD -======= ## One-step automated setup Run the general interactive script directly from the repo — it prompts you @@ -114,7 +120,119 @@ SSH can get you back in. `━━━ PROGRESS ████████████░░░░ 12/17 (70%) ━━━`) before every step, so you always see what's already done and what's coming. ->>>>>>> origin/main +### Package transport security (HTTPS) + +A plaintext `http://` package mirror means anyone who can intercept the +route — a hostile Wi-Fi, a compromised router, an upstream CDN node — can +swap the `.deb` / `.rpm` / `.pkgz` you just downloaded for one of theirs. +Signature checks catch *forged* packages, but they do not stop a +*downgrade* to an older, genuinely-signed, vulnerable build. Only TLS on +the transport closes that gap. + +So every entry point in this repo runs a guard **before** anything is +fetched. It is the first workflow step (on every profile, including +Custom), it runs once per process, and it is idempotent, so re-runs and +`--auto` are cheap. + +```bash +# Apply it on its own (normally automatic) +sudo bash linuxinstall.sh --apt-https + +# Report only; never modifies anything. Exit 1 if plaintext remains. +sudo bash linuxinstall.sh --apt-https-audit + +# Undo: restore every backup and remove the policy drop-in. +sudo bash linuxinstall.sh --apt-https-off + +# Or standalone, no installer needed: +sudo bash lib/apt-https.sh # enforce +sudo bash lib/apt-https.sh --report # audit +sudo bash lib/apt-https.sh --revert # undo +``` + +**On apt** (Debian / Ubuntu / Mint / Pop!_OS / Kali and derivatives) it: + +1. Installs `/etc/apt/apt.conf.d/99neohiro-force-https`: + + ```text + Acquire::https::AllowRedirect "true"; // https -> https redirects are fine + Acquire::http::AllowRedirect "false"; // https -> http is REFUSED, not followed + Acquire::https::Verify-Peer "true"; + Acquire::https::Verify-Host "true"; + Acquire::Retries "3"; + ``` + + The anti-downgrade line is the important one: without it, an + `https://` mirror can silently bounce you down to `http://`. + +2. Rewrites `http://` → `https://` on active lines in + `/etc/apt/sources.list`, `/etc/apt/sources.list.d/*.list` (classic) and + `/etc/apt/sources.list.d/*.sources` (DEB822 `URIs:` field only). + Commented-out lines and DEB822 structural fields (`Suites:`, + `Components:`, `Signed-By:`) are left exactly as they were. + +3. Backs every file up to `/var/backups/neohiro-apt-https/` **before** the + first edit and logs each one to the rollback log, so + `bash linuxinstall.sh --rollback --apply` can undo it too. + +4. Verifies with a real `apt-get update`. If a mirror turns out not to + serve the same paths over TLS, the rewrite is **rolled back + automatically** — you are never left with an unusable package manager. + Set `NEOHIRO_APT_HTTPS_STRICT=1` to keep the rewrite and fail loudly + instead. + +5. Warns if no CA trust store is present, since HTTPS verification is + worthless without one. + +**On dnf / yum / zypper / pacman / flatpak** it audits and reports every +plaintext repo URL but does **not** blindly rewrite: plenty of upstream +mirrors do not serve the same paths over TLS, and silently breaking a +working repo is worse than the threat it prevents. Point those at an +https-capable mirror, or opt in to the rewrite with +`NEOHIRO_APT_HTTPS_REWRITE=1` and re-audit afterwards. + +**Where it is wired in:** `pkg_update` / `pkg_install` / `pkg_upgrade` / +`pkg_autoremove`, `update_system`, `update_kernel`, `updates_only_mode`, +every `_update_*` in `lib/updater.sh` plus the `_run_all_updates` +dispatcher, `restore_ssh.sh` (before installing `openssh-server`), +`DeepClean.sh` (before `apt-get autoremove --purge`), the **Maintenance** +submenu option 1, the `--step apt_https` mode, and the `--apt-https*` +flags. `OptimizeLinuxASR.sh` does not download packages, so it needs no +guard. + +| Variable | Effect | +|---|---| +| `NEOHIRO_APT_HTTPS=1` | enforce (default) | +| `NEOHIRO_APT_HTTPS=audit` | report only, never modify | +| `NEOHIRO_APT_HTTPS=0` | disable the guard entirely | +| `NEOHIRO_APT_HTTPS_REWRITE=1` | allow the http→https rewrite on dnf/yum/zypper/pacman | +| `NEOHIRO_APT_HTTPS_NOVERIFY=1` | skip the post-rewrite `apt-get update` check | +| `NEOHIRO_APT_HTTPS_STRICT=1` | treat leftover plaintext repos as a hard error | +| `NEOHIRO_APT_BLOCK_PORT80=1` | also `ufw deny out 80/tcp` (opt-in, see below) | + +### Optional: block port 80 entirely + +If you want belt-and-braces so that *no* process can open an unencrypted +package connection, add an outbound deny rule: + +```bash +sudo bash linuxinstall.sh --apt-https # make sure every repo is https first +sudo ufw deny out 80/tcp && sudo ufw reload +# or: NEOHIRO_APT_BLOCK_PORT80=1 sudo bash linuxinstall.sh --apt-https +``` + +This is **opt-in** because a blanket outbound block also breaks unrelated +plaintext protocols (local registries, metrics endpoints, captive-portal +checks). The guard refuses to add the rule while any plaintext repo is +still configured, since that would only break those repos. + +Verify: + +```bash +sudo ufw status | grep -E '80/tcp|Status' # outbound DENY present +sudo bash linuxinstall.sh --apt-https-audit # exit 0 = no plaintext repos left +``` + ### Cross-distro kernel update `linuxinstall.sh` auto-detects the package manager and updates the kernel @@ -265,14 +383,15 @@ The Maintenance suite itself is expanded to include: | # | Option | What it does | |---|---|---| -| 1–13 | system / dns / firewall / tor / ssh / fail2ban / unattended / ipv6 / sysctl / apparmor / pam / OptimizeLinuxASR / DeepClean | Re-run any step on demand | -| 14 | SSH diagnostics & lockout fix | Same routine as `--restore-ssh` | -| 15 | Authorized keys | List all keys in every user's `authorized_keys` | -| 16 | SSH config review | Print every key directive from `sshd_config` and drop-ins | -| 17 | SSH self-heal guard | Install / remove / status of the per-minute watchdog | -| 18 | Logs | Tail `/var/log/linux-install-rollback.log` and `/var/log/neohiro-ssh-watchdog.log` | -| 19 | System info | Uptime, load, memory, disk, CPU, listening ports | -| 20 | Back to main menu | — | +| 1 | Force HTTPS for package repos | Enforces TLS for every repo before any download; prints the resulting state | +| 2–14 | system / dns / firewall / tor / ssh / fail2ban / unattended / ipv6 / sysctl / apparmor / pam / OptimizeLinuxASR / DeepClean | Re-run any step on demand | +| 15 | SSH diagnostics & lockout fix | Same routine as `--restore-ssh` | +| 16 | Authorized keys | List all keys in every user's `authorized_keys` | +| 17 | SSH config review | Print every key directive from `sshd_config` and drop-ins | +| 18 | SSH self-heal guard | Install / remove / status of the per-minute watchdog | +| 19 | Logs | Tail `/var/log/linux-install-rollback.log` and `/var/log/neohiro-ssh-watchdog.log` | +| 20 | System info | Uptime, load, memory, disk, CPU, listening ports | +| 21 | Back to main menu | — | The self-heal guard runs as root via `systemd` or cron and **never modifies `authorized_keys` or any credentials** — it only fixes config @@ -541,8 +660,10 @@ dumps — add to `/etc/security/limits.conf`: ### Testing ```bash -bash tests/test_linuxinstall.sh # 65 tests: parse, logic, UX coverage, snapshot -bash tests/test_updater.sh # 43 tests: dispatcher, race safety, version floor +bash tests/run-all.sh # every suite, with a summary +bash tests/test_linuxinstall.sh # 66 tests: parse, logic, UX coverage, snapshot +bash tests/test_apt_https.sh # 84 tests: HTTPS guard, hermetic (fake /etc, fake ufw) +bash tests/test_updater.sh # 45 tests: dispatcher, race safety, version floor shellcheck -S warning *.sh lib/*.sh tests/*.sh # lint ``` diff --git a/SECURITY.md b/SECURITY.md index b6b627d..a19974d 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -28,6 +28,34 @@ multiple distribution families (Debian, RHEL/Fedora, SUSE, Arch). Always review what will be applied, keep backups/restoration points, and test on non-critical systems first. +### Package transport + +Every package download, update, and upgrade is preceded by +`apt_https_guard` (see `lib/apt-https.sh`), which forces repository +traffic over TLS: + +* On apt it installs `/etc/apt/apt.conf.d/99neohiro-force-https` and + rewrites `http://` repo URLs to `https://` in `sources.list`, + `sources.list.d/*.list`, and DEB822 `*.sources`. +* The policy file sets `Acquire::http::AllowRedirect "false"` so an + `https://` mirror cannot silently downgrade a fetch to plaintext HTTP. +* Every file it edits is backed up to `/var/backups/neohiro-apt-https/` + first and logged to `/var/log/linux-install-rollback.log`. +* If the rewritten sources fail a verification `apt-get update`, the + rewrite is rolled back automatically. + +Escape hatches and audit tooling: + +```bash +sudo bash linuxinstall.sh --apt-https-audit # read-only, exit 1 if plaintext remains +sudo bash linuxinstall.sh --apt-https-off # restore backups, remove the drop-in +NEOHIRO_APT_HTTPS=0 # disable the guard for one run +``` + +The optional `NEOHIRO_APT_BLOCK_PORT80=1` adds `ufw deny out 80/tcp`. +It is off by default because a blanket outbound block also affects +unrelated plaintext protocols. + --- Maintained by **[neohiro](https://github.com/neohiro)**. diff --git a/lib/README.md b/lib/README.md index 82e698a..34e70f7 100644 --- a/lib/README.md +++ b/lib/README.md @@ -10,6 +10,23 @@ each file for the API contract. EXIT trap cleans everything. ERR trap in STRICT_RUN / CI mode logs the failing command to `NEOHIRO_DEBUG_LOG` (default `/var/log/neohiro-debug.log`). +- `updater.sh` — the comprehensive cross-distro update engine + (`_run_all_updates` plus one `_update_*` per tool). Sourceable and + runnable standalone (`sudo bash lib/updater.sh`). +- `apt-https.sh` — repository transport guard. `apt_https_guard` is + called by every package entry point so no repo is ever fetched over + plaintext HTTP. Installs `/etc/apt/apt.conf.d/99neohiro-force-https` + (refuses HTTPS→HTTP downgrade redirects), rewrites `http://` repo URLs + to `https://` in `sources.list`, `sources.list.d/*.list` and DEB822 + `*.sources`, backs every original up to + `/var/backups/neohiro-apt-https/`, and auto-reverts if a mirror does + not speak TLS. Audits (never blindly rewrites) dnf/yum/zypper/pacman + and flatpak remotes. API: `apt_https_guard`, `apt_https_enforce`, + `apt_https_report`, `apt_https_revert`; also runnable standalone. + Tests relocate the whole tree via `NEOHIRO_APT_ETC_DIR` / + `NEOHIRO_APT_BACKUP_DIR`, so nothing touches the real `/etc`. +- `sync-inline.sh`, `color-gate.sh` — shared inline-fallback sources + consumed by the `curl | bash` path of the top-level scripts. Top-level scripts (`linuxinstall.sh`, `restore_ssh.sh`, `DeepClean.sh`, `OptimizeLinuxASR.sh`) source these automatically @@ -18,4 +35,8 @@ directory is not on disk, so each script falls back to inline definitions of the same helpers. To regenerate the inline fallbacks, copy the body of each lib file -into the `else` branch in the top-level script. +into the `else` branch in the top-level script. Keep the public +function names identical — `tests/test_apt_https.sh` asserts that +`linuxinstall.sh`'s inline fallback and `lib/apt-https.sh` expose the +same API, and that both guard before the first package command in +every `pkg_*` helper. diff --git a/lib/apt-https.sh b/lib/apt-https.sh new file mode 100644 index 0000000..4e68fd8 --- /dev/null +++ b/lib/apt-https.sh @@ -0,0 +1,754 @@ +#!/usr/bin/env bash +# lib/apt-https.sh - HTTPS-only repository enforcement (precautionary guard). +# +# Purpose: BEFORE any package manager in this repo downloads, updates, or +# upgrades anything, make sure repository traffic is authenticated/TLS +# instead of plaintext HTTP. An on-path attacker who can rewrite an +# http:// mirror URL can otherwise inject arbitrary .deb/.rpm/.pkgz files +# into an otherwise fully-hardened machine. +# +# What "enforce" means per family: +# +# apt (Debian / Ubuntu / Mint / Pop!_OS / Kali / derivatives) +# 1. Install a managed policy drop-in +# /etc/apt/apt.conf.d/99neohiro-force-https: +# - Acquire::https::AllowRedirect "true" -> https mirrors that +# redirect within https are allowed. +# - Acquire::http::AllowRedirect "false" -> a redirect from +# https:// down to http:// is REFUSED instead of silently +# followed. This is the anti-downgrade control. +# - Acquire::https::Verify-Peer/Verify-Host "true" +# - Acquire::Retries "3" +# 2. Rewrite http:// -> https:// in every repo definition: +# /etc/apt/sources.list +# /etc/apt/sources.list.d/*.list (classic one-line format) +# /etc/apt/sources.list.d/*.sources (DEB822: URIs: field only) +# Comments are left untouched. Every touched file is backed up to +# /var/backups/neohiro-apt-https/ before the first edit. +# 3. Verify with a real `apt-get update`. If the rewrite broke the +# mirrors, the backups are restored automatically so the machine is +# never left with an unusable package manager. +# +# dnf / yum / zypper / pacman / flatpak +# No blind rewrite. Many upstream mirrors do not serve the same paths +# over TLS, and silently breaking a working repo is worse than the +# threat. Instead: audit, report every plaintext entry, and offer an +# opt-in rewrite via NEOHIRO_APT_HTTPS_REWRITE=1. +# +# Optional hardening (NOT default, because a global outbound port-80 block +# also breaks legitimate plaintext tooling such as local registries and +# metrics endpoints): +# NEOHIRO_APT_BLOCK_PORT80=1 -> ufw deny out 80/tcp +# +# Full environment surface: +# NEOHIRO_APT_HTTPS=1 enforce (default) +# NEOHIRO_APT_HTTPS=audit report only, never write +# NEOHIRO_APT_HTTPS=0 disable entirely +# NEOHIRO_APT_HTTPS_REWRITE=1 allow http->https rewrite on non-apt families +# NEOHIRO_APT_HTTPS_NOVERIFY=1 skip the post-rewrite `apt-get update` check +# NEOHIRO_APT_BLOCK_PORT80=1 add `ufw deny out 80/tcp` +# NEOHIRO_APT_ETC_DIR=path relocate the /etc tree (used by the tests) +# NEOHIRO_APT_BACKUP_DIR=path relocate the backup tree (used by the tests) +# +# Usage: +# source lib/apt-https.sh +# apt_https_guard # idempotent, once per process; use in hot paths +# apt_https_enforce # force a full pass (menus / CLI) +# apt_https_report # read-only status; rc 0 = all repos are https +# apt_https_revert # restore every backup taken by this lib +# +# Bash 4.0+ is required, matching the rest of the repo. + +if [ "${BASH_VERSINFO[0]:-0}" -lt 4 ]; then + printf '%s\n' "lib/apt-https.sh requires Bash 4.0+; found ${BASH_VERSION:-unknown}." >&2 + return 1 2>/dev/null || exit 1 +fi + +# Guard against double-sourcing. +[ -n "${__NEOHIRO_APT_HTTPS_INIT:-}" ] && return 0 2>/dev/null || true +__NEOHIRO_APT_HTTPS_INIT=1 + +# Re-entrancy / once-per-process latch. Set BEFORE any work so a nested +# call (e.g. the verification `apt-get update` re-entering pkg_update) +# short-circuits instead of recursing. +_APT_HTTPS_DONE="${_APT_HTTPS_DONE:-}" + +# Print helpers. Reuse the host's when available, otherwise define +# self-contained fallbacks so this file works standalone. +if ! declare -F _c >/dev/null 2>&1; then + _c() { if [ "${USE_COLOR:-0}" = "1" ]; then printf '\033[%s%s\033[0m' "$1" "$2"; else printf '%s' "$2"; fi; } +fi +if ! declare -F info >/dev/null 2>&1; then + info() { printf ' %s\n' "$*"; } +fi +if ! declare -F msg >/dev/null 2>&1; then + msg() { echo "=> $*"; } +fi +if ! declare -F ok >/dev/null 2>&1; then + ok() { printf '%s %s\n' "$(_c '1;32m' '[OK]')" "$*"; } +fi +if ! declare -F warn >/dev/null 2>&1; then + warn() { printf '%s %s\n' "$(_c '1;33m' '[WARNING]')" "$*" >&2; } +fi +if ! declare -F err >/dev/null 2>&1; then + err() { printf '%s %s\n' "$(_c '1;31m' '[ERROR]')" "$*" >&2; } +fi + +# ── Paths ──────────────────────────────────────────────────────────────────── + +APT_HTTPS_CONF_NAME="99neohiro-force-https" + +apt_https_etc_dir() { + printf '%s' "${NEOHIRO_APT_ETC_DIR:-/etc}" +} + +apt_https_backup_dir() { + printf '%s' "${NEOHIRO_APT_BACKUP_DIR:-/var/backups/neohiro-apt-https}" +} + +# _apt_https_conf_file — absolute path of the managed apt policy drop-in. +apt_https_conf_file() { + printf '%s' "$(apt_https_etc_dir)/apt/apt.conf.d/${APT_HTTPS_CONF_NAME}" +} + +# ── Privilege / dry-run plumbing ───────────────────────────────────────────── + +_apt_https_is_root() { + [ "${EUID:-$(id -u 2>/dev/null || echo 1000)}" = "0" ] +} + +# _apt_priv — run a privileged command, honouring DRY_RUN. +# Returns 1 when privileges are unavailable so callers can degrade. +_apt_priv() { + if [ "${DRY_RUN:-0}" = "1" ]; then + printf ' DRY: %s\n' "$*" + return 0 + fi + if _apt_https_is_root; then + "$@" + return $? + fi + if command -v sudo >/dev/null 2>&1; then + sudo "$@" + return $? + fi + return 1 +} + +# _apt_https_privileged_ok — true when we can actually write to /etc. +_apt_https_privileged_ok() { + _apt_https_is_root && return 0 + command -v sudo >/dev/null 2>&1 && return 0 + return 1 +} + +# ── Family detection ───────────────────────────────────────────────────────── +# Same precedence as detect_distro() in linuxinstall.sh and pkg_mgr() in +# DeepClean.sh: pacman -> zypper -> dnf -> yum -> apt. +# +# NEOHIRO_APT_FAMILY pins the result. That is what lets the test suite (and +# a CI container running a foreign base image) exercise the apt, dnf, +# zypper, pacman and "none" code paths deterministically. + +apt_https_family() { + if [ -n "${NEOHIRO_APT_FAMILY:-}" ]; then + printf '%s' "$NEOHIRO_APT_FAMILY" + return 0 + fi + if command -v pacman >/dev/null 2>&1 && [ -f /etc/pacman.conf ]; then + printf 'pacman' + elif command -v zypper >/dev/null 2>&1; then + printf 'zypper' + elif command -v dnf >/dev/null 2>&1; then + printf 'dnf' + elif command -v yum >/dev/null 2>&1; then + printf 'yum' + elif command -v apt-get >/dev/null 2>&1 || [ -f /etc/apt/sources.list ]; then + printf 'apt' + else + printf 'none' + fi +} + +# ── Repo file discovery ────────────────────────────────────────────────────── + +# _apt_https_repo_files — one repo-definition path per line (apt only). +_apt_https_repo_files() { + local etc f + etc="$(apt_https_etc_dir)" + [ -f "${etc}/apt/sources.list" ] && printf '%s\n' "${etc}/apt/sources.list" + for f in "${etc}"/apt/sources.list.d/*.list "${etc}"/apt/sources.list.d/*.sources; do + [ -f "$f" ] && printf '%s\n' "$f" + done + return 0 +} + +# _apt_https_foreign_repo_files — repo definitions for the other +# families, so the audit has something to read even when the detected family +# is different (dual-boot / container with a foreign sources dir). +_apt_https_foreign_repo_files() { + local etc f d + etc="$(apt_https_etc_dir)" + case "$1" in + dnf|yum) + for f in "${etc}"/yum.repos.d/*.repo; do [ -f "$f" ] && printf '%s\n' "$f"; done + ;; + zypper) + for f in "${etc}"/zypp/repos.d/*.repo; do [ -f "$f" ] && printf '%s\n' "$f"; done + ;; + pacman) + for f in "${etc}"/pacman.d/*; do [ -f "$f" ] && printf '%s\n' "$f"; done + ;; + *) : ;; + esac + return 0 +} + +# ── Plaintext detection ────────────────────────────────────────────────────── + +# _apt_https_plaintext_in_apt_file +# Prints the offending (line-number + trimmed-line) pairs. DEB822 files are +# matched on the URIs: field only, so a commented-out http:// note never +# counts. Classic files skip comment lines entirely. +_apt_https_plaintext_in_apt_file() { + local f="$1" + [ -f "$f" ] || return 0 + case "$f" in + *.sources) + awk '/^[[:space:]]*URIs:[[:space:]]/ && /http:\/\// { + gsub(/^[[:space:]]+/, "", $0); printf "%s:%d: %s\n", FILENAME, NR, $0 }' "$f" 2>/dev/null + ;; + *) + awk '/^[[:space:]]*#/ { next } /http:\/\// { + gsub(/^[[:space:]]+/, "", $0); printf "%s:%d: %s\n", FILENAME, NR, $0 }' "$f" 2>/dev/null + ;; + esac + return 0 +} + +# _apt_https_plaintext_in_repo_file +# Generic scan for the yum/dnf/zypper/pacman repo layouts. $2 is an ERE +# applied per line; the line must also contain http:// to be reported. +_apt_https_plaintext_in_repo_file() { + local f="$1" pat="$2" + [ -f "$f" ] || return 0 + awk -v pat="$pat" ' + /^[[:space:]]*#/ { next } + $0 ~ pat && /http:\/\// { + line = $0 + gsub(/^[[:space:]]+/, "", line) + printf "%s:%d: %s\n", FILENAME, NR, line + }' "$f" 2>/dev/null + return 0 +} + +# ── Backups ────────────────────────────────────────────────────────────────── + +# _apt_https_backup_path — deterministic, path-flattened backup name. +_apt_https_backup_path() { + local flat + flat="$(printf '%s' "$1" | tr '/' '_')" + printf '%s/%s.orig' "$(apt_https_backup_dir)" "$flat" +} + +# _apt_https_backup_once — copy aside the first time we touch it. +_apt_https_backup_once() { + local src="$1" dst + [ -f "$src" ] || return 0 + dst="$(_apt_https_backup_path "$src")" + [ -f "$dst" ] && return 0 + _apt_priv mkdir -p "$(apt_https_backup_dir)" || return 1 + if ! _apt_priv cp -p "$src" "$dst"; then + warn "Could not back up $src — refusing to edit it." + return 1 + fi + # Hook into the host's rollback log when one exists so `linuxinstall.sh + # --rollback` can undo this too. + if declare -F record_backup >/dev/null 2>&1; then + record_backup "$src" "$dst" 2>/dev/null || true + fi + info "Backed up $src -> $dst" + return 0 +} + +# ── apt policy drop-in ─────────────────────────────────────────────────────── + +# _apt_https_write_conf — install the managed apt.conf.d policy file. +# Idempotent: byte-identical content is left alone (no backup churn). +_apt_https_write_conf() { + local conf tmp rc + conf="$(apt_https_conf_file)" + tmp="$(_apt_https_stage)" + { + printf '%s\n' '// Managed by neohiro/linux (lib/apt-https.sh). Do not edit.' + printf '%s\n' '// Re-running the installer overwrites this file safely;' + printf '%s\n' '// `bash linuxinstall.sh --apt-https-off` removes it.' + printf '%s\n' '' + printf '%s\n' '// Prefer TLS, and never silently downgrade to plaintext.' + printf '%s\n' 'Acquire::https::AllowRedirect "true";' + printf '%s\n' 'Acquire::http::AllowRedirect "false";' + printf '%s\n' 'Acquire::ftp::AllowRedirect "false";' + printf '%s\n' '' + printf '%s\n' '// Authenticate the mirror, not just the channel.' + printf '%s\n' 'Acquire::https::Verify-Peer "true";' + printf '%s\n' 'Acquire::https::Verify-Host "true";' + printf '%s\n' '' + printf '%s\n' '// Transparent mirrors usually need more than one try.' + printf '%s\n' 'Acquire::Retries "3";' + printf '%s\n' '' + printf '%s\n' '// Pin a corporate proxy by uncommenting and editing:' + printf '%s\n' '// Acquire::http::Proxy "http://proxy.corp.example:3128";' + printf '%s\n' '// Acquire::https::Proxy "http://proxy.corp.example:3128";' + } > "$tmp" + # Already correct? Then there is nothing to do. + if [ -f "$conf" ] && cmp -s "$tmp" "$conf"; then + rm -f "$tmp" + return 0 + fi + if ! _apt_https_backup_once "$conf"; then + rm -f "$tmp" + return 1 + fi + _apt_priv mkdir -p "$(apt_https_etc_dir)/apt/apt.conf.d" || { rm -f "$tmp"; return 1; } + if ! _apt_priv install -m 0644 "$tmp" "$conf"; then + # `install` is not present on every minimal image; fall back to cp+chmod. + if _apt_priv cp "$tmp" "$conf" && _apt_priv chmod 0644 "$conf"; then + rc=0 + else + rc=1 + fi + else + rc=0 + fi + rm -f "$tmp" + [ "$rc" = "0" ] || return 1 + info "apt policy: $conf" + return 0 +} + +# _apt_https_stage — private scratch path for generated files. +_apt_https_stage() { + if declare -F _tmpfile >/dev/null 2>&1; then + _tmpfile apt-https + return 0 + fi + mktemp "${TMPDIR:-/tmp}/neohiro-apt-https.XXXXXX" +} + +# ── apt source rewriting ───────────────────────────────────────────────────── +# +# The three functions below report their result through globals +# (_APT_HTTPS_VERDICT / _APT_HTTPS_CHANGED / _APT_HTTPS_REVERTED) instead of +# stdout. They emit progress messages of their own, and a command +# substitution would fold that text into the value it returns -- which then +# lands inside an arithmetic expansion and turns a word like "Restored" into +# a variable lookup. Globals keep the two channels separate. + +_APT_HTTPS_VERDICT="" +_APT_HTTPS_CHANGED=0 +_APT_HTTPS_REVERTED=0 +# Files that DID contain plaintext but could not be rewritten (no writable +# backup, or the privileged copy failed). Tracked separately so +# apt_https_enforce never claims "already https" when it actually gave up. +_APT_HTTPS_REFUSED=0 + +# _apt_https_rewrite_file +# Sets _APT_HTTPS_VERDICT to "changed" or "clean". Backs the file up before +# the first edit and refuses to edit a file it could not back up. +_apt_https_rewrite_file() { + local f="$1" tmp expr + _APT_HTTPS_VERDICT="clean" + [ -f "$f" ] || return 0 + + case "$f" in + *.sources) expr='/^[[:space:]]*URIs:[[:space:]]/ s|http://|https://|g' ;; + *) expr='/^[[:space:]]*#/! s|http://|https://|g' ;; + esac + + tmp="$(_apt_https_stage)" + if ! sed -E "$expr" "$f" > "$tmp" 2>/dev/null; then + rm -f "$tmp" + _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)) + return 0 + fi + if cmp -s "$tmp" "$f"; then + rm -f "$tmp" + return 0 + fi + # Refuse to edit anything we cannot first back up or cannot write. + if ! _apt_https_backup_once "$f"; then + rm -f "$tmp" + _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)) + return 0 + fi + if ! _apt_priv cp "$tmp" "$f"; then + rm -f "$tmp" + _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)) + return 0 + fi + rm -f "$tmp" + _APT_HTTPS_VERDICT="changed" + return 0 +} + +# _apt_https_rewrite_apt_sources — rewrite every apt repo definition. +# Sets _APT_HTTPS_CHANGED to the number of files rewritten. +_apt_https_rewrite_apt_sources() { + local f + _APT_HTTPS_CHANGED=0 + _APT_HTTPS_REFUSED=0 + while IFS= read -r f; do + [ -n "$f" ] || continue + _apt_https_rewrite_file "$f" + if [ "$_APT_HTTPS_VERDICT" = "changed" ]; then + _APT_HTTPS_CHANGED=$((_APT_HTTPS_CHANGED + 1)) + info "Rewrote plaintext repo URLs -> https: $f" + fi + done < <(_apt_https_repo_files) + return 0 +} + +# _apt_https_revert_apt_sources — undo the rewrite from the backups. +# Sets _APT_HTTPS_REVERTED to the number of files restored. +_apt_https_revert_apt_sources() { + local f bak backupdir + _APT_HTTPS_REVERTED=0 + backupdir="$(apt_https_backup_dir)" + [ -d "$backupdir" ] || return 0 + while IFS= read -r f; do + [ -n "$f" ] || continue + bak="$(_apt_https_backup_path "$f")" + if [ -f "$bak" ] && ! cmp -s "$bak" "$f"; then + if _apt_priv cp "$bak" "$f"; then + _APT_HTTPS_REVERTED=$((_APT_HTTPS_REVERTED + 1)) + info "Restored $f from backup" + fi + fi + done < <(_apt_https_repo_files) + return 0 +} + +# ── Non-apt audit / optional rewrite ───────────────────────────────────────── + +_DNF_URL_RE='^([[:space:]]*)(baseurl|metalink|mirrorlist)[[:space:]]*=' +_ZYPPER_URL_RE='^([[:space:]]*)(baseurl|uri|mirrorlist)[[:space:]]*=' +_PACMAN_URL_RE='^([[:space:]]*)Server[[:space:]]*=' + +# _apt_https_audit_family — print plaintext repo lines. +_apt_https_audit_family() { + local fam="$1" f + case "$fam" in + apt) + while IFS= read -r f; do + [ -n "$f" ] && _apt_https_plaintext_in_apt_file "$f" + done < <(_apt_https_repo_files) + ;; + dnf|yum) + while IFS= read -r f; do + [ -n "$f" ] && _apt_https_plaintext_in_repo_file "$f" "$_DNF_URL_RE" + done < <(_apt_https_foreign_repo_files "$fam") + ;; + zypper) + while IFS= read -r f; do + [ -n "$f" ] && _apt_https_plaintext_in_repo_file "$f" "$_ZYPPER_URL_RE" + done < <(_apt_https_foreign_repo_files "$fam") + ;; + pacman) + while IFS= read -r f; do + [ -n "$f" ] && _apt_https_plaintext_in_repo_file "$f" "$_PACMAN_URL_RE" + done < <(_apt_https_foreign_repo_files "$fam") + ;; + *) : ;; + esac + return 0 +} + +# _apt_https_rewrite_family — opt-in http->https for non-apt +# families (NEOHIRO_APT_HTTPS_REWRITE=1). +# Sets _APT_HTTPS_CHANGED to the number of files rewritten. +_apt_https_rewrite_family() { + local fam="$1" f + _APT_HTTPS_CHANGED=0 + case "$fam" in + dnf|yum|zypper|pacman) : ;; + *) return 0 ;; + esac + while IFS= read -r f; do + [ -n "$f" ] || continue + _apt_https_rewrite_file "$f" + [ "$_APT_HTTPS_VERDICT" = "changed" ] && _APT_HTTPS_CHANGED=$((_APT_HTTPS_CHANGED + 1)) + done < <(_apt_https_foreign_repo_files "$fam") + return 0 +} + +# ── flatpak remotes ────────────────────────────────────────────────────────── + +_apt_https_flatpak_plaintext() { + command -v flatpak >/dev/null 2>&1 || return 0 + flatpak remotes --show-details 2>/dev/null | awk '/http:\/\// { print }' || true + return 0 +} + +# ── ca-certificates sanity check ───────────────────────────────────────────── + +# HTTPS verification is worthless without a trust store. Warn (never fail) +# when it is missing, because installing it would itself need a download. +_apt_https_check_ca_certs() { + local fam="$1" p + case "$fam" in + apt) + if [ -e /etc/ssl/certs/ca-certificates.crt ] || [ -e /etc/pki/tls/certs/ca-bundle.crt ]; then + return 0 + fi + p="ca-certificates" + ;; + dnf|yum) [ -e /etc/pki/tls/certs/ca-bundle.crt ] && return 0; p="ca-certificates" ;; + zypper) [ -e /etc/ssl/ca-bundle.pem ] && return 0; p="ca-certificates" ;; + pacman) [ -e /etc/ssl/certs/ca-certificates.crt ] && return 0; p="ca-certificates" ;; + *) return 0 ;; + esac + warn "No system CA trust store found. HTTPS verification of packages will fail." + info "Install '$p' after the sources are trusted, then re-run: apt_https_enforce" + return 1 +} + +# ── Optional outbound port-80 block ────────────────────────────────────────── + +# _apt_https_block_port80 — belt-and-braces so no process can open an +# unencrypted repo connection. Opt-in (NEOHIRO_APT_BLOCK_PORT80=1) because a +# blanket outbound block also affects unrelated plaintext protocols. +_apt_https_block_port80() { + [ "${NEOHIRO_APT_BLOCK_PORT80:-0}" = "1" ] || return 0 + if ! command -v ufw >/dev/null 2>&1; then + if command -v firewall-cmd >/dev/null 2>&1; then + warn "NEOHIRO_APT_BLOCK_PORT80=1 but only firewalld is present." + info "firewalld needs an explicit rich rule; skipping to avoid locking yourself out." + else + warn "NEOHIRO_APT_BLOCK_PORT80=1 but no UFW/firewalld found; skipping." + fi + return 0 + fi + if ufw status 2>/dev/null | grep -qE '^80/tcp[[:space:]]+DENY[[:space:]]+OUT'; then + info "ufw already denies outbound 80/tcp" + return 0 + fi + # Refuse while plaintext repos remain: the block would break them anyway. + if [ -n "$(apt_https_status_text)" ]; then + warn "Skipping the port-80 block: plaintext repos are still configured." + info "Fix them first (see the report above), then re-run with NEOHIRO_APT_BLOCK_PORT80=1." + return 0 + fi + if _apt_priv ufw deny out 80/tcp; then + ok "Blocked outbound TCP/80 (apt traffic is HTTPS-only from now on)." + info "Undo: sudo ufw delete deny out 80/tcp" + else + warn "Could not add the ufw outbound 80/tcp deny rule." + fi + return 0 +} + +# ── Status / report ────────────────────────────────────────────────────────── + +# apt_https_status_text — every plaintext repo line still configured. +apt_https_status_text() { + local fam + fam="$(apt_https_family)" + _apt_https_audit_family "$fam" + _apt_https_flatpak_plaintext + return 0 +} + +# apt_https_report — human-readable state summary. +# Returns 0 when everything is https, 1 when plaintext remains, 2 when the +# family is unknown. +apt_https_report() { + local fam missing conf findings + fam="$(apt_https_family)" + printf '\n%s\n' "$(_c '1;36m' '━━━ Package transport security (HTTPS) ━━━')" + printf ' Detected package manager: %s\n' "$fam" + + case "$fam" in + apt) + conf="$(apt_https_conf_file)" + if [ -f "$conf" ]; then + printf ' %s %s\n' "$(_c '1;32m' '[x]')" "apt policy drop-in active: $conf" + else + printf ' %s %s\n' "$(_c '1;31m' '[ ]')" "apt policy drop-in MISSING: $conf" + fi + ;; + dnf|yum) printf ' %s\n' "dnf/yum: repo files audited (no blind rewrite)." ;; + zypper) printf ' %s\n' "zypper: repo files audited (no blind rewrite)." ;; + pacman) printf ' %s\n' "pacman: mirrorlist audited (no blind rewrite)." ;; + *) printf ' %s\n' "No supported package manager detected." ;; + esac + + findings="$(apt_https_status_text)" + if [ -n "$findings" ]; then + printf '\n %s\n' "$(_c '1;33m' 'Plaintext (http://) repository entries still present:')" + printf '%s\n' "$findings" | sed 's/^/ /' + printf '\n %s\n' " apt: fix with sudo bash linuxinstall.sh --apt-https" + printf '%s\n' " others: set NEOHIRO_APT_HTTPS_REWRITE=1, then re-run the audit" + else + printf '\n %s %s\n' "$(_c '1;32m' '[x]')" "No plaintext repository URLs detected." + fi + _apt_https_check_ca_certs "$fam" || true + printf '\n' + + if [ "$fam" = "none" ]; then return 2; fi + [ -n "$findings" ] && return 1 + return 0 +} + +# ── Enforce ────────────────────────────────────────────────────────────────── + +# _apt_https_verify_apt_sources — run a real `apt-get update` and roll back +# the rewrite if the mirrors turned out not to speak https. +_apt_https_verify_apt_sources() { + [ "${NEOHIRO_APT_HTTPS_NOVERIFY:-0}" = "1" ] && return 0 + [ "${DRY_RUN:-0}" = "1" ] && return 0 + command -v apt-get >/dev/null 2>&1 || return 0 + + info "Verifying the rewritten sources with a real apt-get update..." + if _apt_priv env DEBIAN_FRONTEND=noninteractive apt-get update -qq; then + ok "All apt repositories answered over HTTPS." + return 0 + fi + if [ "${NEOHIRO_APT_HTTPS_STRICT:-0}" = "1" ]; then + err "apt-get update failed after the https rewrite and NEOHIRO_APT_HTTPS_STRICT=1." + err "Nothing was rolled back. Fix the repo URLs, or re-run with --apt-https-off." + return 1 + fi + warn "apt-get update failed after the https rewrite — rolling back to the backups." + _apt_https_revert_apt_sources >/dev/null + warn "Reverted. A mirror does not serve the same paths over HTTPS." + info "Repoint that repo at an https-capable mirror, then re-run --apt-https." + return 1 +} + +# apt_https_enforce [label] — apply the precaution once, unconditionally. +# [label] is a short breadcrumb describing the caller, e.g. "pkg_install". +apt_https_enforce() { + local label="${1:-enforce}" mode fam conf_failed findings rc=0 + mode="${NEOHIRO_APT_HTTPS:-1}" + + fam="$(apt_https_family)" + if [ "$fam" = "none" ]; then + info "No supported package manager detected — HTTPS guard not applicable." + return 0 + fi + + if [ "$mode" = "0" ]; then + info "NEOHIRO_APT_HTTPS=0 — repository transport guard disabled ($label)." + return 0 + fi + + msg "Repository transport guard ($label): $fam" + + if [ "$mode" = "audit" ]; then + info "NEOHIRO_APT_HTTPS=audit — reporting only, nothing is modified." + apt_https_report || true + return 0 + fi + + if ! _apt_https_privileged_ok; then + warn "Not root and no sudo available — cannot enforce HTTPS for repositories." + info "Re-run as root, or set NEOHIRO_APT_HTTPS=0 to silence this." + return 0 + fi + + case "$fam" in + apt) + conf_failed=0 + _apt_https_write_conf || conf_failed=1 + _apt_https_rewrite_apt_sources + if [ "$_APT_HTTPS_CHANGED" -gt 0 ]; then + ok "Rewrote $_APT_HTTPS_CHANGED apt source file(s) to https://" + _apt_https_verify_apt_sources || rc=1 + elif [ "$_APT_HTTPS_REFUSED" -gt 0 ]; then + warn "Could not rewrite $_APT_HTTPS_REFUSED apt source file(s): no writable backup." + info "Check that $(apt_https_backup_dir) is writable by root, then re-run." + rc=1 + else + ok "apt repositories already use https://" + fi + if [ "$conf_failed" = "1" ]; then + warn "Could not install the apt policy drop-in." + rc=1 + fi + ;; + dnf|yum|zypper|pacman) + if [ "${NEOHIRO_APT_HTTPS_REWRITE:-0}" = "1" ]; then + _apt_https_rewrite_family "$fam" + if [ "$_APT_HTTPS_CHANGED" -gt 0 ]; then + ok "Rewrote $_APT_HTTPS_CHANGED $fam repo file(s) to https:// (NEOHIRO_APT_HTTPS_REWRITE=1)" + info "Re-check with --apt-https-audit; not every mirror serves the same paths over TLS." + fi + fi + ;; + *) : ;; + esac + + # A verification failure rolls the rewrite back; report whatever remains. + findings="$(apt_https_status_text)" + if [ -n "$findings" ]; then + warn "Plaintext repository URLs remain for $fam:" + printf '%s\n' "$findings" | sed 's/^/ /' + if [ "$fam" != "apt" ]; then + info "Point those at an https-capable mirror, or opt in to a blind rewrite" + info "with NEOHIRO_APT_HTTPS_REWRITE=1 (verify afterwards)." + [ "${NEOHIRO_APT_HTTPS_STRICT:-0}" = "1" ] && rc=1 + fi + fi + + _apt_https_check_ca_certs "$fam" || true + _apt_https_block_port80 + return "$rc" +} + +# apt_https_guard [label] — hot-path wrapper. Enforces at most once per +# process so every package entry point can call it unconditionally. +apt_https_guard() { + [ -n "$_APT_HTTPS_DONE" ] && return 0 + # Latch first: the verification `apt-get update` below can re-enter the + # package layer, which must not recurse. + _APT_HTTPS_DONE=1 + apt_https_enforce "${1:-guard}" + return 0 +} + +# apt_https_revert — undo everything this lib changed. +apt_https_revert() { + local conf n + conf="$(apt_https_conf_file)" + _apt_https_revert_apt_sources + n="$_APT_HTTPS_REVERTED" + if [ -f "$conf" ] && [ -f "$(_apt_https_backup_path "$conf")" ]; then + if _apt_priv cp "$(_apt_https_backup_path "$conf")" "$conf"; then + info "Restored $conf from backup" + fi + elif [ -f "$conf" ]; then + # No backup means we created it: removing it is the correct revert. + if _apt_priv rm -f "$conf"; then + ok "Removed $conf" + fi + fi + if [ "$_APT_HTTPS_REVERTED" -gt 0 ] 2>/dev/null; then + ok "Reverted $_APT_HTTPS_REVERTED repository file(s) to their pre-guard contents." + fi + _APT_HTTPS_DONE="" + ok "Repository transport guard disabled." + return 0 +} + +# ── Standalone entry point ─────────────────────────────────────────────────── +# sudo bash lib/apt-https.sh enforce +# sudo bash lib/apt-https.sh --report audit only +# sudo bash lib/apt-https.sh --revert undo +if [ "${BASH_SOURCE[0]:-$0}" = "$0" ]; then + case "${1:---enforce}" in + --report|--audit) apt_https_report ;; + --revert) apt_https_revert ;; + --enforce) apt_https_enforce "cli" ;; + *) printf 'usage: bash lib/apt-https.sh [--enforce|--report|--revert]\n' >&2; exit 2 ;; + esac +fi \ No newline at end of file diff --git a/lib/updater.sh b/lib/updater.sh index a527af8..aa46545 100644 --- a/lib/updater.sh +++ b/lib/updater.sh @@ -163,10 +163,21 @@ _cmd() { run "$@" } +# ── Repository transport guard ──────────────────────────────────────────────── +# Sourced here (after the print helpers and `run` exist) so the update engine +# can never fetch a package over plaintext HTTP, whether it is run standalone +# (`sudo bash lib/updater.sh`) or called from linuxinstall.sh. The guard is a +# no-op when it has already run in this process. +# shellcheck disable=SC1091 +if [ -r "$(dirname "${BASH_SOURCE[0]:-$0}")/apt-https.sh" ]; then + source "$(dirname "${BASH_SOURCE[0]:-$0}")/apt-https.sh" +fi + # ── Package managers ───────────────────────────────────────────────────────── _update_apt() { command -v apt >/dev/null 2>&1 || return 0 + apt_https_guard "_update_apt" || true msg "apt: updating package lists..." if ! run sudo env DEBIAN_FRONTEND=noninteractive apt-get update -qq; then err "apt update failed"; _track; return 1 @@ -192,6 +203,7 @@ _update_apt() { _update_dnf() { command -v dnf >/dev/null 2>&1 || return 0 + apt_https_guard "_update_dnf" || true msg "dnf: checking for updates..." if ! run sudo dnf upgrade --refresh -y -q; then err "dnf upgrade failed"; _track; return 1 @@ -203,6 +215,7 @@ _update_dnf() { _update_yum() { command -v yum >/dev/null 2>&1 || return 0 + apt_https_guard "_update_yum" || true msg "yum: checking for updates..." if ! run sudo yum update -y -q; then err "yum update failed"; _track; return 1 @@ -214,6 +227,7 @@ _update_yum() { _update_zypper() { command -v zypper >/dev/null 2>&1 || return 0 + apt_https_guard "_update_zypper" || true msg "zypper: refreshing + updating..." if ! run sudo zypper --quiet refresh; then err "zypper refresh failed"; _track; return 1 @@ -228,6 +242,7 @@ _update_zypper() { _update_pacman() { command -v pacman >/dev/null 2>&1 || return 0 + apt_https_guard "_update_pacman" || true msg "pacman: syncing + upgrading..." if ! run sudo pacman -Syu --noconfirm --quiet; then err "pacman update failed"; _track; return 1 @@ -242,6 +257,7 @@ _update_pacman() { _update_snap() { command -v snap >/dev/null 2>&1 || return 0 + apt_https_guard "_update_snap" || true msg "snap: refreshing all snaps..." # Track snap refresh outcome so the dispatcher summary is accurate. if _cmd sudo snap refresh; then @@ -285,6 +301,7 @@ _update_snap() { _update_flatpak() { command -v flatpak >/dev/null 2>&1 || return 0 + apt_https_guard "_update_flatpak" || true msg "flatpak: updating remote repos + all installations..." local rc=0 # `flatpak remote-ls --updates` exits 0 whether or not there are updates @@ -347,6 +364,7 @@ _update_docker() { _update_brew() { command -v brew >/dev/null 2>&1 || return 0 + apt_https_guard "_update_brew" || true msg "brew: updating..." local brew_failed=0 if ! HOMEBREW_NO_ANALYTICS=1 run brew update 2>/dev/null; then @@ -619,6 +637,10 @@ _update_pihole() { # VERBOSE=2 — trace every command # DRY_RUN=1 — simulate all commands without running them # +# Repository transport: apt_https_guard runs before the first sub-step, so +# nothing in this dispatcher can pull a package over plaintext HTTP. It is +# idempotent, so the per-_update_* guards cost nothing. +# # Exit: 0 = all succeeded, 1 = one or more sub-steps had errors. _run_all_updates() { @@ -632,6 +654,9 @@ _run_all_updates() { esac done + # Precaution before a single byte is downloaded. + apt_https_guard "_run_all_updates" || true + msg "=== Comprehensive system update ===" local start_sec=$SECONDS @@ -763,6 +788,26 @@ else # Called as a script. Pass all arguments to _run_all_updates. set -euo pipefail SECONDS=0 + # Sub-commands handled here rather than by the dispatcher, because they + # only touch the repository transport and never run an update. + case "${1:-}" in + --apt-https|--enforce-https) + if [ "${2:-}" = "--audit" ]; then + apt_https_report + else + apt_https_enforce "updater --apt-https" + fi + exit $? + ;; + --apt-https-audit) + apt_https_report + exit $? + ;; + --apt-https-off|--disable-https) + apt_https_revert + exit $? + ;; + esac _run_all_updates "$@" exit $? fi diff --git a/linuxinstall.sh b/linuxinstall.sh index 444c072..4e41c89 100644 --- a/linuxinstall.sh +++ b/linuxinstall.sh @@ -40,6 +40,10 @@ if [ -n "$_NEOHIRO_LIB_DIR" ] && [ -r "$_NEOHIRO_LIB_DIR/color.sh" ]; then if [ -r "$_NEOHIRO_LIB_DIR/updater.sh" ]; then source "$_NEOHIRO_LIB_DIR/updater.sh" fi + # shellcheck disable=SC1091 + if [ -r "$_NEOHIRO_LIB_DIR/apt-https.sh" ]; then + source "$_NEOHIRO_LIB_DIR/apt-https.sh" + fi else # Inline fallback for run-from-pipe (curl ... | bash) where the lib # directory is not on disk. Sources the canonical color-gate function from @@ -158,6 +162,7 @@ if ! declare -F _run_all_updates >/dev/null 2>&1; then VERBOSE="${VERBOSE:-0}"; UPDATED=0; FAILED=0 _log() { [ "$VERBOSE" = "1" ] && info "$*" || true; } _update_apt() { command -v apt >/dev/null 2>&1 || return 0 + apt_https_guard "_update_apt" || true run sudo env DEBIAN_FRONTEND=noninteractive apt-get update -qq || return 1 run sudo env DEBIAN_FRONTEND=noninteractive apt-get -y -qq full-upgrade run sudo env DEBIAN_FRONTEND=noninteractive apt-get -y autoremove -qq @@ -208,6 +213,380 @@ if ! declare -F _run_all_updates >/dev/null 2>&1; then printf '\n'; ok "Update engine complete."; } fi +# Inline fallback for lib/apt-https.sh (curl|bash path). Same public API as +# the canonical library: apt_https_guard / apt_https_enforce / +# apt_https_report / apt_https_revert. Keep the function names in sync with +# lib/apt-https.sh - tests/test_apt_https.sh asserts parity between the two. +if ! declare -F apt_https_guard >/dev/null 2>&1; then + _APT_HTTPS_DONE="${_APT_HTTPS_DONE:-}" + APT_HTTPS_CONF_NAME="99neohiro-force-https" + apt_https_etc_dir() { printf '%s' "${NEOHIRO_APT_ETC_DIR:-/etc}"; } + apt_https_backup_dir() { printf '%s' "${NEOHIRO_APT_BACKUP_DIR:-/var/backups/neohiro-apt-https}"; } + apt_https_conf_file() { printf '%s' "$(apt_https_etc_dir)/apt/apt.conf.d/${APT_HTTPS_CONF_NAME}"; } + _apt_https_is_root() { [ "${EUID:-$(id -u 2>/dev/null || echo 1000)}" = "0" ]; } + _apt_priv() { + if [ "${DRY_RUN:-0}" = "1" ]; then printf ' DRY: %s\n' "$*"; return 0; fi + if _apt_https_is_root; then "$@"; return $?; fi + if command -v sudo >/dev/null 2>&1; then sudo "$@"; return $?; fi + return 1 + } + _apt_https_privileged_ok() { + _apt_https_is_root && return 0 + command -v sudo >/dev/null 2>&1 && return 0 + return 1 + } + apt_https_family() { + if [ -n "${NEOHIRO_APT_FAMILY:-}" ]; then printf '%s' "$NEOHIRO_APT_FAMILY"; return 0; fi + if command -v pacman >/dev/null 2>&1 && [ -f /etc/pacman.conf ]; then printf 'pacman' + elif command -v zypper >/dev/null 2>&1; then printf 'zypper' + elif command -v dnf >/dev/null 2>&1; then printf 'dnf' + elif command -v yum >/dev/null 2>&1; then printf 'yum' + elif command -v apt-get >/dev/null 2>&1 || [ -f /etc/apt/sources.list ]; then printf 'apt' + else printf 'none'; fi + } + _apt_https_repo_files() { + local etc f + etc="$(apt_https_etc_dir)" + [ -f "${etc}/apt/sources.list" ] && printf '%s\n' "${etc}/apt/sources.list" + for f in "${etc}"/apt/sources.list.d/*.list "${etc}"/apt/sources.list.d/*.sources; do + [ -f "$f" ] && printf '%s\n' "$f" + done + return 0 + } + _apt_https_foreign_repo_files() { + local etc f + etc="$(apt_https_etc_dir)" + case "$1" in + dnf|yum) for f in "${etc}"/yum.repos.d/*.repo; do [ -f "$f" ] && printf '%s\n' "$f"; done ;; + zypper) for f in "${etc}"/zypp/repos.d/*.repo; do [ -f "$f" ] && printf '%s\n' "$f"; done ;; + pacman) for f in "${etc}"/pacman.d/*; do [ -f "$f" ] && printf '%s\n' "$f"; done ;; + *) : ;; + esac + return 0 + } + _apt_https_plaintext_in_apt_file() { + local f="$1" + [ -f "$f" ] || return 0 + case "$f" in + *.sources) awk '/^[[:space:]]*URIs:[[:space:]]/ && /http:\/\// { gsub(/^[[:space:]]+/, "", $0); printf "%s:%d: %s\n", FILENAME, NR, $0 }' "$f" 2>/dev/null ;; + *) awk '/^[[:space:]]*#/ { next } /http:\/\// { gsub(/^[[:space:]]+/, "", $0); printf "%s:%d: %s\n", FILENAME, NR, $0 }' "$f" 2>/dev/null ;; + esac + return 0 + } + _apt_https_plaintext_in_repo_file() { + local f="$1" pat="$2" + [ -f "$f" ] || return 0 + awk -v pat="$pat" '/^[[:space:]]*#/ { next } $0 ~ pat && /http:\/\// { + line = $0; gsub(/^[[:space:]]+/, "", line); printf "%s:%d: %s\n", FILENAME, NR, line }' "$f" 2>/dev/null + return 0 + } + _apt_https_backup_path() { + local flat + flat="$(printf '%s' "$1" | tr '/' '_')" + printf '%s/%s.orig' "$(apt_https_backup_dir)" "$flat" + } + _apt_https_backup_once() { + local src="$1" dst + [ -f "$src" ] || return 0 + dst="$(_apt_https_backup_path "$src")" + [ -f "$dst" ] && return 0 + _apt_priv mkdir -p "$(apt_https_backup_dir)" || return 1 + _apt_priv cp -p "$src" "$dst" || { warn "Could not back up $src - refusing to edit it."; return 1; } + if declare -F record_backup >/dev/null 2>&1; then record_backup "$src" "$dst" 2>/dev/null || true; fi + info "Backed up $src -> $dst" + return 0 + } + _apt_https_stage() { + if declare -F _tmpfile >/dev/null 2>&1; then _tmpfile apt-https; return 0; fi + mktemp "${TMPDIR:-/tmp}/neohiro-apt-https.XXXXXX" + } + _apt_https_write_conf() { + local conf tmp rc + conf="$(apt_https_conf_file)" + tmp="$(_apt_https_stage)" + { + printf '%s\n' '// Managed by neohiro/linux (lib/apt-https.sh). Do not edit.' + printf '%s\n' '// Re-running the installer overwrites this file safely;' + printf '%s\n' '// `bash linuxinstall.sh --apt-https-off` removes it.' + printf '%s\n' '' + printf '%s\n' '// Prefer TLS, and never silently downgrade to plaintext.' + printf '%s\n' 'Acquire::https::AllowRedirect "true";' + printf '%s\n' 'Acquire::http::AllowRedirect "false";' + printf '%s\n' 'Acquire::ftp::AllowRedirect "false";' + printf '%s\n' '' + printf '%s\n' '// Authenticate the mirror, not just the channel.' + printf '%s\n' 'Acquire::https::Verify-Peer "true";' + printf '%s\n' 'Acquire::https::Verify-Host "true";' + printf '%s\n' '' + printf '%s\n' '// Transparent mirrors usually need more than one try.' + printf '%s\n' 'Acquire::Retries "3";' + printf '%s\n' '' + printf '%s\n' '// Pin a corporate proxy by uncommenting and editing:' + printf '%s\n' '// Acquire::http::Proxy "http://proxy.corp.example:3128";' + printf '%s\n' '// Acquire::https::Proxy "http://proxy.corp.example:3128";' + } > "$tmp" + if [ -f "$conf" ] && cmp -s "$tmp" "$conf"; then rm -f "$tmp"; return 0; fi + if ! _apt_https_backup_once "$conf"; then rm -f "$tmp"; return 1; fi + _apt_priv mkdir -p "$(apt_https_etc_dir)/apt/apt.conf.d" || { rm -f "$tmp"; return 1; } + if _apt_priv install -m 0644 "$tmp" "$conf"; then + rc=0 + elif _apt_priv cp "$tmp" "$conf" && _apt_priv chmod 0644 "$conf"; then + rc=0 + else + rc=1 + fi + rm -f "$tmp" + [ "$rc" = "0" ] || return 1 + info "apt policy: $conf" + return 0 + } + # Result globals, not stdout: these helpers print progress messages, and a + # command substitution would fold that text into the value it returns. + _APT_HTTPS_VERDICT="" + _APT_HTTPS_CHANGED=0 + _APT_HTTPS_REVERTED=0 + _APT_HTTPS_REFUSED=0 + _apt_https_rewrite_file() { + local f="$1" tmp expr + _APT_HTTPS_VERDICT="clean" + [ -f "$f" ] || return 0 + case "$f" in + *.sources) expr='/^[[:space:]]*URIs:[[:space:]]/ s|http://|https://|g' ;; + *) expr='/^[[:space:]]*#/! s|http://|https://|g' ;; + esac + tmp="$(_apt_https_stage)" + if ! sed -E "$expr" "$f" > "$tmp" 2>/dev/null; then rm -f "$tmp"; _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)); return 0; fi + if cmp -s "$tmp" "$f"; then rm -f "$tmp"; return 0; fi + if ! _apt_https_backup_once "$f"; then rm -f "$tmp"; _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)); return 0; fi + if ! _apt_priv cp "$tmp" "$f"; then rm -f "$tmp"; _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)); return 0; fi + rm -f "$tmp" + _APT_HTTPS_VERDICT="changed" + return 0 + } + _apt_https_rewrite_apt_sources() { + local f + _APT_HTTPS_CHANGED=0 + _APT_HTTPS_REFUSED=0 + while IFS= read -r f; do + [ -n "$f" ] || continue + _apt_https_rewrite_file "$f" + if [ "$_APT_HTTPS_VERDICT" = "changed" ]; then + _APT_HTTPS_CHANGED=$((_APT_HTTPS_CHANGED + 1)) + info "Rewrote plaintext repo URLs -> https: $f" + fi + done < <(_apt_https_repo_files) + return 0 + } + _apt_https_revert_apt_sources() { + local f bak backupdir + _APT_HTTPS_REVERTED=0 + backupdir="$(apt_https_backup_dir)" + [ -d "$backupdir" ] || return 0 + while IFS= read -r f; do + [ -n "$f" ] || continue + bak="$(_apt_https_backup_path "$f")" + if [ -f "$bak" ] && ! cmp -s "$bak" "$f" && _apt_priv cp "$bak" "$f"; then + _APT_HTTPS_REVERTED=$((_APT_HTTPS_REVERTED + 1)) + info "Restored $f from backup" + fi + done < <(_apt_https_repo_files) + return 0 + } + _apt_https_audit_family() { + local fam="$1" f + case "$fam" in + apt) while IFS= read -r f; do [ -n "$f" ] && _apt_https_plaintext_in_apt_file "$f"; done < <(_apt_https_repo_files) ;; + dnf|yum) while IFS= read -r f; do [ -n "$f" ] && _apt_https_plaintext_in_repo_file "$f" '^([[:space:]]*)(baseurl|metalink|mirrorlist)[[:space:]]*='; done < <(_apt_https_foreign_repo_files "$fam") ;; + zypper) while IFS= read -r f; do [ -n "$f" ] && _apt_https_plaintext_in_repo_file "$f" '^([[:space:]]*)(baseurl|uri|mirrorlist)[[:space:]]*='; done < <(_apt_https_foreign_repo_files "$fam") ;; + pacman) while IFS= read -r f; do [ -n "$f" ] && _apt_https_plaintext_in_repo_file "$f" '^([[:space:]]*)Server[[:space:]]*='; done < <(_apt_https_foreign_repo_files "$fam") ;; + *) : ;; + esac + return 0 + } + _apt_https_rewrite_family() { + local fam="$1" f + _APT_HTTPS_CHANGED=0 + case "$fam" in dnf|yum|zypper|pacman) : ;; *) return 0 ;; esac + while IFS= read -r f; do + [ -n "$f" ] || continue + _apt_https_rewrite_file "$f" + [ "$_APT_HTTPS_VERDICT" = "changed" ] && _APT_HTTPS_CHANGED=$((_APT_HTTPS_CHANGED + 1)) + done < <(_apt_https_foreign_repo_files "$fam") + return 0 + } + _apt_https_flatpak_plaintext() { + command -v flatpak >/dev/null 2>&1 || return 0 + flatpak remotes --show-details 2>/dev/null | awk '/http:\/\// { print }' || true + return 0 + } + _apt_https_check_ca_certs() { + local fam="$1" + case "$fam" in + apt) { [ -e /etc/ssl/certs/ca-certificates.crt ] || [ -e /etc/pki/tls/certs/ca-bundle.crt ]; } || { warn "No system CA trust store found; HTTPS package verification will fail."; return 1; } ;; + dnf|yum) [ -e /etc/pki/tls/certs/ca-bundle.crt ] || { warn "No system CA trust store found."; return 1; } ;; + zypper) [ -e /etc/ssl/ca-bundle.pem ] || { warn "No system CA trust store found."; return 1; } ;; + pacman) [ -e /etc/ssl/certs/ca-certificates.crt ] || { warn "No system CA trust store found."; return 1; } ;; + *) : ;; + esac + return 0 + } + _apt_https_block_port80() { + [ "${NEOHIRO_APT_BLOCK_PORT80:-0}" = "1" ] || return 0 + command -v ufw >/dev/null 2>&1 || { warn "NEOHIRO_APT_BLOCK_PORT80=1 but no UFW found; skipping."; return 0; } + if ufw status 2>/dev/null | grep -qE '^80/tcp[[:space:]]+DENY[[:space:]]+OUT'; then + info "ufw already denies outbound 80/tcp"; return 0 + fi + if [ -n "$(apt_https_status_text)" ]; then + warn "Skipping the port-80 block: plaintext repos are still configured." + return 0 + fi + if _apt_priv ufw deny out 80/tcp; then + ok "Blocked outbound TCP/80 (apt traffic is HTTPS-only from now on)." + info "Undo: sudo ufw delete deny out 80/tcp" + else + warn "Could not add the ufw outbound 80/tcp deny rule." + fi + return 0 + } + apt_https_status_text() { + _apt_https_audit_family "$(apt_https_family)" + _apt_https_flatpak_plaintext + return 0 + } + apt_https_report() { + local fam conf findings + fam="$(apt_https_family)" + printf '\n%s\n' "$(_c '1;36m' '━━━ Package transport security (HTTPS) ━━━')" + printf ' Detected package manager: %s\n' "$fam" + case "$fam" in + apt) + conf="$(apt_https_conf_file)" + if [ -f "$conf" ]; then printf ' %s %s\n' "$(_c '1;32m' '[x]')" "apt policy drop-in active: $conf" + else printf ' %s %s\n' "$(_c '1;31m' '[ ]')" "apt policy drop-in MISSING: $conf"; fi ;; + dnf|yum) printf ' %s\n' "dnf/yum: repo files audited (no blind rewrite)." ;; + zypper) printf ' %s\n' "zypper: repo files audited (no blind rewrite)." ;; + pacman) printf ' %s\n' "pacman: mirrorlist audited (no blind rewrite)." ;; + *) printf ' %s\n' "No supported package manager detected." ;; + esac + findings="$(apt_https_status_text)" + if [ -n "$findings" ]; then + printf '\n %s\n' "$(_c '1;33m' 'Plaintext (http://) repository entries still present:')" + printf '%s\n' "$findings" | sed 's/^/ /' + printf '\n %s\n' " apt: fix with sudo bash linuxinstall.sh --apt-https" + printf '%s\n' " others: set NEOHIRO_APT_HTTPS_REWRITE=1, then re-run the audit" + else + printf '\n %s %s\n' "$(_c '1;32m' '[x]')" "No plaintext repository URLs detected." + fi + _apt_https_check_ca_certs "$fam" || true + printf '\n' + if [ "$fam" = "none" ]; then return 2; fi + [ -n "$findings" ] && return 1 + return 0 + } + _apt_https_verify_apt_sources() { + [ "${NEOHIRO_APT_HTTPS_NOVERIFY:-0}" = "1" ] && return 0 + [ "${DRY_RUN:-0}" = "1" ] && return 0 + command -v apt-get >/dev/null 2>&1 || return 0 + info "Verifying the rewritten sources with a real apt-get update..." + if _apt_priv env DEBIAN_FRONTEND=noninteractive apt-get update -qq; then + ok "All apt repositories answered over HTTPS."; return 0 + fi + if [ "${NEOHIRO_APT_HTTPS_STRICT:-0}" = "1" ]; then + err "apt-get update failed after the https rewrite and NEOHIRO_APT_HTTPS_STRICT=1." + return 1 + fi + warn "apt-get update failed after the https rewrite - rolling back to the backups." + _apt_https_revert_apt_sources >/dev/null + warn "Reverted. A mirror does not serve the same paths over HTTPS." + return 1 + } + apt_https_enforce() { + local label="${1:-enforce}" mode fam conf_failed findings rc=0 + mode="${NEOHIRO_APT_HTTPS:-1}" + fam="$(apt_https_family)" + if [ "$fam" = "none" ]; then + info "No supported package manager detected - HTTPS guard not applicable." + return 0 + fi + if [ "$mode" = "0" ]; then + info "NEOHIRO_APT_HTTPS=0 - repository transport guard disabled ($label)." + return 0 + fi + msg "Repository transport guard ($label): $fam" + if [ "$mode" = "audit" ]; then + info "NEOHIRO_APT_HTTPS=audit - reporting only, nothing is modified." + apt_https_report || true + return 0 + fi + if ! _apt_https_privileged_ok; then + warn "Not root and no sudo available - cannot enforce HTTPS for repositories." + info "Re-run as root, or set NEOHIRO_APT_HTTPS=0 to silence this." + return 0 + fi + case "$fam" in + apt) + conf_failed=0 + _apt_https_write_conf || conf_failed=1 + _apt_https_rewrite_apt_sources + if [ "$_APT_HTTPS_CHANGED" -gt 0 ]; then + ok "Rewrote $_APT_HTTPS_CHANGED apt source file(s) to https://" + _apt_https_verify_apt_sources || rc=1 + elif [ "$_APT_HTTPS_REFUSED" -gt 0 ]; then + warn "Could not rewrite $_APT_HTTPS_REFUSED apt source file(s): no writable backup." + rc=1 + else + ok "apt repositories already use https://" + fi + [ "$conf_failed" = "1" ] && { warn "Could not install the apt policy drop-in."; rc=1; } + ;; + dnf|yum|zypper|pacman) + if [ "${NEOHIRO_APT_HTTPS_REWRITE:-0}" = "1" ]; then + _apt_https_rewrite_family "$fam" + [ "$_APT_HTTPS_CHANGED" -gt 0 ] && ok "Rewrote $_APT_HTTPS_CHANGED $fam repo file(s) to https:// (NEOHIRO_APT_HTTPS_REWRITE=1)" + fi + ;; + *) : ;; + esac + findings="$(apt_https_status_text)" + if [ -n "$findings" ]; then + warn "Plaintext repository URLs remain for $fam:" + printf '%s\n' "$findings" | sed 's/^/ /' + if [ "$fam" != "apt" ]; then + info "Point those at an https-capable mirror, or opt in to a blind rewrite" + info "with NEOHIRO_APT_HTTPS_REWRITE=1 (verify afterwards)." + [ "${NEOHIRO_APT_HTTPS_STRICT:-0}" = "1" ] && rc=1 + fi + fi + _apt_https_check_ca_certs "$fam" || true + _apt_https_block_port80 + return "$rc" + } + apt_https_guard() { + [ -n "$_APT_HTTPS_DONE" ] && return 0 + _APT_HTTPS_DONE=1 + apt_https_enforce "${1:-guard}" + return 0 + } + apt_https_revert() { + local conf bak n + conf="$(apt_https_conf_file)" + bak="$(_apt_https_backup_path "$conf")" + _apt_https_revert_apt_sources + n="$_APT_HTTPS_REVERTED" + if [ -f "$conf" ] && [ -f "$bak" ]; then + _apt_priv cp "$bak" "$conf" && info "Restored $conf from backup" + elif [ -f "$conf" ]; then + _apt_priv rm -f "$conf" && ok "Removed $conf" + fi + if [ "${n:-0}" -gt 0 ] 2>/dev/null; then + ok "Reverted $n repository file(s) to their pre-guard contents." + fi + _APT_HTTPS_DONE="" + ok "Repository transport guard disabled." + return 0 + } +fi + RECOVERY_CMD="tmux attach -t linux-setup # reconnect after SSH disconnect" ROLLBACK_LOG="${ROLLBACK_LOG:-/var/log/linux-install-rollback.log}" @@ -873,6 +1252,9 @@ detect_distro() { } pkg_update() { + # Precaution: no repository traffic leaves this host in plaintext. Runs + # once per process, so the four pkg_* helpers below cost nothing extra. + apt_https_guard "pkg_update" || true case "$PKG_MGR" in apt) run sudo env DEBIAN_FRONTEND=noninteractive apt-get update ;; dnf) info "Checking for updates (dnf check-update)..." @@ -892,6 +1274,7 @@ pkg_update() { } pkg_install() { + apt_https_guard "pkg_install" || true case "$PKG_MGR" in apt) run sudo env DEBIAN_FRONTEND=noninteractive apt-get install -y "$@" ;; dnf) run sudo dnf install -y "$@" ;; @@ -903,6 +1286,7 @@ pkg_install() { } pkg_upgrade() { + apt_https_guard "pkg_upgrade" || true case "$PKG_MGR" in apt) run sudo env DEBIAN_FRONTEND=noninteractive apt-get -y full-upgrade ;; dnf) run sudo dnf upgrade --refresh -y ;; @@ -914,6 +1298,7 @@ pkg_upgrade() { } pkg_autoremove() { + apt_https_guard "pkg_autoremove" || true case "$PKG_MGR" in apt) run sudo env DEBIAN_FRONTEND=noninteractive apt-get -y autoremove ;; dnf) run sudo dnf autoremove -y ;; @@ -1210,6 +1595,7 @@ _BAR_Filled='#'; _BAR_Empty='-' declare -A CHECKLIST=( [tmux_wrap]=pending [env_detect]=pending + [apt_https]=pending [system]=pending [system_update]=pending [dns]=pending @@ -1232,6 +1618,7 @@ declare -A CHECKLIST=( ) CHECKLIST_LABEL_tmux_wrap="Auto-wrap SSH session in tmux" CHECKLIST_LABEL_env_detect="Detect environment (desktop/server)" +CHECKLIST_LABEL_apt_https="Force HTTPS for all package repos" CHECKLIST_LABEL_system="System update + base packages" CHECKLIST_LABEL_system_update="System update + base packages" CHECKLIST_LABEL_dns="DNSCrypt + DNS routing" @@ -1299,7 +1686,7 @@ _should_run_step() { # ask_category_enabled() in main(), otherwise --step will be rejected as # "unknown" even for legitimate steps. Aliases (e.g. system_update, # ssh_hardening) are accepted alongside the short keys for convenience. -_VALID_STEPS="system system_update dns dnscrypt firewall tor ssh ssh_hardening fail2ban unattended ipv6 sysctl apparmor pam optimize optimize_asr deepclean" +_VALID_STEPS="system system_update dns dnscrypt apt_https firewall tor ssh ssh_hardening fail2ban unattended ipv6 sysctl apparmor pam optimize optimize_asr deepclean" _valid_step() { case " $_VALID_STEPS " in *" $1 "*) return 0 ;; esac return 1 @@ -1308,6 +1695,7 @@ _valid_step() { # Short preview text for each step — printed before the step runs. # Keep entries under ~70 chars so the terminal doesn't wrap. declare -A _STEP_PREVIEWS +_STEP_PREVIEWS["apt_https"]="force HTTPS for every apt/repo source; blocks plaintext downgrade." _STEP_PREVIEWS["system"]="apt update + upgrade, install base packages (curl, fail2ban, etc.)." _STEP_PREVIEWS["system_update"]="apt update + upgrade, install base packages (curl, fail2ban, etc.)." _STEP_PREVIEWS["dns"]="install dnscrypt-proxy; you choose between DoH, DNSCrypt, and DoT." @@ -1326,7 +1714,7 @@ _STEP_PREVIEWS["optimize"]="run OptimizeLinuxASR.sh (network/disk tweaks). Rever _STEP_PREVIEWS["optimize_asr"]="run OptimizeLinuxASR.sh (network/disk tweaks). Reversible." _STEP_PREVIEWS["deepclean"]="run DeepClean.sh (apt cache, journal, old kernels). Safe but uses disk." -_CHECKLIST_ORDER="tmux_wrap env_detect system_update dnscrypt firewall tor ssh_hardening fail2ban unattended ipv6 sysctl apparmor pam optimize_asr deepclean other_scripts summary" +_CHECKLIST_ORDER="tmux_wrap env_detect apt_https system_update dnscrypt firewall tor ssh_hardening fail2ban unattended ipv6 sysctl apparmor pam optimize_asr deepclean other_scripts summary" show_progress() { local done=0 total=0 key status @@ -1588,7 +1976,7 @@ ask_profile() { printf ' %s %s\n' "$(_c '1;31m' '6) Restore SSH')" "$(_c '1;37m' 'Diagnose & fix common SSH lockout causes; offers self-heal guard.')" printf ' %s %s\n' "" "$(_c '1;30m' 'Safe recovery tool — re-run anytime without re-hardening.')" printf '\n' - printf ' %s %s\n' "$(_c '1;1;35m' '7) Maintenance')" "$(_c '1;37m' '20 individual tools: inspect, update, recover, optimize.')" + printf ' %s %s\n' "$(_c '1;1;35m' '7) Maintenance')" "$(_c '1;37m' '21 individual tools: HTTPS guard, inspect, update, recover, optimize.')" printf ' %s %s\n' "" "$(_c '1;30m' 'Persistent menu — go in and out without restarting.')" printf '\n' printf ' %s %s\n' "$(_c '1;32m' '8) DeepClean')" "$(_c '1;37m' 'Run DeepClean.sh: journal, logs, apt/dnf/pacman cache,')" @@ -1608,7 +1996,7 @@ ask_profile() { "Full (includes Tor, IPv6 disable, attack-surface reduction, deep clean)" \ "Custom (individual per-category prompts)" \ "Restore SSH (diagnose & fix common lockout causes)" \ - "Maintenance suite (20 individual tools, return to this menu)" \ + "Maintenance suite (21 individual tools, return to this menu)" \ "DeepClean (cleanup + auto-prune; no hardening)" \ "Attack Surface Reduction (OptimizeLinuxASR.sh; service-by-service)" \ "Updates only (system update + kernel + optional packages + auto-update)" @@ -1624,6 +2012,15 @@ ask_profile() { _auto_skip_if_done() { local key="$1" case "$key" in + apt_https) + # Idempotent: if nothing plaintext is left, the guard has already run + # (a repo added since would show up here again and force a re-run). + if [ -z "$(apt_https_status_text)" ]; then + info "[AUTO] Repository sources are already HTTPS-only — skipping." + return 0 + fi + return 1 + ;; dnscrypt) if pkg_is_installed dnscrypt-proxy && systemctl is-active --quiet dnscrypt-proxy 2>/dev/null; then info "[AUTO] dnscrypt-proxy already installed and running — skipping." @@ -1711,6 +2108,14 @@ _auto_skip_if_done() { ask_category_enabled() { local key="$1" desc="$2" default="$3" _should_run_step "$key" || return 1 + # Repository-transport hardening is a precaution, not an optional + # hardening choice: it is enabled for every profile, including Custom, + # so there is no profile in which apt can fetch over plaintext HTTP. + # NEOHIRO_APT_HTTPS=0 remains the documented escape hatch. + if [ "$key" = "apt_https" ]; then + _auto_skip_if_done "$key" && return 0 + return 0 + fi # AUTO_MODE: every step uses its per-profile default (Standard for desktop, # Full for server). No interactive prompts; decisions are logged by prompt_yn. # Also skip any step that is already fully applied (idempotent re-runs). @@ -1845,13 +2250,15 @@ _maintenance_sysinfo() { } # Maintenance menu: expanded suite for SSH recovery, config inspection, key -# management, self-heal controls, and system diagnostics. -# The menu uses a magenta header and numbered choices (20 = exit). +# management, self-heal controls, and system diagnostics. Option 1 forces +# HTTPS for every package repository before anything is downloaded. +# The menu uses a magenta header and numbered choices (21 = exit). maintenance_menu() { local choice rc while true; do printf '\n%s\n' "$(_c '1;35m' '━━━ Maintenance suite ━━━')" - prompt_choice "Maintenance — pick a category (20=exit)" \ + prompt_choice "Maintenance — pick a category (21=exit)" \ + "Force HTTPS for package repos (precaution before any download)" \ "System update (apt update + upgrade + base packages)" \ "DNSCrypt (DNS encryption)" \ "Firewall (UFW)" \ @@ -1875,23 +2282,24 @@ maintenance_menu() { choice=$REPLY_CHOICE rc=0 case $choice in - 0) mark_step system_update "running"; show_progress; update_system || rc=$?; mark_step system_update "$([ $rc -eq 0 ] && echo done || echo skip)"; update_kernel || rc=$?;; - 1) mark_step dnscrypt "running"; show_progress; setup_dnscrypt || rc=$?; mark_step dnscrypt "$([ $rc -eq 0 ] && echo done || echo skip)";; - 2) mark_step firewall "running"; show_progress; setup_firewall || rc=$?; mark_step firewall "$([ $rc -eq 0 ] && echo done || echo skip)";; - 3) mark_step tor "running"; show_progress; setup_tor || rc=$?; mark_step tor "$([ $rc -eq 0 ] && echo done || echo skip)";; - 4) mark_step ssh_hardening "running"; show_progress; harden_ssh || rc=$?; mark_step ssh_hardening "$([ $rc -eq 0 ] && echo done || echo skip)";; - 5) mark_step fail2ban "running"; show_progress; setup_fail2ban || rc=$?; mark_step fail2ban "$([ $rc -eq 0 ] && echo done || echo skip)";; - 6) mark_step unattended "running"; show_progress; configure_unattended_upgrades || rc=$?; mark_step unattended "$([ $rc -eq 0 ] && echo done || echo skip)";; - 7) mark_step ipv6 "running"; show_progress; disable_ipv6 || rc=$?; mark_step ipv6 "$([ $rc -eq 0 ] && echo done || echo skip)";; - 8) mark_step sysctl "running"; show_progress; harden_sysctl || rc=$?; mark_step sysctl "$([ $rc -eq 0 ] && echo done || echo skip)";; - 9) mark_step apparmor "running"; show_progress; setup_apparmor || rc=$?; mark_step apparmor "$([ $rc -eq 0 ] && echo done || echo skip)";; - 10) mark_step pam "running"; show_progress; harden_passwords || rc=$?; mark_step pam "$([ $rc -eq 0 ] && echo done || echo skip)";; - 11) mark_step optimize_asr "running"; show_progress; run_optimize_asr || rc=$?; mark_step optimize_asr "$([ $rc -eq 0 ] && echo done || echo skip)";; - 12) mark_step deepclean "running"; show_progress; run_deepclean || rc=$?; mark_step deepclean "$([ $rc -eq 0 ] && echo done || echo skip)";; - 13) bold "=== SSH diagnostics & lockout fix ==="; restore_ssh_mode || rc=$?;; - 14) bold "=== Authorized keys ==="; _maintenance_list_keys;; - 15) bold "=== SSH config review ==="; _ssh_inspect_config || rc=$?;; - 16) bold "=== SSH self-heal guard ===" + 0) mark_step apt_https "running"; show_progress; apt_https_enforce "maintenance menu" || rc=$?; apt_https_report || true; mark_step apt_https "$([ $rc -eq 0 ] && echo done || echo skip)";; + 1) mark_step system_update "running"; show_progress; update_system || rc=$?; mark_step system_update "$([ $rc -eq 0 ] && echo done || echo skip)"; update_kernel || rc=$?;; + 2) mark_step dnscrypt "running"; show_progress; setup_dnscrypt || rc=$?; mark_step dnscrypt "$([ $rc -eq 0 ] && echo done || echo skip)";; + 3) mark_step firewall "running"; show_progress; setup_firewall || rc=$?; mark_step firewall "$([ $rc -eq 0 ] && echo done || echo skip)";; + 4) mark_step tor "running"; show_progress; setup_tor || rc=$?; mark_step tor "$([ $rc -eq 0 ] && echo done || echo skip)";; + 5) mark_step ssh_hardening "running"; show_progress; harden_ssh || rc=$?; mark_step ssh_hardening "$([ $rc -eq 0 ] && echo done || echo skip)";; + 6) mark_step fail2ban "running"; show_progress; setup_fail2ban || rc=$?; mark_step fail2ban "$([ $rc -eq 0 ] && echo done || echo skip)";; + 7) mark_step unattended "running"; show_progress; configure_unattended_upgrades || rc=$?; mark_step unattended "$([ $rc -eq 0 ] && echo done || echo skip)";; + 8) mark_step ipv6 "running"; show_progress; disable_ipv6 || rc=$?; mark_step ipv6 "$([ $rc -eq 0 ] && echo done || echo skip)";; + 9) mark_step sysctl "running"; show_progress; harden_sysctl || rc=$?; mark_step sysctl "$([ $rc -eq 0 ] && echo done || echo skip)";; + 10) mark_step apparmor "running"; show_progress; setup_apparmor || rc=$?; mark_step apparmor "$([ $rc -eq 0 ] && echo done || echo skip)";; + 11) mark_step pam "running"; show_progress; harden_passwords || rc=$?; mark_step pam "$([ $rc -eq 0 ] && echo done || echo skip)";; + 12) mark_step optimize_asr "running"; show_progress; run_optimize_asr || rc=$?; mark_step optimize_asr "$([ $rc -eq 0 ] && echo done || echo skip)";; + 13) mark_step deepclean "running"; show_progress; run_deepclean || rc=$?; mark_step deepclean "$([ $rc -eq 0 ] && echo done || echo skip)";; + 14) bold "=== SSH diagnostics & lockout fix ==="; restore_ssh_mode || rc=$?;; + 15) bold "=== Authorized keys ==="; _maintenance_list_keys;; + 16) bold "=== SSH config review ==="; _ssh_inspect_config || rc=$?;; + 17) bold "=== SSH self-heal guard ===" if [ -f /var/run/neohiro-ssh-watchdog.armed ]; then info "Self-heal guard is installed and armed." if prompt_yn "Remove the self-heal guard?" "n"; then @@ -1904,9 +2312,9 @@ maintenance_menu() { fi fi ;; - 17) bold "=== Log viewer ==="; _maintenance_logs;; - 18) bold "=== System info ==="; _maintenance_sysinfo;; - 19) info "Returning to main menu."; break;; + 18) bold "=== Log viewer ==="; _maintenance_logs;; + 19) bold "=== System info ==="; _maintenance_sysinfo;; + 20) info "Returning to main menu."; break;; esac if [ $rc -eq 0 ]; then ok "Done." @@ -1920,6 +2328,7 @@ maintenance_menu() { update_system() { msg "Updating system and installing base packages..." _warn_if_not_tmux + apt_https_guard "update_system" || true local upgradable=0 case "$PKG_MGR" in apt) upgradable=$(apt list --upgradable 2>/dev/null | grep -c '/') || upgradable=0 ;; @@ -1951,6 +2360,7 @@ update_system() { update_kernel() { msg "System and kernel update..." _warn_if_not_tmux + apt_https_guard "update_kernel" || true if [ "$EUID" -ne 0 ] && ! command -v sudo >/dev/null 2>&1; then err "sudo required for kernel update." @@ -3258,6 +3668,25 @@ run_deepclean() { # about enabling automatic (security) updates. Never touches firewall, # SSH, sysctl, AppArmor, or any hardening. Use this for a "just sync # the system and have the latest binaries" pass. +# Force HTTPS for every configured package repository, then report the +# resulting state. Shared by the main workflow step, the maintenance menu, +# and the --apt-https CLI flag so all three behave identically. +_step_apt_https() { + local rc=0 + # `if ! cmd; then rc=$?; fi` would capture the negation's status, not the + # command's, so use the explicit `|| rc=$?` form. + apt_https_enforce "step:apt_https" || rc=$? + # Count any backups the guard took so the run summary stays honest. + local bdir n=0 b + bdir="$(apt_https_backup_dir)" + if [ -d "$bdir" ]; then + for b in "$bdir"/*.orig; do [ -f "$b" ] && n=$((n + 1)); done + fi + [ "$n" -gt 0 ] && metrics_add configs_backed_up "$n" + apt_https_report || true + return "$rc" +} + updates_only_mode() { bold "=== Updates-only mode ===" info "This runs the comprehensive update engine (every available package" @@ -3266,6 +3695,10 @@ updates_only_mode() { info "enables automatic (security) updates. No firewall, SSH, or hardening." echo + # Precaution before anything is fetched: force repository traffic over + # HTTPS (and roll back automatically if a mirror cannot speak TLS). + apt_https_guard "updates_only_mode" || true + # 1) Comprehensive system update — every package manager + snap/flatpak # + docker images + brew + firmware. Same engine lives in lib/updater.sh # and is also runnable standalone: sudo bash lib/updater.sh @@ -3883,6 +4316,21 @@ main() { rollback_mode "${1#--rollback=}" exit $? ;; + --apt-https|--enforce-https) + bold "neohiro/linux - HTTPS package repositories (enforce)" + _step_apt_https + exit $? + ;; + --apt-https-audit) + bold "neohiro/linux - HTTPS package repositories (audit only)" + apt_https_report + exit $? + ;; + --apt-https-off|--disable-https) + bold "neohiro/linux - Revert HTTPS package repository enforcement" + apt_https_revert + exit $? + ;; --dry-run) DRY_RUN=1; shift bold "[DRY-RUN] Preview mode - no changes will be made" @@ -3914,7 +4362,7 @@ main() { ;; -h|--help) cat <<'USAGE' -Usage: sudo bash linuxinstall.sh [--auto|-y] [--dry-run] [--step STEP] [--restore-ssh] [--restore-etc-snapshot] [--rollback [--apply]] [-h] +Usage: sudo bash linuxinstall.sh [--auto|-y] [--dry-run] [--step STEP] [--restore-ssh] [--restore-etc-snapshot] [--rollback [--apply]] [--apt-https[=audit|off]] [-h] (no flag) Run the full interactive setup & hardening. --auto, -y, --yes Run unattended: intelligent profile selection, no prompts, @@ -3940,6 +4388,19 @@ Usage: sudo bash linuxinstall.sh [--auto|-y] [--dry-run] [--step STEP] [--restor --rollback Dry-prints the inverse cp commands needed to undo every change recorded in /var/log/linux-install-rollback.log. --rollback --apply Run those cp commands (latest backup wins). + --apt-https Force HTTPS for every configured package repository + (writes /etc/apt/apt.conf.d/99neohiro-force-https and + rewrites http:// repo URLs to https://). Runs + automatically before every apt/dnf/yum/zypper/pacman + download/update/upgrade, so this flag is only needed to + apply it on its own. Backups land in + /var/backups/neohiro-apt-https/. A mirror that cannot + speak HTTPS is detected and the rewrite is rolled back + automatically. Alias: --enforce-https + --apt-https-audit Report the current transport security of every repo + without changing anything (exit 1 if plaintext remains). + --apt-https-off Revert: restore every backed-up repo file and remove + the apt policy drop-in. Alias: --disable-https -h, --help Show this help. Environment variables: @@ -3950,6 +4411,20 @@ Environment variables: TOR_NICK=... Override the Tor relay nickname (default: hostname). TOR_CONTACT=... Override the Tor relay contact (default: you@example.com). NEOHIRO_DEBUG_LOG=path Override debug log location. + +Repository transport guard (applies to every package manager): + NEOHIRO_APT_HTTPS=1 Enforce HTTPS. Default. + NEOHIRO_APT_HTTPS=audit Report only; never modify anything. + NEOHIRO_APT_HTTPS=0 Disable the guard entirely. + NEOHIRO_APT_HTTPS_REWRITE=1 Allow http->https rewrite on dnf/yum/zypper/ + pacman repo files (off by default; not every + mirror serves the same paths over TLS). + NEOHIRO_APT_HTTPS_NOVERIFY=1 Skip the post-rewrite `apt-get update` check. + NEOHIRO_APT_HTTPS_STRICT=1 Treat leftover plaintext repos as a hard error. + NEOHIRO_APT_BLOCK_PORT80=1 Also add `ufw deny out 80/tcp` so no process can + open a plaintext package connection at all. + Off by default because a blanket outbound + block also affects unrelated protocols. USAGE exit 0 ;; @@ -4062,13 +4537,17 @@ USAGE done } # Workflow order, from most aggressive cleaning to most targeted update: - # 1) Attack-surface reduction FIRST — stop unneeded services so they are + # 0) HTTPS enforcement for every package repository. This runs FIRST and + # on every profile so nothing below can fetch a package over plaintext + # HTTP (a MITM on the mirror can inject arbitrary .deb/.rpm files). + # 1) Attack-surface reduction - stop unneeded services so they are # no longer holding files / kernel modules when we upgrade. - # 2) Comprehensive system update + kernel prune — every package manager, + # 2) Comprehensive system update + kernel prune - every package manager, # snap, flatpak, docker, brew, firmware, geoip. # 3) Hardening layer: firewall, SSH, fail2ban, sysctl, AppArmor, ... # Rationale: dropping unwanted services first means the system-update # transaction does not need to chase daemons that are about to go away. + _run_step apt_https "Force HTTPS for all package repos" "" _step_apt_https y _run_step optimize_asr "Attack-surface reduction (stop unneeded services)" "" run_optimize_asr n # System step is special: it always follows with update_kernel so the # new kernel can be detected before the run ends. The comprehensive diff --git a/restore_ssh.sh b/restore_ssh.sh index 627159d..6735aa9 100644 --- a/restore_ssh.sh +++ b/restore_ssh.sh @@ -22,6 +22,13 @@ if [ -r "$(dirname "$(readlink -f "${BASH_SOURCE[0]:-$0}")")/lib/color.sh" ]; th source "$(dirname "$(readlink -f "${BASH_SOURCE[0]:-$0}")")/lib/color.sh" fi +# Repository transport guard: installing openssh-server is a package +# download, so HTTPS is enforced first (no-op once per process). +# shellcheck disable=SC1091 +if [ -r "$(dirname "$(readlink -f "${BASH_SOURCE[0]:-$0}")")/lib/apt-https.sh" ]; then + source "$(dirname "$(readlink -f "${BASH_SOURCE[0]:-$0}")")/lib/apt-https.sh" +fi + # --- print helpers (format preserved from original restore_ssh.sh) --- # If lib/color.sh was sourced, USE_COLOR is set but C_* constants are not # (lib/color.sh defines _c, not C_RED/C_GRN/etc.). Run the gate once and @@ -111,6 +118,10 @@ detect_pkg_mgr() { PKG_MGR=$(detect_pkg_mgr) pkg_install_ssh() { + # Precaution: never fetch openssh over a plaintext mirror. + if declare -F apt_https_guard >/dev/null 2>&1; then + apt_https_guard "restore_ssh:pkg_install_ssh" || true + fi case "$PKG_MGR" in apt) run sudo env DEBIAN_FRONTEND=noninteractive apt-get install -y openssh-server ;; dnf) run sudo dnf install -y openssh-server ;; @@ -365,6 +376,27 @@ apply_fixes() { } main() { + # Repository-transport sub-commands: these never touch SSH at all, so + # they short-circuit before diagnose/apply_fixes. + case "${1:-}" in + --apt-https|--enforce-https) + bold "neohiro/linux - HTTPS package repositories (enforce)" + require_root + apt_https_enforce "restore_ssh --apt-https" + apt_https_report || true + return $? + ;; + --apt-https-audit) + apt_https_report + return $? + ;; + --apt-https-off|--disable-https) + bold "neohiro/linux - Revert HTTPS package repository enforcement" + require_root + apt_https_revert + return $? + ;; + esac bold "neohiro/linux - Restore SSH (standalone)" require_root diagnose diff --git a/tests/run-all.sh b/tests/run-all.sh index 5d9e72e..cee9d99 100644 --- a/tests/run-all.sh +++ b/tests/run-all.sh @@ -41,6 +41,7 @@ SUITES=( "fuzz_smoke|tests/test_color_fuzz.sh|50" "color_fuzz|tests/test_color_fuzz.sh|200" "linuxinstall|tests/test_linuxinstall.sh|" + "apt_https|tests/test_apt_https.sh|" "updater|tests/test_updater.sh|" "verify_sync|verify_sync.sh|" ) diff --git a/tests/test_apt_https.sh b/tests/test_apt_https.sh new file mode 100644 index 0000000..e651fb6 --- /dev/null +++ b/tests/test_apt_https.sh @@ -0,0 +1,785 @@ +#!/usr/bin/env bash +# tests/test_apt_https.sh - hermetic unit tests for the repository transport +# guard (lib/apt-https.sh). +# +# Nothing here touches the real /etc, /var/backups, or any package manager: +# +# * NEOHIRO_APT_ETC_DIR / NEOHIRO_APT_BACKUP_DIR relocate the tree the lib +# reads and writes into a temp sandbox. +# * After sourcing the lib we replace apt_https_family() so the suite +# behaves identically on Ubuntu, in the bash:4 / bash:latest docker +# images used by CI, and on a developer laptop with no package manager. +# * We replace _apt_priv() and _apt_https_privileged_ok() so the suite +# behaves as if it were root, without needing root. The real _apt_priv is +# exercised separately through its DRY_RUN path in a subshell. +# * A fake `ufw` on PATH makes the optional port-80 block deterministic. +# * NEOHIRO_APT_HTTPS_NOVERIFY=1 means the verification `apt-get update` +# never runs, so no network call is ever made. +# +# Run: bash tests/test_apt_https.sh +set -u +PASS=0; FAIL=0 +if [ -t 1 ]; then + C_RED=$'\033[1;31m'; C_GRN=$'\033[1;32m'; C_RST=$'\033[0m' +else C_RED=""; C_GRN=""; C_RST=""; fi + +ok_t() { printf ' %s[OK] %s%s\n' "$C_GRN" "$1" "$C_RST"; PASS=$((PASS+1)); } +fail_t() { printf ' %s[FAIL]%s %s\n %s\n' "$C_RED" "$C_RST" "$1" "$2"; FAIL=$((FAIL+1)); } + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +LIB="$ROOT/lib/apt-https.sh" +SRC="$ROOT/linuxinstall.sh" +[ -f "$LIB" ] || { echo "lib/apt-https.sh not found at $LIB"; exit 2; } +[ -f "$SRC" ] || { echo "linuxinstall.sh not found at $SRC"; exit 2; } + +WD="$(mktemp -d)" +_TMP_STAGED=() +cleanup() { rm -rf "$WD" 2>/dev/null || true; rm -f "${_TMP_STAGED[@]}" 2>/dev/null || true; } +trap cleanup EXIT + +SANDBOX="$WD/etc" +BACKUPS="$WD/backups" +FAKEBIN="$WD/bin" +mkdir -p "$FAKEBIN" + +# Fake ufw: `ufw status` reports nothing installed yet; any mutating call is +# recorded so a test can assert it happened (or did not). +cat > "$FAKEBIN/ufw" <<'FAKEEOF' +#!/bin/sh +case "${1:-}" in + status) exit 0 ;; + *) echo "$*" >> "$FAKE_UFW_LOG"; exit 0 ;; +esac +FAKEEOF +chmod +x "$FAKEBIN/ufw" +export FAKE_UFW_LOG="$WD/ufw-calls.log" +export PATH="$FAKEBIN:$PATH" + +# --- source the library, THEN install the stubs ------------------------------ +# Order matters: the lib defines _apt_priv and apt_https_family itself, so +# anything we want to fake has to be replaced afterwards. +# shellcheck disable=SC1090 +. "$LIB" + +# _tmpfile is what the lib prefers for scratch files; a tracked version keeps +# everything inside the temp dir. +_tmpfile() { + local f + f="$(mktemp "$WD/stage.XXXXXX")" || return 1 + _TMP_STAGED+=("$f") + printf '%s' "$f" +} +# Simulate root without needing root. +_apt_priv() { "$@"; } +_apt_https_privileged_ok() { return 0; } +# Force the family per-test via TEST_FAMILY. Note the lib's own +# NEOHIRO_APT_FAMILY override is what the CLI paths use; this stub exists so +# the unit tests can flip families without re-exporting on every reset. +apt_https_family() { printf '%s' "${TEST_FAMILY:-apt}"; } + +reset_sandbox() { + TEST_FAMILY=apt + rm -rf "$SANDBOX" "$BACKUPS" 2>/dev/null || true + mkdir -p "$SANDBOX/apt/apt.conf.d" "$SANDBOX/apt/sources.list.d" "$BACKUPS" + export NEOHIRO_APT_ETC_DIR="$SANDBOX" + export NEOHIRO_APT_BACKUP_DIR="$BACKUPS" + export NEOHIRO_APT_HTTPS_NOVERIFY=1 + unset NEOHIRO_APT_HTTPS NEOHIRO_APT_HTTPS_REWRITE NEOHIRO_APT_FAMILY \ + NEOHIRO_APT_HTTPS_STRICT NEOHIRO_APT_BLOCK_PORT80 2>/dev/null || true + _APT_HTTPS_DONE="" + DRY_RUN=0 +} + +# ============================================================================ +# NEOHIRO_APT_FAMILY pin (this is what the CLI paths use) +# ============================================================================ +reset_sandbox +PINNED="$(env NEOHIRO_APT_FAMILY=zypper bash -c ". '$LIB' >/dev/null 2>&1; apt_https_family" 2>/dev/null)" +if [ "$PINNED" = "zypper" ]; then + ok_t "NEOHIRO_APT_FAMILY pins the detected family" +else + fail_t "NEOHIRO_APT_FAMILY pins the detected family" "got: ${PINNED:-}" +fi + +if grep -q 'NEOHIRO_APT_FAMILY' "$SRC" && grep -q 'NEOHIRO_APT_FAMILY' "$LIB"; then + ok_t "NEOHIRO_APT_FAMILY is honoured by the inline fallback too" +else + fail_t "NEOHIRO_APT_FAMILY is honoured by the inline fallback too" "missing in one of the two copies" +fi + +# ============================================================================ +# Path helpers +# ============================================================================ +reset_sandbox + +if [ "$(apt_https_etc_dir)" = "$SANDBOX" ]; then + ok_t "apt_https_etc_dir honors NEOHIRO_APT_ETC_DIR" +else + fail_t "apt_https_etc_dir honors NEOHIRO_APT_ETC_DIR" "got: $(apt_https_etc_dir)" +fi + +if [ "$(apt_https_backup_dir)" = "$BACKUPS" ]; then + ok_t "apt_https_backup_dir honors NEOHIRO_APT_BACKUP_DIR" +else + fail_t "apt_https_backup_dir honors NEOHIRO_APT_BACKUP_DIR" "got: $(apt_https_backup_dir)" +fi + +EXPECT_CONF="$SANDBOX/apt/apt.conf.d/99neohiro-force-https" +if [ "$(apt_https_conf_file)" = "$EXPECT_CONF" ]; then + ok_t "apt_https_conf_file points at the managed drop-in" +else + fail_t "apt_https_conf_file points at the managed drop-in" "got: $(apt_https_conf_file)" +fi + +# Default paths must be the real system paths when the overrides are unset. +# These run in a subshell so the sandbox exports do not leak. +DEFAULT_ETC="$(env -u NEOHIRO_APT_ETC_DIR bash -c ". '$LIB' >/dev/null 2>&1; apt_https_etc_dir" 2>/dev/null)" +if [ "$DEFAULT_ETC" = "/etc" ]; then + ok_t "apt_https_etc_dir defaults to /etc" +else + fail_t "apt_https_etc_dir defaults to /etc" "got: ${DEFAULT_ETC:-}" +fi + +DEFAULT_BAK="$(env -u NEOHIRO_APT_BACKUP_DIR bash -c ". '$LIB' >/dev/null 2>&1; apt_https_backup_dir" 2>/dev/null)" +if [ "$DEFAULT_BAK" = "/var/backups/neohiro-apt-https" ]; then + ok_t "apt_https_backup_dir defaults to /var/backups/neohiro-apt-https" +else + fail_t "apt_https_backup_dir defaults to /var/backups/neohiro-apt-https" "got: ${DEFAULT_BAK:-}" +fi + +# ============================================================================ +# Classic sources.list rewriting +# ============================================================================ +reset_sandbox +cat > "$SANDBOX/apt/sources.list" <<'EOF' +deb http://archive.ubuntu.com/ubuntu noble main restricted +# deb http://commented.example.com/ubuntu noble main +deb-src https://already.secure.example.com/ubuntu noble main +deb [arch=amd64] http://mirror.internal:8080/debian bookworm main +EOF +_apt_https_rewrite_apt_sources +if [ "$_APT_HTTPS_CHANGED" = "1" ]; then + ok_t "classic rewrite reports 1 changed file" +else + fail_t "classic rewrite reports 1 changed file" "got N=$_APT_HTTPS_CHANGED" +fi + +if grep -q '^deb https://archive.ubuntu.com/ubuntu noble main restricted$' "$SANDBOX/apt/sources.list"; then + ok_t "classic: http:// deb line rewritten to https://" +else + fail_t "classic: http:// deb line rewritten to https://" "line not rewritten" +fi + +if grep -q '^# deb http://commented.example.com' "$SANDBOX/apt/sources.list"; then + ok_t "classic: commented-out http:// line left untouched" +else + fail_t "classic: commented-out http:// line left untouched" "comment was modified" +fi + +if grep -q '^deb-src https://already.secure.example.com' "$SANDBOX/apt/sources.list"; then + ok_t "classic: already-https line unchanged" +else + fail_t "classic: already-https line unchanged" "line was altered" +fi + +if grep -q '^deb \[arch=amd64\] https://mirror.internal:8080/debian' "$SANDBOX/apt/sources.list"; then + ok_t "classic: bracketed options preserved during rewrite" +else + fail_t "classic: bracketed options preserved during rewrite" "options mangled" +fi + +BAK_REL="$(printf '%s' "$SANDBOX" | tr '/' '_')" +BAK_REL="${BAK_REL#/}" +BAK="$BACKUPS/${BAK_REL}_apt_sources.list.orig" +if [ -f "$BAK" ]; then + ok_t "classic: original file was backed up before editing" +else + fail_t "classic: original file was backed up before editing" \ + "no .orig at $BAK (found: $(ls -1 "$BACKUPS" 2>/dev/null | tr '\n' ' '))" +fi + +if [ -f "$BAK" ] && grep -q '^deb http://archive.ubuntu.com' "$BAK"; then + ok_t "classic: backup holds the PRE-edit (plaintext) contents" +else + fail_t "classic: backup holds the PRE-edit (plaintext) contents" \ + "backup does not contain the original http:// line" +fi + +# Second pass must be a no-op: idempotency is what keeps re-runs cheap and +# keeps AUTO_MODE from churning backups. +_apt_https_rewrite_apt_sources +if [ "$_APT_HTTPS_CHANGED" = "0" ]; then + ok_t "classic rewrite is idempotent (second pass changes nothing)" +else + fail_t "classic rewrite is idempotent (second pass changes nothing)" "got N2=$_APT_HTTPS_CHANGED" +fi + +# ============================================================================ +# DEB822 (.sources) rewriting +# ============================================================================ +reset_sandbox +cat > "$SANDBOX/apt/sources.list.d/debian.sources" <<'EOF' +Types: deb +URIs: http://deb.debian.org/debian bookworm main +# http://disabled.example.com/debian bookworm main +Suites: bookworm +Components: main contrib +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg +EOF +_apt_https_rewrite_apt_sources >/dev/null + +if grep -q '^URIs: https://deb.debian.org/debian' "$SANDBOX/apt/sources.list.d/debian.sources"; then + ok_t "DEB822: URIs: field rewritten to https://" +else + fail_t "DEB822: URIs: field rewritten to https://" "not rewritten" +fi + +if grep -q '^# http://disabled.example.com' "$SANDBOX/apt/sources.list.d/debian.sources"; then + ok_t "DEB822: commented http:// left untouched" +else + fail_t "DEB822: commented http:// left untouched" "comment was modified" +fi + +if grep -v '^#' "$SANDBOX/apt/sources.list.d/debian.sources" | grep -q 'http://'; then + fail_t "DEB822: no active plaintext URI remains" "still plaintext" +else + ok_t "DEB822: no active plaintext URI remains" +fi + +if grep -q '^Suites: bookworm$' "$SANDBOX/apt/sources.list.d/debian.sources" && \ + grep -q '^Components: main contrib$' "$SANDBOX/apt/sources.list.d/debian.sources"; then + ok_t "DEB822: Suites/Components fields preserved" +else + fail_t "DEB822: Suites/Components fields preserved" "structural fields changed" +fi + +# ============================================================================ +# apt policy drop-in content +# ============================================================================ +reset_sandbox +apt_https_enforce "test" >/dev/null 2>&1 +CONF="$(apt_https_conf_file)" +if [ -f "$CONF" ]; then + ok_t "enforce: apt policy drop-in installed" +else + fail_t "enforce: apt policy drop-in installed" "$CONF missing" +fi + +if [ -f "$CONF" ] && grep -q '^Acquire::http::AllowRedirect "false";' "$CONF"; then + ok_t "policy: refuses https->http downgrade redirects" +else + fail_t "policy: refuses https->http downgrade redirects" 'Acquire::http::AllowRedirect "false" missing' +fi + +if [ -f "$CONF" ] && grep -q '^Acquire::https::AllowRedirect "true";' "$CONF"; then + ok_t "policy: allows https->https mirror redirects" +else + fail_t "policy: allows https->https mirror redirects" 'Acquire::https::AllowRedirect "true" missing' +fi + +if [ -f "$CONF" ] && grep -q '^Acquire::https::Verify-Peer "true";' "$CONF" && \ + grep -q '^Acquire::https::Verify-Host "true";' "$CONF"; then + ok_t "policy: peer + host verification enabled" +else + fail_t "policy: peer + host verification enabled" "Verify-Peer/Verify-Host missing" +fi + +if [ -f "$CONF" ] && grep -q '^Acquire::Retries "3";' "$CONF"; then + ok_t "policy: retries configured for flaky mirrors" +else + fail_t "policy: retries configured for flaky mirrors" "Acquire::Retries missing" +fi + +# Security by default: the drop-in must not be group/world writable. +if [ -f "$CONF" ]; then + PERM="$(ls -l "$CONF" 2>/dev/null | cut -c1-10)" + case "$PERM" in + -rw-r--r--) ok_t "policy: drop-in mode is 0644" ;; + *) fail_t "policy: drop-in mode is 0644" "got: $PERM" ;; + esac +else + fail_t "policy: drop-in mode is 0644" "drop-in missing" +fi + +# Idempotency: a second enforce must not create extra backups. +BAKS_BEFORE="$(find "$BACKUPS" -name '*.orig' 2>/dev/null | wc -l)" +apt_https_enforce "test-2" >/dev/null 2>&1 +BAKS_AFTER="$(find "$BACKUPS" -name '*.orig' 2>/dev/null | wc -l)" +if [ "$BAKS_BEFORE" = "$BAKS_AFTER" ]; then + ok_t "enforce: re-running does not churn backups" +else + fail_t "enforce: re-running does not churn backups" "$BAKS_BEFORE -> $BAKS_AFTER" +fi + +# ============================================================================ +# Status / report exit codes +# ============================================================================ +reset_sandbox +printf 'deb https://secure.example.com/ubuntu noble main\n' > "$SANDBOX/apt/sources.list" +apt_https_report >/dev/null 2>&1 +RC=$? +if [ "$RC" = "0" ]; then + ok_t "apt_https_report rc=0 when every repo is https" +else + fail_t "apt_https_report rc=0 when every repo is https" "rc=$RC" +fi + +printf 'deb http://plaintext.example.com/ubuntu noble main\n' > "$SANDBOX/apt/sources.list" +apt_https_report >/dev/null 2>&1 +RC=$? +if [ "$RC" = "1" ]; then + ok_t "apt_https_report rc=1 when plaintext repos remain" +else + fail_t "apt_https_report rc=1 when plaintext repos remain" "rc=$RC" +fi + +if apt_https_status_text | grep -q 'plaintext.example.com'; then + ok_t "apt_https_status_text names the offending repo" +else + fail_t "apt_https_status_text names the offending repo" "offender not reported" +fi + +TEST_FAMILY=none +apt_https_report >/dev/null 2>&1 +RC=$? +if [ "$RC" = "2" ]; then + ok_t "apt_https_report rc=2 when no package manager is detected" +else + fail_t "apt_https_report rc=2 when no package manager is detected" "rc=$RC" +fi + +if apt_https_enforce "test" >/dev/null 2>&1; then + ok_t "enforce: no-op (rc=0) when no package manager is detected" +else + fail_t "enforce: no-op (rc=0) when no package manager is detected" "returned non-zero" +fi + +# ============================================================================ +# Env-var kill switches +# ============================================================================ +reset_sandbox +printf 'deb http://plaintext.example.com/ubuntu noble main\n' > "$SANDBOX/apt/sources.list" +export NEOHIRO_APT_HTTPS=0 +_APT_HTTPS_DONE="" +apt_https_guard "test-off" >/dev/null 2>&1 +if grep -q 'http://plaintext.example.com' "$SANDBOX/apt/sources.list" && \ + [ ! -f "$(apt_https_conf_file)" ]; then + ok_t "NEOHIRO_APT_HTTPS=0 disables the guard without touching anything" +else + fail_t "NEOHIRO_APT_HTTPS=0 disables the guard without touching anything" "files were modified" +fi + +reset_sandbox +printf 'deb http://plaintext.example.com/ubuntu noble main\n' > "$SANDBOX/apt/sources.list" +export NEOHIRO_APT_HTTPS=audit +_APT_HTTPS_DONE="" +apt_https_guard "test-audit" >/dev/null 2>&1 +if grep -q 'http://plaintext.example.com' "$SANDBOX/apt/sources.list" && \ + [ ! -f "$(apt_https_conf_file)" ]; then + ok_t "NEOHIRO_APT_HTTPS=audit reports without modifying" +else + fail_t "NEOHIRO_APT_HTTPS=audit reports without modifying" "files were modified" +fi + +# ============================================================================ +# once-per-process latch +# ============================================================================ +reset_sandbox +printf 'deb http://plaintext.example.com/ubuntu noble main\n' > "$SANDBOX/apt/sources.list" +_APT_HTTPS_DONE="" +apt_https_guard "first" >/dev/null 2>&1 +if [ -z "$(apt_https_status_text)" ]; then + ok_t "guard: first call enforces (latch set, work done)" +else + fail_t "guard: first call enforces (latch set, work done)" "repo still plaintext" +fi + +# Break the sources on purpose. A second guard call must NOT re-enforce: it +# is a hot-path helper called by every pkg_* helper, so it has to be cheap. +printf 'deb http://regressed.example.com/ubuntu noble main\n' > "$SANDBOX/apt/sources.list" +apt_https_guard "second" >/dev/null 2>&1 +if grep -q 'regressed.example.com' "$SANDBOX/apt/sources.list"; then + ok_t "guard: second call is a no-op (once per process)" +else + fail_t "guard: second call is a no-op (once per process)" "second call re-ran enforcement" +fi + +# The latch must be resettable so the CLI entry points can force a re-run. +_APT_HTTPS_DONE="" +apt_https_guard "third" >/dev/null 2>&1 +if [ -z "$(apt_https_status_text)" ]; then + ok_t "guard: latch is resettable so the CLI flags can re-run it" +else + fail_t "guard: latch is resettable so the CLI flags can re-run it" "regression not repaired" +fi + +# ============================================================================ +# Un-backuppable files must be refused, not corrupted +# ============================================================================ +reset_sandbox +printf 'deb http://nope.example.com/ubuntu noble main\n' > "$SANDBOX/apt/sources.list" +# Make the backup path a regular file so `mkdir -p` cannot succeed. +rm -rf "$BACKUPS" +: > "$BACKUPS" +_APT_HTTPS_DONE="" +REFUSE_OUT="$(apt_https_enforce "no-backup" 2>&1)" +if grep -q 'http://nope.example.com' "$SANDBOX/apt/sources.list"; then + ok_t "refuses to edit a repo file it cannot back up" +else + fail_t "refuses to edit a repo file it cannot back up" "file was edited with no backup" +fi +if ! printf '%s' "$REFUSE_OUT" | grep -q 'already use https://'; then + ok_t "reports a refusal instead of claiming 'already https'" +else + fail_t "reports a refusal instead of claiming 'already https'" \ + "misleading success message emitted" +fi +if printf '%s' "$REFUSE_OUT" | grep -q 'no writable backup'; then + ok_t "refusal explains why (no writable backup)" +else + fail_t "refusal explains why (no writable backup)" "no reason given" +fi +if apt_https_enforce "no-backup-rc" >/dev/null 2>&1; then + fail_t "enforce returns non-zero when it could not enforce" "rc=0" +else + ok_t "enforce returns non-zero when it could not enforce" +fi +rm -f "$BACKUPS" + +# ============================================================================ +# Revert +# ============================================================================ +reset_sandbox +ORIG='deb http://original.example.com/ubuntu noble main' +printf '%s\n' "$ORIG" > "$SANDBOX/apt/sources.list" +_APT_HTTPS_DONE="" +apt_https_enforce "revert-test" >/dev/null 2>&1 +if grep -q 'https://original.example.com' "$SANDBOX/apt/sources.list"; then + ok_t "revert: precondition - sources were rewritten to https" +else + fail_t "revert: precondition - sources were rewritten to https" "rewrite did not happen" +fi +apt_https_revert >/dev/null 2>&1 +if [ "$(cat "$SANDBOX/apt/sources.list")" = "$ORIG" ]; then + ok_t "revert: restores the pre-guard sources file byte for byte" +else + fail_t "revert: restores the pre-guard sources file byte for byte" \ + "got: $(cat "$SANDBOX/apt/sources.list")" +fi +if [ ! -f "$(apt_https_conf_file)" ]; then + ok_t "revert: removes the drop-in it created" +else + fail_t "revert: removes the drop-in it created" "drop-in still present" +fi + +# ============================================================================ +# dnf / yum / zypper / pacman: audit, no blind rewrite +# ============================================================================ +reset_sandbox +TEST_FAMILY=dnf +mkdir -p "$SANDBOX/yum.repos.d" +cat > "$SANDBOX/yum.repos.d/epel.repo" <<'EOF' +[epel] +name=Extra Packages +baseurl=http://mirrors.example.com/epel/9/Everything/x86_64/ +metalink=https://mirrors.example.com/metalink?repo=epel-9 +gpgcheck=1 +EOF + +if apt_https_status_text | grep -q 'mirrors.example.com/epel'; then + ok_t "dnf: plaintext baseurl detected" +else + fail_t "dnf: plaintext baseurl detected" "nothing reported" +fi + +# Default: audit only, no rewrite. +DNFOUT="$(apt_https_enforce "dnf-default" 2>&1)" +if grep -q '^baseurl=http://' "$SANDBOX/yum.repos.d/epel.repo"; then + ok_t "dnf: no blind rewrite by default" +else + fail_t "dnf: no blind rewrite by default" "file was rewritten without opt-in" +fi +if printf '%s' "$DNFOUT" | grep -q 'Plaintext repository URLs remain'; then + ok_t "dnf: leftover plaintext is reported loudly" +else + fail_t "dnf: leftover plaintext is reported loudly" "no warning emitted" +fi + +# Opt-in rewrite. +export NEOHIRO_APT_HTTPS_REWRITE=1 +_APT_HTTPS_DONE="" +apt_https_enforce "dnf-optin" >/dev/null 2>&1 +if grep -q '^baseurl=https://' "$SANDBOX/yum.repos.d/epel.repo"; then + ok_t "dnf: NEOHIRO_APT_HTTPS_REWRITE=1 rewrites baseurl" +else + fail_t "dnf: NEOHIRO_APT_HTTPS_REWRITE=1 rewrites baseurl" "not rewritten" +fi +if grep -q '^gpgcheck=1$' "$SANDBOX/yum.repos.d/epel.repo"; then + ok_t "dnf: rewrite leaves non-URL settings alone" +else + fail_t "dnf: rewrite leaves non-URL settings alone" "gpgcheck altered" +fi +unset NEOHIRO_APT_HTTPS_REWRITE + +reset_sandbox +TEST_FAMILY=zypper +mkdir -p "$SANDBOX/zypp/repos.d" +cat > "$SANDBOX/zypp/repos.d/oss.repo" <<'EOF' +[repo-oss] +name=openSUSE-Leap +baseurl=http://download.opensuse.org/repositories/oss/ +enabled=1 +EOF +if apt_https_status_text | grep -q 'download.opensuse.org'; then + ok_t "zypper: plaintext baseurl detected" +else + fail_t "zypper: plaintext baseurl detected" "nothing reported" +fi + +reset_sandbox +TEST_FAMILY=pacman +mkdir -p "$SANDBOX/pacman.d" +cat > "$SANDBOX/pacman.d/mirrorlist" <<'EOF' +## Worldwide +Server=http://mirror.example.com/archlinux/$repo/os/$arch +Server=https://secure.example.com/archlinux/$repo/os/$arch +EOF +if apt_https_status_text | grep -q 'http://mirror.example.com'; then + ok_t "pacman: plaintext Server= detected" +else + fail_t "pacman: plaintext Server= detected" "nothing reported" +fi +if ! apt_https_status_text | grep -q 'secure.example.com'; then + ok_t "pacman: already-https mirror is not flagged" +else + fail_t "pacman: already-https mirror is not flagged" "false positive" +fi + +# ============================================================================ +# DRY_RUN and the optional port-80 block +# ============================================================================ +reset_sandbox +# The real _apt_priv (not the stub) is exercised in a subshell so the DRY_RUN +# branch is covered without needing root. +DRY_OUT="$(env DRY_RUN=1 NEOHIRO_APT_ETC_DIR="$SANDBOX" NEOHIRO_APT_BACKUP_DIR="$BACKUPS" \ + bash -c ". '$LIB' >/dev/null 2>&1; _apt_priv ufw deny out 80/tcp" 2>&1)" +if printf '%s' "$DRY_OUT" | grep -q 'DRY: ufw deny out 80/tcp'; then + ok_t "_apt_priv honours DRY_RUN (prints instead of executing)" +else + fail_t "_apt_priv honours DRY_RUN (prints instead of executing)" "got: $DRY_OUT" +fi + +reset_sandbox +TEST_FAMILY=dnf +mkdir -p "$SANDBOX/yum.repos.d" +printf '[x]\nbaseurl=http://plain.example.com/repo/\n' > "$SANDBOX/yum.repos.d/x.repo" +export NEOHIRO_APT_BLOCK_PORT80=1 +_APT_HTTPS_DONE="" +: > "$FAKE_UFW_LOG" +BLOCK_OUT="$(apt_https_enforce "port80" 2>&1)" +if printf '%s' "$BLOCK_OUT" | grep -q 'Skipping the port-80 block'; then + ok_t "port-80 block is skipped while plaintext repos remain" +else + fail_t "port-80 block is skipped while plaintext repos remain" \ + "block attempted anyway: $(cat "$FAKE_UFW_LOG" 2>/dev/null | tr '\n' ' ')" +fi +if [ ! -s "$FAKE_UFW_LOG" ]; then + ok_t "port-80 block emits no ufw rule when it declines" +else + fail_t "port-80 block emits no ufw rule when it declines" "ufw was called" +fi + +reset_sandbox +printf 'deb https://secure.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +export NEOHIRO_APT_BLOCK_PORT80=1 +_APT_HTTPS_DONE="" +: > "$FAKE_UFW_LOG" +apt_https_enforce "port80-clean" >/dev/null 2>&1 +if grep -q 'deny out 80/tcp' "$FAKE_UFW_LOG" 2>/dev/null; then + ok_t "port-80 block proceeds once all repos are https" +else + fail_t "port-80 block proceeds once all repos are https" "no ufw rule emitted" +fi +unset NEOHIRO_APT_BLOCK_PORT80 + +# Default: the block must be completely opt-in. +reset_sandbox +: > "$FAKE_UFW_LOG" +_APT_HTTPS_DONE="" +apt_https_enforce "port80-default" >/dev/null 2>&1 +if [ ! -s "$FAKE_UFW_LOG" ]; then + ok_t "port-80 block is opt-in (no ufw rule by default)" +else + fail_t "port-80 block is opt-in (no ufw rule by default)" \ + "ufw called: $(cat "$FAKE_UFW_LOG" | tr '\n' ' ')" +fi + +# ============================================================================ +# Integration: every package entry point calls the guard +# ============================================================================ +GUARD_FNS="pkg_update pkg_install pkg_upgrade pkg_autoremove update_system update_kernel" +for fn in $GUARD_FNS; do + BODY="$(awk -v fn="$fn" ' + $0 ~ "^"fn"\\(\\) \\{" { inside = 1 } + inside { print } + inside && /^}/ { exit } + ' "$SRC")" + if printf '%s' "$BODY" | grep -q 'apt_https_guard'; then + ok_t "integration: $fn() calls apt_https_guard" + else + fail_t "integration: $fn() calls apt_https_guard" "no guard call found in $fn()" + fi +done + +# The guard must appear BEFORE the first package-manager invocation in the +# body, otherwise it is decorative. +for fn in pkg_update pkg_install pkg_upgrade pkg_autoremove; do + LINE_GUARD="$(grep -n "apt_https_guard \"$fn\"" "$SRC" | head -1 | cut -d: -f1)" + LINE_PKG="$(grep -n "^${fn}() {" "$SRC" | head -1 | cut -d: -f1)" + FIRST_USE="$(awk -v start="$LINE_PKG" 'NR>start && /apt-get|dnf |yum |zypper |pacman /{print NR; exit}' "$SRC")" + if [ -n "$LINE_GUARD" ] && [ -n "$FIRST_USE" ] && [ "$LINE_GUARD" -lt "$FIRST_USE" ]; then + ok_t "integration: $fn() guards before its first package command" + else + fail_t "integration: $fn() guards before its first package command" \ + "guard@${LINE_GUARD:-none} firstpkg@${FIRST_USE:-none}" + fi +done + +# lib/updater.sh must guard the dispatcher and every sub-step. +UP="$ROOT/lib/updater.sh" +for fn in _update_apt _update_dnf _update_yum _update_zypper _update_pacman \ + _update_snap _update_flatpak _update_brew _run_all_updates; do + if grep -q "apt_https_guard \"$fn\"" "$UP"; then + ok_t "integration: updater $fn() calls apt_https_guard" + else + fail_t "integration: updater $fn() calls apt_https_guard" "no guard call" + fi +done + +if grep -q 'apt-https.sh' "$UP" && grep -q 'source .*apt-https.sh' "$ROOT/restore_ssh.sh" && \ + grep -q 'source .*apt-https.sh' "$ROOT/DeepClean.sh" && \ + grep -q 'source .*apt-https.sh' "$SRC"; then + ok_t "integration: every entry-point script sources lib/apt-https.sh" +else + fail_t "integration: every entry-point script sources lib/apt-https.sh" "source line missing somewhere" +fi + +if grep -q 'apt_https_guard "restore_ssh:pkg_install_ssh"' "$ROOT/restore_ssh.sh"; then + ok_t "integration: restore_ssh.sh guards before installing openssh-server" +else + fail_t "integration: restore_ssh.sh guards before installing openssh-server" "no guard call" +fi + +if grep -q 'apt_https_guard "DeepClean"' "$ROOT/DeepClean.sh"; then + ok_t "integration: DeepClean.sh guards before its apt-get autoremove pass" +else + fail_t "integration: DeepClean.sh guards before its apt-get autoremove pass" "no guard call" +fi + +# Menus / CLI surfaces must be reachable. +if grep -q '_run_step apt_https' "$SRC"; then + ok_t "integration: apt_https is a first-class workflow step" +else + fail_t "integration: apt_https is a first-class workflow step" "step missing" +fi + +if grep -q 'apt_https_enforce "maintenance menu"' "$SRC"; then + ok_t "integration: maintenance submenu exposes the guard" +else + fail_t "integration: maintenance submenu exposes the guard" "submenu entry missing" +fi + +for flag in '--apt-https|--enforce-https' '--apt-https-audit' '--apt-https-off|--disable-https'; do + if grep -q -- "$flag)" "$SRC"; then + ok_t "integration: CLI flag parsed: $flag" + else + fail_t "integration: CLI flag parsed: $flag" "flag not handled in main()" + fi +done + +for flag in '--apt-https|--enforce-https' '--apt-https-audit' '--apt-https-off|--disable-https'; do + if grep -q -- "$flag)" "$UP"; then + ok_t "integration: updater standalone CLI flag parsed: $flag" + else + fail_t "integration: updater standalone CLI flag parsed: $flag" "flag missing" + fi +done + +if grep -q '_VALID_STEPS="[^"]*apt_https' "$SRC"; then + ok_t "integration: --step apt_https is a valid step key" +else + fail_t "integration: --step apt_https is a valid step key" "key missing from _VALID_STEPS" +fi + +if grep -q '\[apt_https\]=pending' "$SRC" && grep -q 'CHECKLIST_LABEL_apt_https=' "$SRC" && \ + grep -q '_CHECKLIST_ORDER="[^"]*apt_https' "$SRC"; then + ok_t "integration: progress checklist shows the guard step" +else + fail_t "integration: progress checklist shows the guard step" "checklist wiring incomplete" +fi + +# ============================================================================ +# Parity: linuxinstall.sh's inline fallback must expose the same API +# ============================================================================ +INLINE_FNS="apt_https_guard apt_https_enforce apt_https_report apt_https_revert \ +apt_https_family apt_https_status_text apt_https_etc_dir apt_https_backup_dir apt_https_conf_file \ +_apt_https_repo_files _apt_https_foreign_repo_files _apt_https_plaintext_in_apt_file \ +_apt_https_plaintext_in_repo_file _apt_https_backup_path _apt_https_backup_once _apt_https_stage \ +_apt_https_write_conf _apt_https_rewrite_file _apt_https_rewrite_apt_sources \ +_apt_https_revert_apt_sources _apt_https_audit_family _apt_https_rewrite_family \ +_apt_https_flatpak_plaintext _apt_https_check_ca_certs _apt_https_block_port80 _apt_priv \ +_apt_https_verify_apt_sources" +MISSING="" +for fn in $INLINE_FNS; do + grep -qE "^[[:space:]]+${fn}\(\)" "$SRC" || MISSING="$MISSING inline:$fn" + grep -qE "^${fn}\(\)" "$LIB" || MISSING="$MISSING lib:$fn" +done +if [ -z "$MISSING" ]; then + ok_t "inline fallback and lib/apt-https.sh expose the same API" +else + fail_t "inline fallback and lib/apt-https.sh expose the same API" "missing:$MISSING" +fi + +# ============================================================================ +# _step_apt_https exit-status plumbing +# ============================================================================ +# `if ! cmd; then rc=$?; fi` captures the *negation's* status, so a failed +# enforcement would silently report success. Assert the correct form is used. +if grep -q 'apt_https_enforce "step:apt_https" || rc=\$?' "$SRC"; then + ok_t "_step_apt_https propagates the real exit status" +else + fail_t "_step_apt_https propagates the real exit status" \ + "expected 'apt_https_enforce ... || rc=\$?' in _step_apt_https" +fi + +if grep -qE 'if ! apt_https_enforce[^|]*; then rc=\$\?' "$SRC"; then + fail_t "_step_apt_https avoids the 'if ! cmd; then rc=\$?' trap" "trap present" +else + ok_t "_step_apt_https avoids the 'if ! cmd; then rc=\$?' trap" +fi + +# Behavioural check: a failing enforcement must make the step fail. +reset_sandbox +printf 'deb http://nope.example.com/ubuntu noble main\n' > "$SANDBOX/apt/sources.list" +rm -rf "$BACKUPS"; : > "$BACKUPS" +# Mirror _step_apt_https's body against the sandboxed lib. +_APT_HTTPS_DONE="" +_S_RC=0 +apt_https_enforce "step:apt_https" || _S_RC=$? +if [ "$_S_RC" -ne 0 ]; then + ok_t "failed enforcement yields a non-zero status (verified at runtime)" +else + fail_t "failed enforcement yields a non-zero status (verified at runtime)" "got 0" +fi +rm -f "$BACKUPS" + +# ============================================================================ +# Summary +# ============================================================================ +printf '\n' +if [ "$FAIL" -eq 0 ]; then + printf 'All %d test(s) passed.\n' "$PASS" + exit 0 +fi +printf '%d passed, %d FAILED.\n' "$PASS" "$FAIL" +exit 1 \ No newline at end of file diff --git a/tests/test_color_gate.sh b/tests/test_color_gate.sh index 50de368..cb353c7 100644 --- a/tests/test_color_gate.sh +++ b/tests/test_color_gate.sh @@ -431,12 +431,18 @@ assert_out "tput-empty emits plain text" "sample" "$out" # was called twice and C_RED got leaked. RESTORE="$ROOT/restore_ssh.sh" if [ -f "$RESTORE" ]; then - # Pull the block we care about (lines 19-50 in current restore_ssh.sh). - # Rewrite EUID check so we don't trip the root check. - { - sed -n '19,54p' "$RESTORE" - printf 'printf "C_RED=[%%s]\\n" "$C_RED"\n' - } > "$WD/restore_head.sh" + # Pull the block we care about: _color_safe() through the C_* constant + # assignment, i.e. up to (but not including) the ok() helper. + # Anchored on function names, not line numbers, so inserting a source + # block near the top of restore_ssh.sh cannot silently shift the slice. + _RS_START=$(grep -n '^_color_safe() {' "$RESTORE" | head -1 | cut -d: -f1) + _RS_END=$(grep -n '^ok()' "$RESTORE" | head -1 | cut -d: -f1) + _RS_END=$((_RS_END - 1)) + if [ -n "$_RS_START" ] && [ -n "$_RS_END" ] && [ "$_RS_END" -gt "$_RS_START" ]; then + { + sed -n "${_RS_START},${_RS_END}p" "$RESTORE" + printf 'printf "C_RED=[%%s]\\n" "$C_RED"\n' + } > "$WD/restore_head.sh" # Gate closed -> C_RED must be empty out="$(NO_COLOR=1 /usr/bin/bash "$WD/restore_head.sh" 2>&1; printf 'C_RED=[%s]\n' "$C_RED")" @@ -451,6 +457,7 @@ if [ -f "$RESTORE" ]; then *$'\033[1;31m'*) ok_t "restore_ssh: NEOHIRO_COLOR=1 -> C_RED contains CSI" ;; *) fail_t "restore_ssh: NEOHIRO_COLOR=1 -> C_RED contains CSI" "got: $out" ;; esac + fi fi echo diff --git a/tests/test_linuxinstall.sh b/tests/test_linuxinstall.sh index dd89679..0f2e978 100644 --- a/tests/test_linuxinstall.sh +++ b/tests/test_linuxinstall.sh @@ -95,7 +95,7 @@ else fi # --- _valid_step: every documented alias must be accepted --- -EXPECTED_KEYS="system system_update dns dnscrypt firewall tor ssh ssh_hardening fail2ban unattended ipv6 sysctl apparmor pam optimize optimize_asr deepclean" +EXPECTED_KEYS="system system_update dns dnscrypt apt_https firewall tor ssh ssh_hardening fail2ban unattended ipv6 sysctl apparmor pam optimize optimize_asr deepclean" for k in $EXPECTED_KEYS; do if _valid_step "$k"; then ok_t "_valid_step accepts: $k" From 52b146ef2249f10f70393c807d93f63b11e93dc5 Mon Sep 17 00:00:00 2001 From: neohiro Date: Sun, 4 Oct 2026 04:56:54 +0200 Subject: [PATCH 02/17] fix: close three HTTPS-guard gaps found in self-review The guard was correct on every path the tests exercised, but a review pass found three places where a package could still be fetched over plaintext HTTP. All three are now closed. 1. curl|bash inline fallback had unguarded updaters. lib/updater.sh is never on disk under `curl | sudo bash`, which is the documented primary install method. linuxinstall.sh carries its own inline copy of every _update_*, and only _update_apt was guarded - so the dnf, yum, zypper, pacman, snap, flatpak, docker, brew and firmware paths ran with no precaution at all, and the inline _run_all_updates dispatcher did not guard either. Every one of them is guarded now. 2. Four lib/updater.sh sub-steps that hit the network were unguarded. _update_docker (docker pull), _update_firmware (fwupdmgr refresh/update against LVFS), _update_geoip and _update_pihole are guarded. The dispatcher already covered them on a normal run, but each is also reachable directly, so they should not depend on the caller's ordering. _update_virsh, _update_suse_snapper and _update_btrfs_balance stay unguarded on purpose - they only read or write local state. 3. Subscripts fetched at runtime lost the guard entirely. run_remote_script pulls DeepClean.sh / OptimizeLinuxASR.sh into $TMP_DIR. A script resolves its helpers relative to its own location, so $TMP_DIR/lib/ does not exist and the fetched subscript found no apt_https_guard at all - meaning DeepClean.sh's `apt-get autoremove --purge` could resolve and fetch dependencies over plaintext HTTP even though the parent process had just enforced HTTPS. The child is a separate process with its own once-per-process latch, so the parent's enforcement does not carry over. run_remote_script now prefetches lib/apt-https.sh next to the subscript, best-effort: a failure there logs and continues, because the parent has already enforced for this operation. A subscript curl|bash'd entirely on its own and finding no library now says so out loud instead of skipping the precaution silently (restore_ssh.sh, DeepClean.sh). Also: GEOIP_URL is the one operator-supplied download target in the update engine, and a GeoIP database decides which country a packet counts as being in, so a tampered copy is a traffic-tunneling primitive. An http:// value is now refused outright rather than fetched. tests/test_apt_https.sh grows to 120 assertions: the inline _update_* guards, the GEOIP_URL refusal (checked both structurally and at runtime for http and https), and a runtime exercise of the lib prefetch that drives run_remote_script with a fake curl - proving the prefetch happens, lands in the right place, is non-fatal when it fails, and leaves no empty stub behind. That last group was negative-controlled: stripping the prefetch makes it fail. Full suite: 551 passed, 0 failures. bash -n clean on every .sh. --- DeepClean.sh | 5 ++ README.md | 40 +++++++-- lib/updater.sh | 22 +++++ linuxinstall.sh | 37 ++++++++ restore_ssh.sh | 6 ++ tests/test_apt_https.sh | 191 +++++++++++++++++++++++++++++++++++++++- 6 files changed, 292 insertions(+), 9 deletions(-) diff --git a/DeepClean.sh b/DeepClean.sh index f5ff522..7c4f019 100644 --- a/DeepClean.sh +++ b/DeepClean.sh @@ -81,6 +81,11 @@ msg "Detected package manager: ${PM:-none}" # (curl|bash of DeepClean.sh on its own). if declare -F apt_https_guard >/dev/null 2>&1 && [ "$PM" = "apt" ]; then apt_https_guard "DeepClean" || true +elif [ "$PM" = "apt" ]; then + # Reached when this file is curl|bash'd on its own, with no lib/ next to + # it. Say so rather than silently skipping the precaution. + echo "[!] lib/apt-https.sh not found next to this script; skipping the" >&2 + echo "[!] repository transport guard before 'apt-get autoremove --purge'." >&2 fi msg "Starting DeepClean..." diff --git a/README.md b/README.md index 6be08de..5818107 100644 --- a/README.md +++ b/README.md @@ -193,12 +193,37 @@ https-capable mirror, or opt in to the rewrite with **Where it is wired in:** `pkg_update` / `pkg_install` / `pkg_upgrade` / `pkg_autoremove`, `update_system`, `update_kernel`, `updates_only_mode`, -every `_update_*` in `lib/updater.sh` plus the `_run_all_updates` -dispatcher, `restore_ssh.sh` (before installing `openssh-server`), -`DeepClean.sh` (before `apt-get autoremove --purge`), the **Maintenance** -submenu option 1, the `--step apt_https` mode, and the `--apt-https*` -flags. `OptimizeLinuxASR.sh` does not download packages, so it needs no -guard. +`restore_ssh.sh` (before installing `openssh-server`), `DeepClean.sh` +(before `apt-get autoremove --purge`), the **Maintenance** submenu option 1, +the `--step apt_https` mode, and the `--apt-https*` flags. + +In `lib/updater.sh` it guards the `_run_all_updates` dispatcher plus every +sub-step that touches the network: `_update_apt`, `_update_dnf`, +`_update_yum`, `_update_zypper`, `_update_pacman`, `_update_snap`, +`_update_flatpak`, `_update_docker`, `_update_brew`, `_update_firmware`, +`_update_geoip`, `_update_pihole`. (`_update_virsh`, `_update_suse_snapper` +and `_update_btrfs_balance` only read or write local state, so they are +deliberately not guarded.) `linuxinstall.sh` carries a mirrored set of +guards for its own inline `_update_*` copies, which is what runs under +`curl | sudo bash` — the documented primary install method, where +`lib/updater.sh` is never on disk. + +Two subtleties worth knowing: + +- **Fetched subscripts get the guard too.** `run_remote_script` pulls + `DeepClean.sh` / `OptimizeLinuxASR.sh` into a temp directory, and a script + resolves its helpers relative to its own location — so without help it + would find no `lib/`, and its `apt-get autoremove --purge` would run + unguarded even though the parent had already enforced. The installer + therefore prefetches `lib/apt-https.sh` next to the subscript. If a + subscript is `curl | bash`'d entirely on its own and finds no library, it + says so out loud rather than skipping the precaution silently. +- **`GEOIP_URL` must be `https://`.** It is the one operator-supplied + download target in the update engine, and a GeoIP database decides which + country a packet counts as being in — a tampered copy is a + traffic-tunneling primitive. An `http://` value is refused outright. + +`OptimizeLinuxASR.sh` does not download packages, so it needs no guard. | Variable | Effect | |---|---| @@ -209,6 +234,7 @@ guard. | `NEOHIRO_APT_HTTPS_NOVERIFY=1` | skip the post-rewrite `apt-get update` check | | `NEOHIRO_APT_HTTPS_STRICT=1` | treat leftover plaintext repos as a hard error | | `NEOHIRO_APT_BLOCK_PORT80=1` | also `ufw deny out 80/tcp` (opt-in, see below) | +| `NEOHIRO_APT_FAMILY=apt\|dnf\|yum\|zypper\|pacman\|none` | pin the detected family (CI containers, testing) | ### Optional: block port 80 entirely @@ -662,7 +688,7 @@ dumps — add to `/etc/security/limits.conf`: ```bash bash tests/run-all.sh # every suite, with a summary bash tests/test_linuxinstall.sh # 66 tests: parse, logic, UX coverage, snapshot -bash tests/test_apt_https.sh # 84 tests: HTTPS guard, hermetic (fake /etc, fake ufw) +bash tests/test_apt_https.sh # 120 tests: HTTPS guard, hermetic (fake /etc, fake ufw) bash tests/test_updater.sh # 45 tests: dispatcher, race safety, version floor shellcheck -S warning *.sh lib/*.sh tests/*.sh # lint ``` diff --git a/lib/updater.sh b/lib/updater.sh index aa46545..5666264 100644 --- a/lib/updater.sh +++ b/lib/updater.sh @@ -335,6 +335,7 @@ _update_flatpak() { _update_docker() { command -v docker >/dev/null 2>&1 || return 0 + apt_https_guard "_update_docker" || true msg "docker: pulling latest images..." local images images=$(docker images --format '{{.Repository}}:{{.Tag}}' 2>/dev/null | grep -v '' || true) @@ -383,6 +384,7 @@ _update_brew() { _update_firmware() { command -v fwupdmgr >/dev/null 2>&1 || return 0 + apt_https_guard "_update_firmware" || true msg "fwupdmgr: refreshing metadata..." run sudo fwupdmgr refresh 2>/dev/null || true local available @@ -417,6 +419,7 @@ _update_geoip() { # Export: GEOIP_URL=https://your-mirror.example.com/GeoLite2-Country.mmdb # (default) — community-maintained fork (no account needed). local _geoip_url="" _key + apt_https_guard "_update_geoip" || true if [ -n "${MAXMIND_LICENSE_KEY:-}" ]; then _key=$(printf '%s' "${MAXMIND_LICENSE_KEY}" | tr -d '[:space:]') if [ -n "$_key" ]; then @@ -429,6 +432,24 @@ _update_geoip() { if [ -z "$_geoip_url" ]; then _geoip_url="https://raw.githubusercontent.com/maccurry/GeoIP-country/main/GeoLite2-Country.mmdb" fi + # GEOIP_URL is operator-supplied, so it is the one download target in this + # engine that can be pointed at plaintext. A GeoIP database decides which + # country a packet is "in", so a tampered copy is a traffic-tunneling + # primitive. Refuse http:// outright rather than silently fetching it. + case "$_geoip_url" in + https://*) : ;; + http://*) + err "GEOIP_URL is plaintext (http://). Refusing to download the GeoIP database over HTTP." + info "Use an https:// mirror, or unset GEOIP_URL to use the default fork." + _track + return 1 + ;; + *) + err "GEOIP_URL must be an https:// URL; got: ${_geoip_url%%\?*}" + _track + return 1 + ;; + esac # Common locations for GeoIP Country database. local _geoip_db geoip_candidates=( "/usr/share/GeoIP/GeoLite2-Country.mmdb" @@ -593,6 +614,7 @@ _update_btrfs_balance() { _update_pihole() { command -v pihole >/dev/null 2>&1 || return 0 + apt_https_guard "_update_pihole" || true if [ "$EUID" -ne 0 ]; then _log "pihole: requires root — skipping" return 0 diff --git a/linuxinstall.sh b/linuxinstall.sh index 4e41c89..f43f4ae 100644 --- a/linuxinstall.sh +++ b/linuxinstall.sh @@ -168,37 +168,47 @@ if ! declare -F _run_all_updates >/dev/null 2>&1; then run sudo env DEBIAN_FRONTEND=noninteractive apt-get -y autoremove -qq run sudo apt-get clean -qq; } _update_dnf() { command -v dnf >/dev/null 2>&1 || return 0 + apt_https_guard "_update_dnf" || true run sudo dnf upgrade --refresh -y -q || return 1 run sudo dnf autoremove -y -q; } _update_yum() { command -v yum >/dev/null 2>&1 || return 0 + apt_https_guard "_update_yum" || true run sudo yum update -y -q || return 1 run sudo yum autoremove -y -q; } _update_zypper(){ command -v zypper >/dev/null 2>&1 || return 0 + apt_https_guard "_update_zypper" || true run sudo zypper --quiet refresh run sudo zypper update -y --quiet || return 1 run sudo zypper --quiet clean; } _update_pacman(){ command -v pacman >/dev/null 2>&1 || return 0 + apt_https_guard "_update_pacman" || true run sudo pacman -Syu --noconfirm --quiet || return 1 run sudo pacman -Scc --noconfirm -q; } _update_snap() { command -v snap >/dev/null 2>&1 || return 0 + apt_https_guard "_update_snap" || true run sudo snap refresh 2>/dev/null || true; } _update_flatpak(){ command -v flatpak >/dev/null 2>&1 || return 0 + apt_https_guard "_update_flatpak" || true run flatpak update -y --assumeyes 2>/dev/null || true run flatpak uninstall --unused -y --assumeyes 2>/dev/null || true; } _update_docker() { command -v docker >/dev/null 2>&1 || return 0 + apt_https_guard "_update_docker" || true while IFS= read -r img; do [ -z "$img" ] && continue docker pull "$img" >/dev/null 2>&1 || true done < <(docker images --format '{{.Repository}}:{{.Tag}}' 2>/dev/null | grep -v '') docker image prune -f >/dev/null 2>&1 || true; } _update_brew() { command -v brew >/dev/null 2>&1 || return 0 + apt_https_guard "_update_brew" || true HOMEBREW_NO_ANALYTICS=1 run brew update 2>/dev/null || true run brew upgrade 2>/dev/null || true run brew cleanup -s -q 2>/dev/null || true; } _update_firmware(){ command -v fwupdmgr >/dev/null 2>&1 || return 0 + apt_https_guard "_update_firmware" || true run sudo fwupdmgr refresh 2>/dev/null || true run sudo fwupdmgr update -y --no-reboot-check 2>/dev/null || true; } _run_all_updates() { + apt_https_guard "_run_all_updates" || true msg "=== Comprehensive system update ===" _update_apt || true _update_dnf || true @@ -1074,6 +1084,33 @@ run_remote_script() { fi chmod +x "$dst" + # Fetch lib/apt-https.sh alongside the subscript. + # + # A subscript fetched into $TMP_DIR resolves its helpers relative to its own + # location ($TMP_DIR/lib/...), and $TMP_DIR has no lib/ directory. Without + # this, DeepClean.sh's `apt-get autoremove --purge` would run with no + # transport guard even though the parent process already enforces one - + # the child is a separate process with its own once-per-process latch. + # Best effort: a failure here must not abort the run, because the parent + # has already enforced HTTPS for this operation. + _fetch_https_lib_for_subscript() { + local libdir="${TMP_DIR}/lib" libdst="${TMP_DIR}/lib/apt-https.sh" + local liburl="${REPO_RAW_BASE}/lib/apt-https.sh" + mkdir -p "$libdir" 2>/dev/null || return 1 + if command -v curl >/dev/null 2>&1; then + curl -fsSL "$liburl" -o "$libdst" 2>/dev/null || return 1 + elif command -v wget >/dev/null 2>&1; then + wget -qO "$libdst" "$liburl" 2>/dev/null || return 1 + else + return 1 + fi + [ -s "$libdst" ] || { rm -f "$libdst" 2>/dev/null; return 1; } + return 0 + } + if ! _fetch_https_lib_for_subscript; then + _log "could not prefetch lib/apt-https.sh; $name will run without the transport guard" + fi + # Optional GPG verification. Disabled by default; enable by setting # NEOSIGN_GPG_LEVEL=required NEOSIGN_GPG_FPR=<40-hex fingerprint> # in the environment. `advisory` warns but does not abort. The diff --git a/restore_ssh.sh b/restore_ssh.sh index 6735aa9..3105f69 100644 --- a/restore_ssh.sh +++ b/restore_ssh.sh @@ -121,6 +121,12 @@ pkg_install_ssh() { # Precaution: never fetch openssh over a plaintext mirror. if declare -F apt_https_guard >/dev/null 2>&1; then apt_https_guard "restore_ssh:pkg_install_ssh" || true + else + # Reached when this file is curl|bash'd on its own, with no lib/ next to + # it. Say so rather than silently skipping the precaution. + echo "[WARN] lib/apt-https.sh not found next to this script; skipping the" >&2 + echo "[WARN] repository transport guard. Run 'sudo bash restore_ssh.sh" >&2 + echo "[WARN] --apt-https-audit' from a clone to check your repos." >&2 fi case "$PKG_MGR" in apt) run sudo env DEBIAN_FRONTEND=noninteractive apt-get install -y openssh-server ;; diff --git a/tests/test_apt_https.sh b/tests/test_apt_https.sh index e651fb6..d1fc568 100644 --- a/tests/test_apt_https.sh +++ b/tests/test_apt_https.sh @@ -645,10 +645,13 @@ for fn in pkg_update pkg_install pkg_upgrade pkg_autoremove; do fi done -# lib/updater.sh must guard the dispatcher and every sub-step. +# lib/updater.sh must guard the dispatcher and every sub-step that touches +# the network. _update_virsh, _update_suse_snapper and _update_btrfs_balance +# are deliberately excluded: they only read/write local state. UP="$ROOT/lib/updater.sh" for fn in _update_apt _update_dnf _update_yum _update_zypper _update_pacman \ - _update_snap _update_flatpak _update_brew _run_all_updates; do + _update_snap _update_flatpak _update_docker _update_brew _update_firmware \ + _update_geoip _update_pihole _run_all_updates; do if grep -q "apt_https_guard \"$fn\"" "$UP"; then ok_t "integration: updater $fn() calls apt_https_guard" else @@ -656,6 +659,69 @@ for fn in _update_apt _update_dnf _update_yum _update_zypper _update_pacman \ fi done +# The curl|bash inline fallback in linuxinstall.sh has its own copies of every +# _update_*; those must be guarded too, since curl|bash is the documented +# primary install method and never sees lib/updater.sh. +for fn in _update_apt _update_dnf _update_yum _update_zypper _update_pacman \ + _update_snap _update_flatpak _update_docker _update_brew _update_firmware \ + _run_all_updates; do + BODY="$(awk -v fn="$fn" ' + $0 ~ "^[[:space:]]*"fn"\\(\\)[[:space:]]*\\{" { inside = 1 } + inside { print } + inside && /^[[:space:]]*}/ { exit } + ' "$SRC")" + if printf '%s' "$BODY" | grep -q 'apt_https_guard'; then + ok_t "integration: inline $fn() calls apt_https_guard" + else + fail_t "integration: inline $fn() calls apt_https_guard" "no guard call" + fi +done + +# GEOIP_URL is operator-supplied and decides which country a packet is "in", +# so a plaintext mirror is a traffic-tunneling primitive. Must be refused. +GEOIP_BODY="$(awk '/^_update_geoip\(\) \{/,/^}/' "$UP")" +if printf '%s' "$GEOIP_BODY" | grep -q 'http://\*' && \ + printf '%s' "$GEOIP_BODY" | grep -q 'Refusing to download the GeoIP database over HTTP'; then + ok_t "integration: _update_geoip refuses a plaintext GEOIP_URL" +else + fail_t "integration: _update_geoip refuses a plaintext GEOIP_URL" "scheme check missing" +fi +# The check must come AFTER the URL is fully resolved, otherwise the default +# fork (which is already https) would be validated against nothing. +GEOIP_LINE="$(printf '%s' "$GEOIP_BODY" | grep -n 'case "$_geoip_url" in' | cut -d: -f1)" +DEFAULT_LINE="$(printf '%s' "$GEOIP_BODY" | grep -n 'maccurry/GeoIP-country' | cut -d: -f1)" +if [ -n "$GEOIP_LINE" ] && [ -n "$DEFAULT_LINE" ] && [ "$GEOIP_LINE" -gt "$DEFAULT_LINE" ]; then + ok_t "integration: _update_geoip validates the URL scheme after resolving it" +else + fail_t "integration: _update_geoip validates the URL scheme after resolving it" \ + "check@${GEOIP_LINE:-none} must come after default@${DEFAULT_LINE:-none}" +fi + +# Subscripts fetched via run_remote_script land in $TMP_DIR with no lib/, so +# their transport guard would silently not exist. The installer must prefetch +# lib/apt-https.sh next to them. +if grep -q '_fetch_https_lib_for_subscript' "$SRC" && \ + grep -q 'lib/apt-https.sh' "$SRC"; then + ok_t "integration: run_remote_script prefetches lib/apt-https.sh for subscripts" +else + fail_t "integration: run_remote_script prefetches lib/apt-https.sh for subscripts" \ + "lib prefetch missing" +fi +if awk '/^run_remote_script\(\) \{/,/^}/' "$SRC" | grep -q '_fetch_https_lib_for_subscript'; then + ok_t "integration: the lib prefetch lives inside run_remote_script" +else + fail_t "integration: the lib prefetch lives inside run_remote_script" "not in run_remote_script" +fi + +# A subscript without the lib must say so rather than skip silently. +for f in restore_ssh.sh DeepClean.sh; do + if grep -q 'not found next to this script' "$ROOT/$f"; then + ok_t "integration: $f warns when lib/apt-https.sh is unavailable" + else + fail_t "integration: $f warns when lib/apt-https.sh is unavailable" "no warning" + fi +done + if grep -q 'apt-https.sh' "$UP" && grep -q 'source .*apt-https.sh' "$ROOT/restore_ssh.sh" && \ grep -q 'source .*apt-https.sh' "$ROOT/DeepClean.sh" && \ grep -q 'source .*apt-https.sh' "$SRC"; then @@ -773,6 +839,127 @@ else fi rm -f "$BACKUPS" +# ============================================================================ +# Runtime: run_remote_script prefetches lib/apt-https.sh for subscripts +# ============================================================================ +# A subscript fetched into $TMP_DIR resolves helpers relative to its own +# location, and $TMP_DIR has no lib/ directory. Extract the function and +# drive it with a fake curl so the prefetch is actually executed. +{ + awk '/^run_remote_script\(\) \{/,/^}/' "$SRC" +} > "$WD/rrs.sh" + +run_rrs_case() { + # $1 = fake curl behaviour: "ok" writes a payload, "fail" exits non-zero. + local mode="$1" out + ( + set -u + TMP_DIR="$WD/rrs_tmp" + rm -rf "$TMP_DIR"; mkdir -p "$TMP_DIR" + REPO_RAW_BASE="https://raw.githubusercontent.com/neohiro/linux/main" + _log() { :; } + ok() { printf 'OK:%s\n' "$*"; } + warn(){ printf 'WARN:%s\n' "$*"; } + err() { printf 'ERR:%s\n' "$*"; } + # Stub curl/wget: record the URL, optionally fail. + curl() { + case "$1" in + -*) : ;; + esac + # Last two args are the URL and the -o target. + local url="" outp="" + while [ $# -gt 0 ]; do + case "$1" in + -o) outp="$2"; shift 2 ;; + http*) url="$1"; shift ;; + *) shift ;; + esac + done + printf '%s\n' "$url" >> "$WD/curl-urls.log" + if [ "$mode" = "fail" ] && [ "$url" != "${REPO_RAW_BASE}/DeepClean.sh" ]; then + return 22 + fi + printf '# fetched payload for %s\n' "$url" > "$outp" + return 0 + } + # shellcheck disable=SC1090 + . "$WD/rrs.sh" + # Stop before the execution-mode decision; we only care about the fetch. + run_remote_script "DeepClean.sh" >/dev/null 2>&1 || true + printf 'CURL_URLS=%s\n' "$(cat "$WD/curl-urls.log" 2>/dev/null | tr '\n' ' ')" + if [ -s "$TMP_DIR/lib/apt-https.sh" ]; then + printf 'LIB_PRESENT=yes\n' + else + printf 'LIB_PRESENT=no\n' + fi + printf 'SUB_PRESENT=%s\n' "$([ -s "$TMP_DIR/DeepClean.sh" ] && echo yes || echo no)" + ) 2>/dev/null +} + +: > "$WD/curl-urls.log" +RRS_OK="$(run_rrs_case ok)" +if printf '%s' "$RRS_OK" | grep -q 'lib/apt-https.sh'; then + ok_t "run_remote_script requests lib/apt-https.sh from the repo" +else + fail_t "run_remote_script requests lib/apt-https.sh from the repo" \ + "URLs seen: $(printf '%s' "$RRS_OK" | grep '^CURL_URLS=')" +fi +if printf '%s' "$RRS_OK" | grep -q 'LIB_PRESENT=yes'; then + ok_t "run_remote_script places lib/apt-https.sh next to the subscript" +else + fail_t "run_remote_script places lib/apt-https.sh next to the subscript" \ + "$(printf '%s' "$RRS_OK" | grep LIB_PRESENT=)" +fi +if printf '%s' "$RRS_OK" | grep -q 'SUB_PRESENT=yes'; then + ok_t "run_remote_script still fetched the subscript itself" +else + fail_t "run_remote_script still fetched the subscript itself" "subscript missing" +fi + +: > "$WD/curl-urls.log" +RRS_FAIL="$(run_rrs_case fail)" +if printf '%s' "$RRS_FAIL" | grep -q 'SUB_PRESENT=yes'; then + ok_t "a failed lib prefetch is non-fatal (subscript still fetched)" +else + fail_t "a failed lib prefetch is non-fatal (subscript still fetched)" \ + "a lib fetch failure aborted the run" +fi +if printf '%s' "$RRS_FAIL" | grep -q 'LIB_PRESENT=no'; then + ok_t "a failed lib prefetch leaves no empty stub file behind" +else + fail_t "a failed lib prefetch leaves no empty stub file behind" "stale file left" +fi + +# ============================================================================ +# Runtime: _update_geoip refuses a plaintext GEOIP_URL +# ============================================================================ +GEOIP_RT="$(env NEOHIRO_APT_FAMILY=apt NEOHIRO_APT_HTTPS=0 GEOIP_URL='http://evil.example.com/Geo.mmdb' \ + bash -c " + . '$UP' >/dev/null 2>&1 + UPDATED=0; FAILED=0 + _update_geoip 2>&1 + printf 'RC=%s\n' \$? + " 2>&1)" +if printf '%s' "$GEOIP_RT" | grep -q 'Refusing to download the GeoIP database over HTTP'; then + ok_t "_update_geoip refuses an http:// GEOIP_URL at runtime" +else + fail_t "_update_geoip refuses an http:// GEOIP_URL at runtime" \ + "got: $(printf '%s' "$GEOIP_RT" | tr '\n' ' ')" +fi + +GEOIP_OK="$(env NEOHIRO_APT_FAMILY=apt NEOHIRO_APT_HTTPS=0 GEOIP_URL='https://good.example.com/Geo.mmdb' \ + bash -c " + . '$UP' >/dev/null 2>&1 + UPDATED=0; FAILED=0 + _update_geoip 2>&1 + printf 'RC=%s\n' \$? + " 2>&1)" +if ! printf '%s' "$GEOIP_OK" | grep -q 'Refusing to download'; then + ok_t "_update_geoip accepts an https:// GEOIP_URL" +else + fail_t "_update_geoip accepts an https:// GEOIP_URL" "false rejection" +fi + # ============================================================================ # Summary # ============================================================================ From 9c9d1e0d4a94fe8ae60cb0871fc789542113a89c Mon Sep 17 00:00:00 2001 From: neohiro Date: Sun, 4 Oct 2026 05:29:34 +0200 Subject: [PATCH 03/17] fix: make the guard's exit-code contract explicit under `set -e` Third review pass. restore_ssh.sh, DeepClean.sh and the lib/updater.sh standalone branch all run under `set -e`, which the transport flags were not accounting for. - restore_ssh.sh --apt-https called `apt_https_enforce` bare. On failure `set -e` aborted the script, so the report never printed and the user saw a truncated run. The `return $?` that followed `apt_https_report || true` would also have reported the *report's* status, which is always 0 - so the exit code was right only by coincidence, on the path where it was least needed. - The same latent pattern sat in lib/updater.sh's standalone flag branches, where the correct exit code came out right only because `set -e` happened to propagate it, skipping the report. Both now capture the enforcement status explicitly (`cmd || _rc=$?`), always print the report, and propagate enforcement's own result. apt_https_guard is also now guaranteed to return 0. It is called from pkg_install and friends, some of which run under `set -e`; a precaution should never be the reason a package operation aborts. Enforcement problems surface through apt_https_report and the enforce exit status instead. lib/apt-https.sh's own standalone entry point now sets its exit code explicitly rather than inheriting whichever command ran last, so the documented contract (0 clean / 1 plaintext remains / 2 bad usage) is a guarantee instead of an accident. tests/test_apt_https.sh grows to 145 assertions. The new ones drive the real scripts and assert both the exit code and that the report still printed: restore_ssh.sh through an extracted-main() harness with root and the print helpers stubbed (set -e is deliberately left active, and main() is invoked outside any condition so set -e is not silently disabled inside its body), plus lib/updater.sh run directly, plus apt_https_guard's never-fail contract and the standalone exit-code contract. Negative-controlled: re-introducing the original bare-call-plus-`return 0` in restore_ssh.sh makes 5 assertions fail, including "prints the report even when enforce fails" with the exact predicted reason. Full suite: 576 passed, 0 failures. bash -n clean on every .sh. --- README.md | 2 +- lib/apt-https.sh | 24 +++- lib/updater.sh | 20 +-- linuxinstall.sh | 2 +- restore_ssh.sh | 22 +++- tests/test_apt_https.sh | 264 ++++++++++++++++++++++++++++++++++++++++ 6 files changed, 314 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 5818107..36be21e 100644 --- a/README.md +++ b/README.md @@ -688,7 +688,7 @@ dumps — add to `/etc/security/limits.conf`: ```bash bash tests/run-all.sh # every suite, with a summary bash tests/test_linuxinstall.sh # 66 tests: parse, logic, UX coverage, snapshot -bash tests/test_apt_https.sh # 120 tests: HTTPS guard, hermetic (fake /etc, fake ufw) +bash tests/test_apt_https.sh # 145 tests: HTTPS guard, hermetic (fake /etc, fake ufw) bash tests/test_updater.sh # 45 tests: dispatcher, race safety, version floor shellcheck -S warning *.sh lib/*.sh tests/*.sh # lint ``` diff --git a/lib/apt-https.sh b/lib/apt-https.sh index 4e68fd8..63a1ce8 100644 --- a/lib/apt-https.sh +++ b/lib/apt-https.sh @@ -707,12 +707,19 @@ apt_https_enforce() { # apt_https_guard [label] — hot-path wrapper. Enforces at most once per # process so every package entry point can call it unconditionally. +# +# It always returns 0. A guard must never be the reason a package operation +# fails: call sites are things like `pkg_install`, which run under `set -e` +# in some of the standalone scripts, and a non-zero return there would abort +# the caller before it ever reached the package manager. Enforcement failures +# are reported through apt_https_report and the enforce exit status, not by +# failing the hot path. apt_https_guard() { [ -n "$_APT_HTTPS_DONE" ] && return 0 # Latch first: the verification `apt-get update` below can re-enter the # package layer, which must not recurse. _APT_HTTPS_DONE=1 - apt_https_enforce "${1:-guard}" + apt_https_enforce "${1:-guard}" || true return 0 } @@ -745,10 +752,17 @@ apt_https_revert() { # sudo bash lib/apt-https.sh --report audit only # sudo bash lib/apt-https.sh --revert undo if [ "${BASH_SOURCE[0]:-$0}" = "$0" ]; then + # Standalone: this file sets no `set -e`, but capture the status anyway so + # the documented exit-code contract (0 = clean, 1 = plaintext remains, + # 2 = no package manager) is explicit rather than an accident of which + # command happened to run last. + _ap_https_rc=0 case "${1:---enforce}" in - --report|--audit) apt_https_report ;; - --revert) apt_https_revert ;; - --enforce) apt_https_enforce "cli" ;; - *) printf 'usage: bash lib/apt-https.sh [--enforce|--report|--revert]\n' >&2; exit 2 ;; + --report|--audit) apt_https_report || _ap_https_rc=$? ;; + --revert) apt_https_revert || _ap_https_rc=$? ;; + --enforce) apt_https_enforce "cli" || _ap_https_rc=$? ;; + *) printf 'usage: bash lib/apt-https.sh [--enforce|--report|--revert]\n' >&2 + _ap_https_rc=2 ;; esac + exit "$_ap_https_rc" fi \ No newline at end of file diff --git a/lib/updater.sh b/lib/updater.sh index 5666264..1562d1e 100644 --- a/lib/updater.sh +++ b/lib/updater.sh @@ -812,22 +812,28 @@ else SECONDS=0 # Sub-commands handled here rather than by the dispatcher, because they # only touch the repository transport and never run an update. + # + # `set -euo pipefail` is active in this branch, so a bare call that returns + # non-zero would exit before `exit $?` runs and skip the report. Capture + # the status explicitly instead of relying on that coincidence. + _ap_https_rc=0 case "${1:-}" in --apt-https|--enforce-https) if [ "${2:-}" = "--audit" ]; then - apt_https_report + apt_https_report || _ap_https_rc=$? else - apt_https_enforce "updater --apt-https" + apt_https_enforce "updater --apt-https" || _ap_https_rc=$? + apt_https_report || true fi - exit $? + exit "$_ap_https_rc" ;; --apt-https-audit) - apt_https_report - exit $? + apt_https_report || _ap_https_rc=$? + exit "$_ap_https_rc" ;; --apt-https-off|--disable-https) - apt_https_revert - exit $? + apt_https_revert || _ap_https_rc=$? + exit "$_ap_https_rc" ;; esac _run_all_updates "$@" diff --git a/linuxinstall.sh b/linuxinstall.sh index f43f4ae..df1f4f5 100644 --- a/linuxinstall.sh +++ b/linuxinstall.sh @@ -574,7 +574,7 @@ if ! declare -F apt_https_guard >/dev/null 2>&1; then apt_https_guard() { [ -n "$_APT_HTTPS_DONE" ] && return 0 _APT_HTTPS_DONE=1 - apt_https_enforce "${1:-guard}" + apt_https_enforce "${1:-guard}" || true return 0 } apt_https_revert() { diff --git a/restore_ssh.sh b/restore_ssh.sh index 3105f69..d134a26 100644 --- a/restore_ssh.sh +++ b/restore_ssh.sh @@ -384,23 +384,33 @@ apply_fixes() { main() { # Repository-transport sub-commands: these never touch SSH at all, so # they short-circuit before diagnose/apply_fixes. + local _rc=0 case "${1:-}" in --apt-https|--enforce-https) bold "neohiro/linux - HTTPS package repositories (enforce)" require_root - apt_https_enforce "restore_ssh --apt-https" + # This script runs under `set -eo pipefail`, so a bare + # `apt_https_enforce` that returns non-zero would abort the script + # before the report printed, and `return $?` after a + # `... || true` would report the *report's* status (always 0). + # Capture the status explicitly, always show the report, then + # propagate enforcement's own result. + _rc=0 + apt_https_enforce "restore_ssh --apt-https" || _rc=$? apt_https_report || true - return $? + return "$_rc" ;; --apt-https-audit) - apt_https_report - return $? + _rc=0 + apt_https_report || _rc=$? + return "$_rc" ;; --apt-https-off|--disable-https) bold "neohiro/linux - Revert HTTPS package repository enforcement" require_root - apt_https_revert - return $? + _rc=0 + apt_https_revert || _rc=$? + return "$_rc" ;; esac bold "neohiro/linux - Restore SSH (standalone)" diff --git a/tests/test_apt_https.sh b/tests/test_apt_https.sh index d1fc568..8c3e90b 100644 --- a/tests/test_apt_https.sh +++ b/tests/test_apt_https.sh @@ -960,6 +960,270 @@ else fail_t "_update_geoip accepts an https:// GEOIP_URL" "false rejection" fi +# ============================================================================ +# Runtime: exit codes survive `set -eo pipefail` +# ============================================================================ +# restore_ssh.sh, DeepClean.sh and the lib/updater.sh standalone branch all +# run under `set -e`. A bare call that returns non-zero exits before the +# report prints, and `return $?` after a `... || true` reports the wrong +# status. Drive the real scripts with a failing enforcement and assert both +# the exit code and that the report still printed. +_rrc() { + local script="$1" flag="$2" + env NEOHIRO_APT_ETC_DIR="$SANDBOX" NEOHIRO_APT_BACKUP_DIR="$BACKUPS" \ + NEOHIRO_APT_FAMILY=apt NEOHIRO_APT_HTTPS_NOVERIFY=1 \ + bash "$ROOT/$script" "$flag" 2>&1 + printf '__RC=%s\n' "$?" +} + +reset_sandbox +printf 'deb https://secure.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +OUT="$(_rrc lib/updater.sh --apt-https-audit)" +if printf '%s' "$OUT" | grep -q '__RC=0'; then + ok_t "lib/updater.sh --apt-https-audit exits 0 on a clean system" +else + fail_t "lib/updater.sh --apt-https-audit exits 0 on a clean system" \ + "$(printf '%s' "$OUT" | grep '__RC=')" +fi + +reset_sandbox +printf 'deb https://secure.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +# Force enforcement to fail: make the backup path unwritable. +rm -rf "$BACKUPS"; : > "$BACKUPS" +OUT="$(_rrc restore_ssh.sh --apt-https-audit)" +if printf '%s' "$OUT" | grep -q '__RC=0'; then + ok_t "restore_ssh.sh --apt-https-audit exits 0 on a clean system" +else + fail_t "restore_ssh.sh --apt-https-audit exits 0 on a clean system" \ + "$(printf '%s' "$OUT" | grep '__RC=')" +fi +if printf '%s' "$OUT" | grep -q 'Package transport security'; then + ok_t "restore_ssh.sh --apt-https-audit prints the report" +else + fail_t "restore_ssh.sh --apt-https-audit prints the report" "report missing" +fi + +# Now dirty the sources so the audit must exit 1 even under `set -e`. +printf 'deb http://plain.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +OUT="$(_rrc restore_ssh.sh --apt-https-audit)" +if printf '%s' "$OUT" | grep -q '__RC=1'; then + ok_t "restore_ssh.sh --apt-https-audit exits 1 when plaintext remains" +else + fail_t "restore_ssh.sh --apt-https-audit exits 1 when plaintext remains" \ + "$(printf '%s' "$OUT" | grep '__RC=')" +fi +rm -f "$BACKUPS" + +# restore_ssh.sh --apt-https* run under `set -eo pipefail` and gate on +# require_root, so drive its real main() through a harness with root and the +# print helpers stubbed. set -e is inherited by the harness, which is the +# whole point: main() must survive it. The harness runs as its own script so +# its exit code IS main()'s exit code, without putting main() in a condition +# (which would disable set -e inside its body). +build_rs_harness() { + { + printf '%s\n' 'set -eo pipefail' + printf '%s\n' ". \"\$RS_LIB\"" + printf '%s\n' '_apt_priv() { "$@"; }' + printf '%s\n' '_apt_https_privileged_ok() { return 0; }' + printf '%s\n' 'apt_https_family() { printf "apt"; }' + printf '%s\n' 'bold() { :; }' + printf '%s\n' 'require_root() { return 0; }' + printf '%s\n' 'diagnose() { err "diagnose must not run for a transport flag"; return 9; }' + printf '%s\n' 'apply_fixes() { err "apply_fixes must not run for a transport flag"; return 9; }' + awk '/^main\(\) \{/,/^\}/' "$ROOT/restore_ssh.sh" + printf '%s\n' 'FIXES=()' + printf '%s\n' 'main "$@"' + } > "$WD/rs_harness.sh" +} +build_rs_harness +if bash -n "$WD/rs_harness.sh" 2>/dev/null; then + ok_t "restore_ssh.sh transport flags are reachable in an isolated harness" +else + fail_t "restore_ssh.sh transport flags are reachable in an isolated harness" \ + "harness does not parse" +fi + +_rs_run() { + env RS_LIB="$LIB" NEOHIRO_APT_ETC_DIR="$SANDBOX" NEOHIRO_APT_BACKUP_DIR="$BACKUPS" \ + NEOHIRO_APT_HTTPS_NOVERIFY=1 \ + bash "$WD/rs_harness.sh" "$@" 2>&1 + printf '__RC=%s\n' "$?" +} + +reset_sandbox +printf 'deb https://secure.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +OUT="$(_rs_run --apt-https-audit)" +if printf '%s' "$OUT" | grep -q '__RC=0'; then + ok_t "restore_ssh.sh --apt-https-audit exits 0 on a clean system" +else + fail_t "restore_ssh.sh --apt-https-audit exits 0 on a clean system" \ + "$(printf '%s' "$OUT" | grep '__RC=')" +fi +if printf '%s' "$OUT" | grep -q 'Package transport security'; then + ok_t "restore_ssh.sh --apt-https-audit prints the report" +else + fail_t "restore_ssh.sh --apt-https-audit prints the report" "report missing" +fi + +printf 'deb http://plain.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +OUT="$(_rs_run --apt-https-audit)" +if printf '%s' "$OUT" | grep -q '__RC=1'; then + ok_t "restore_ssh.sh --apt-https-audit exits 1 when plaintext remains" +else + fail_t "restore_ssh.sh --apt-https-audit exits 1 when plaintext remains" \ + "$(printf '%s' "$OUT" | grep '__RC=')" +fi + +# The regression these guard: enforcement fails -> the report must STILL +# print, and the exit code must be enforcement's, not the report's. +reset_sandbox +printf 'deb http://refuse.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +rm -rf "$BACKUPS"; : > "$BACKUPS" +OUT="$(_rs_run --apt-https)" +if printf '%s' "$OUT" | grep -q '__RC=1'; then + ok_t "restore_ssh.sh --apt-https propagates enforcement failure (not 0)" +else + fail_t "restore_ssh.sh --apt-https propagates enforcement failure (not 0)" \ + "$(printf '%s' "$OUT" | grep '__RC=')" +fi +if printf '%s' "$OUT" | grep -q 'Package transport security'; then + ok_t "restore_ssh.sh --apt-https prints the report even when enforce fails" +else + fail_t "restore_ssh.sh --apt-https prints the report even when enforce fails" \ + "set -e aborted before the report" +fi +if printf '%s' "$OUT" | grep -q 'diagnose must not run'; then + fail_t "restore_ssh.sh --apt-https short-circuits before diagnose" "SSH path was entered" +else + ok_t "restore_ssh.sh --apt-https short-circuits before diagnose" +fi +rm -f "$BACKUPS" + +reset_sandbox +printf 'deb https://secure.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +OUT="$(_rs_run --apt-https-off)" +if printf '%s' "$OUT" | grep -q '__RC=0'; then + ok_t "restore_ssh.sh --apt-https-off exits 0" +else + fail_t "restore_ssh.sh --apt-https-off exits 0" "$(printf '%s' "$OUT" | grep '__RC=')" +fi + +# Same contract for the lib/updater.sh standalone CLI. +reset_sandbox +printf 'deb http://upd.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +rm -rf "$BACKUPS"; : > "$BACKUPS" +UPD_OUT="$(_rrc lib/updater.sh --apt-https)" +if printf '%s' "$UPD_OUT" | grep -q '__RC=1'; then + ok_t "lib/updater.sh --apt-https propagates enforcement failure" +else + fail_t "lib/updater.sh --apt-https propagates enforcement failure" \ + "$(printf '%s' "$UPD_OUT" | grep '__RC=')" +fi +if printf '%s' "$UPD_OUT" | grep -q 'Package transport security'; then + ok_t "lib/updater.sh --apt-https prints the report even when enforce fails" +else + fail_t "lib/updater.sh --apt-https prints the report even when enforce fails" \ + "set -e aborted before the report" +fi +rm -f "$BACKUPS" + +reset_sandbox +printf 'deb http://upd.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +OUT="$(_rrc lib/updater.sh --apt-https-audit)" +if printf '%s' "$OUT" | grep -q '__RC=1'; then + ok_t "lib/updater.sh --apt-https-audit exits 1 when plaintext remains" +else + fail_t "lib/updater.sh --apt-https-audit exits 1 when plaintext remains" \ + "$(printf '%s' "$OUT" | grep '__RC=')" +fi + +# And the revert flag must report success. +reset_sandbox +printf 'deb https://secure.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +OUT="$(_rrc lib/updater.sh --apt-https-off)" +if printf '%s' "$OUT" | grep -q '__RC=0'; then + ok_t "lib/updater.sh --apt-https-off exits 0" +else + fail_t "lib/updater.sh --apt-https-off exits 0" "$(printf '%s' "$OUT" | grep '__RC=')" +fi + +# Structural guards so the pattern cannot silently regress. +for f in restore_ssh.sh lib/updater.sh; do + if grep -qE 'apt_https_enforce[^|]*\|\| _?[a-z_]*rc=\$\?' "$ROOT/$f"; then + ok_t "$f captures the enforce status instead of relying on set -e" + else + fail_t "$f captures the enforce status instead of relying on set -e" \ + "no '|| _rc=\$?' after apt_https_enforce" + fi +done + +# ============================================================================ +# apt_https_guard must NEVER fail its caller +# ============================================================================ +# Call sites include pkg_install, which some standalone scripts run under +# `set -e`. A guard is a precaution, not a gate: it must not be the reason a +# package operation aborts. +GUARD_BODY="$(awk '/^apt_https_guard\(\) \{/,/^\}/' "$LIB")" +if printf '%s' "$GUARD_BODY" | grep -q 'apt_https_enforce .* || true'; then + ok_t "apt_https_guard swallows enforcement failure (lib)" +else + fail_t "apt_https_guard swallows enforcement failure (lib)" \ + "enforce call is not guarded with '|| true'" +fi +if printf '%s' "$GUARD_BODY" | grep -q 'return 0'; then + ok_t "apt_https_guard always returns 0 (lib)" +else + fail_t "apt_https_guard always returns 0 (lib)" "no unconditional return 0" +fi +INLINE_GUARD="$(awk '/^[[:space:]]+apt_https_guard\(\) \{/,/^[[:space:]]+\}/' "$SRC")" +if printf '%s' "$INLINE_GUARD" | grep -q 'apt_https_enforce .* || true'; then + ok_t "apt_https_guard swallows enforcement failure (inline fallback)" +else + fail_t "apt_https_guard swallows enforcement failure (inline fallback)" \ + "enforce call is not guarded with '|| true'" +fi + +# Behaviour: a guard call must return 0 even when enforcement fails hard. +reset_sandbox +printf 'deb http://guardfail.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +rm -rf "$BACKUPS"; : > "$BACKUPS" +if apt_https_guard "unit" >/dev/null 2>&1; then + ok_t "apt_https_guard returns 0 despite an enforcement failure" +else + fail_t "apt_https_guard returns 0 despite an enforcement failure" \ + "guard propagated non-zero" +fi +rm -f "$BACKUPS" + +# Standalone lib exit-code contract: 0 clean / 1 plaintext / 2 bad usage. +reset_sandbox +printf 'deb https://secure.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +env NEOHIRO_APT_ETC_DIR="$SANDBOX" NEOHIRO_APT_BACKUP_DIR="$BACKUPS" \ + NEOHIRO_APT_FAMILY=apt NEOHIRO_APT_HTTPS_NOVERIFY=1 \ + bash "$LIB" --report >/dev/null 2>&1 +if [ $? -eq 0 ]; then + ok_t "lib/apt-https.sh --report exits 0 on a clean system" +else + fail_t "lib/apt-https.sh --report exits 0 on a clean system" "rc=$?" +fi +printf 'deb http://plain.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +env NEOHIRO_APT_ETC_DIR="$SANDBOX" NEOHIRO_APT_BACKUP_DIR="$BACKUPS" \ + NEOHIRO_APT_FAMILY=apt NEOHIRO_APT_HTTPS_NOVERIFY=1 \ + bash "$LIB" --report >/dev/null 2>&1 +if [ $? -eq 1 ]; then + ok_t "lib/apt-https.sh --report exits 1 when plaintext remains" +else + fail_t "lib/apt-https.sh --report exits 1 when plaintext remains" "rc=$?" +fi +env NEOHIRO_APT_ETC_DIR="$SANDBOX" NEOHIRO_APT_BACKUP_DIR="$BACKUPS" \ + NEOHIRO_APT_FAMILY=apt bash "$LIB" --nonsense >/dev/null 2>&1 +if [ $? -eq 2 ]; then + ok_t "lib/apt-https.sh exits 2 on an unknown argument" +else + fail_t "lib/apt-https.sh exits 2 on an unknown argument" "rc=$?" +fi + # ============================================================================ # Summary # ============================================================================ From eb23ffa92aa7090fc56200713e6751d0ad6ce968 Mon Sep 17 00:00:00 2001 From: neohiro Date: Sun, 4 Oct 2026 06:21:30 +0200 Subject: [PATCH 04/17] chore: save in-progress local changes Automated capture of uncommitted modifications to tracked files. --- lib/apt-https.sh | 553 +++++++++++++++++++++++++++++++++------- linuxinstall.sh | 55 +++- tests/test_apt_https.sh | 219 +++++++++++++++- 3 files changed, 717 insertions(+), 110 deletions(-) diff --git a/lib/apt-https.sh b/lib/apt-https.sh index 63a1ce8..0b5fe1b 100644 --- a/lib/apt-https.sh +++ b/lib/apt-https.sh @@ -242,6 +242,33 @@ _apt_https_plaintext_in_repo_file() { return 0 } +# _apt_https_plaintext_generic +# +# The format-agnostic detector: any non-comment line carrying an http:// URL. +# Used for every non-apt store (apk repositories, docker daemon.json, pip.conf, +# .npmrc, cargo config.toml, .gemrc, nix.conf, fwupd remotes, ...). +# +# Being generic is the point. A per-dialect key list would miss gpgkey=, +# metalink=, and whatever the next distro release adds, and a miss is exactly +# the failure this library exists to prevent. Comment lines are skipped so a +# documentation link in a config file is not mistaken for a fetch target. +_apt_https_plaintext_generic() { + local f="$1" id="${2:-}" + [ -f "$f" ] || return 0 + awk -v id="$id" ' + /^[[:space:]]*(#|;)/ { next } + /http:\/\// { + line = $0 + gsub(/^[[:space:]]+/, "", line) + printf "%s:%d: %s\n", FILENAME, NR, line + }' "$f" 2>/dev/null + return 0 +} + +# _apt_https_rewrite_generic — http:// -> https:// on non-comment lines. +# Same substitution the apt rewrite uses, so behaviour is identical. +_APT_HTTPS_GENERIC_RE='/^[[:space:]]*(#|;)/! s|http://|https://|g' + # ── Backups ────────────────────────────────────────────────────────────────── # _apt_https_backup_path — deterministic, path-flattened backup name. @@ -310,15 +337,11 @@ _apt_https_write_conf() { return 1 fi _apt_priv mkdir -p "$(apt_https_etc_dir)/apt/apt.conf.d" || { rm -f "$tmp"; return 1; } - if ! _apt_priv install -m 0644 "$tmp" "$conf"; then - # `install` is not present on every minimal image; fall back to cp+chmod. - if _apt_priv cp "$tmp" "$conf" && _apt_priv chmod 0644 "$conf"; then - rc=0 - else - rc=1 - fi - else + # Atomic rename into place; see _apt_https_atomic_replace for why. + if _apt_https_atomic_replace "$conf" 0644 _apt_https_prep_conf_stage "$tmp"; then rc=0 + else + rc=1 fi rm -f "$tmp" [ "$rc" = "0" ] || return 1 @@ -335,6 +358,68 @@ _apt_https_stage() { mktemp "${TMPDIR:-/tmp}/neohiro-apt-https.XXXXXX" } +# _apt_https_atomic_replace [prep-args...] +# +# Runs [prep-args...] to build a staging file that lives +# in the *target's own directory*, then renames it over the target. +# +# Why not just `cp tmp target`: cp truncates the destination and then +# writes. If the process dies mid-copy — OOM, SIGKILL, a container being +# stopped — the repo file is left truncated, i.e. a broken package manager. +# rename(2) within a single filesystem is atomic, so a reader (apt itself, +# or a concurrent run of this script) sees either the whole old file or the +# whole new one, never a half-written mix. Staging in the target's own +# directory guarantees the same filesystem, which is what makes the rename +# atomic instead of a slow cross-device copy. +# +# The staging name ends in the PID and therefore never ends in ".list" or +# ".sources", so apt's own sources.list.d globs cannot pick it up. +# +# is a chmod mode for the result ("-" = leave it alone, which is how +# the repo rewrite preserves the original file's mode and ownership). +_apt_https_atomic_replace() { + local target="$1" mode="$2" prep="$3"; shift 3 + local stage + stage="${target}.neohiro-rewrite.$$" + _apt_priv rm -f "$stage" 2>/dev/null || true + if ! "$prep" "$stage" "$@"; then + _apt_priv rm -f "$stage" 2>/dev/null || true + return 1 + fi + if [ "$mode" != "-" ]; then + if ! _apt_priv chmod "$mode" "$stage"; then + _apt_priv rm -f "$stage" 2>/dev/null || true + return 1 + fi + fi + if ! _apt_priv mv -f "$stage" "$target"; then + _apt_priv rm -f "$stage" 2>/dev/null || true + return 1 + fi + return 0 +} + +# _apt_https_prep_conf_stage — plain content copy. +_apt_https_prep_conf_stage() { + _apt_priv cp "$2" "$1" +} + +# _apt_https_prep_repo_stage +# +# Builds the staged rewrite so the renamed result keeps the original file's +# mode and ownership. `cp -p` clones those attributes onto the staging file, +# then `cp` over the *existing* staging file replaces only its content -- +# POSIX only applies the source's permissions when cp CREATES the +# destination, so the cloned attributes survive. +# +# This deliberately avoids `sed -i`: whether GNU sed, busybox sed, or a BSD +# variant preserve the mode across an in-place edit is implementation +# detail, and this file is edited while root on a machine whose package +# manager must keep working. +_apt_https_prep_repo_stage() { + _apt_priv cp -p "$2" "$1" && _apt_priv cp "$3" "$1" +} + # ── apt source rewriting ───────────────────────────────────────────────────── # # The three functions below report their result through globals @@ -351,19 +436,28 @@ _APT_HTTPS_REVERTED=0 # backup, or the privileged copy failed). Tracked separately so # apt_https_enforce never claims "already https" when it actually gave up. _APT_HTTPS_REFUSED=0 +# Latch for the CA-trust-store warning, which would otherwise repeat once per +# call site (enforce AND report both check it). This MUST be initialised here, +# at the top level: a report-only run never calls the rewrite path, so +# initialising it inside a function left it unbound under `set -u`. +_APT_HTTPS_CA_WARNED="" # _apt_https_rewrite_file # Sets _APT_HTTPS_VERDICT to "changed" or "clean". Backs the file up before # the first edit and refuses to edit a file it could not back up. _apt_https_rewrite_file() { - local f="$1" tmp expr + local f="$1" tmp expr="${2:-}" _APT_HTTPS_VERDICT="clean" [ -f "$f" ] || return 0 - case "$f" in - *.sources) expr='/^[[:space:]]*URIs:[[:space:]]/ s|http://|https://|g' ;; - *) expr='/^[[:space:]]*#/! s|http://|https://|g' ;; - esac + if [ -z "$expr" ]; then + # DEB822 must only be rewritten on the URIs: field; everything else is a + # flat key=value or URL-per-line format, where any active line counts. + case "$f" in + *.sources) expr='/^[[:space:]]*URIs:[[:space:]]/ s|http://|https://|g' ;; + *) expr="$_APT_HTTPS_GENERIC_RE" ;; + esac + fi tmp="$(_apt_https_stage)" if ! sed -E "$expr" "$f" > "$tmp" 2>/dev/null; then @@ -375,13 +469,14 @@ _apt_https_rewrite_file() { rm -f "$tmp" return 0 fi - # Refuse to edit anything we cannot first back up or cannot write. + # Refuse to edit anything we cannot first back up. if ! _apt_https_backup_once "$f"; then rm -f "$tmp" _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)) return 0 fi - if ! _apt_priv cp "$tmp" "$f"; then + # Atomic replace that keeps the original file's mode and ownership. + if ! _apt_https_atomic_replace "$f" - _apt_https_prep_repo_stage "$f" "$tmp"; then rm -f "$tmp" _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)) return 0 @@ -408,9 +503,9 @@ _apt_https_rewrite_apt_sources() { return 0 } -# _apt_https_revert_apt_sources — undo the rewrite from the backups. -# Sets _APT_HTTPS_REVERTED to the number of files restored. -_apt_https_revert_apt_sources() { +# _apt_https_revert_all_sources — undo the rewrite for EVERY source, not just +# apt. Sets _APT_HTTPS_REVERTED to the number of files restored. +_apt_https_revert_all_sources() { local f bak backupdir _APT_HTTPS_REVERTED=0 backupdir="$(apt_https_backup_dir)" @@ -424,7 +519,240 @@ _apt_https_revert_apt_sources() { info "Restored $f from backup" fi fi - done < <(_apt_https_repo_files) + done < <(_apt_https_managed_files) + return 0 +} + +# Kept as an alias: the original name is referenced by the inline fallback and +# by callers that only ever touched apt. +_apt_https_revert_apt_sources() { + _apt_https_revert_all_sources +} + +# ── App-store source registry ──────────────────────────────────────────────── +# The precaution is not apt-specific: every distribution and language has its +# own "app store", and several of them default to plaintext HTTP. This +# registry is the single list of every source this library inspects. +# +# Three per-source questions: +# files — where the transport is configured on disk +# env — where it is configured through the environment +# family — the repo-file dialect (drives which keys hold URLs) +# +# Detection is deliberately generic: ANY non-comment line containing an +# http:// URL in one of those files is reported. Per-dialect key regexes +# (baseurl=, Server=, index-url, registry, substituters, ...) would miss +# gpgkey=, metalkink= and whatever the next release adds, and a miss is +# exactly the failure mode this library exists to prevent. +# +# Rewriting is a separate, opt-in step (see apt_https_enforce) because +# whether https:// actually exists is not knowable +# without a network round trip. Rewriting blind would turn a working mirror +# into a broken one, which is worse than the threat. apt is the exception: +# it is verified with a real `apt-get update` and rolled back on failure. + +# _apt_https_source_ids — every source id, one per line. +_apt_https_source_ids() { + printf '%s\n' \ + apt dnf yum zypper pacman apk \ + flatpak snap docker brew pip npm cargo gem nix fwupd +} + +# _apt_https_source_label — human name for the report. +_apt_https_source_label() { + case "$1" in + apt) printf 'apt (Debian/Ubuntu/Mint/Pop!/Kali)' ;; + dnf) printf 'dnf (RHEL 8+/Fedora/Alma/Rocky)' ;; + yum) printf 'yum (CentOS 7/RHEL 7)' ;; + zypper) printf 'zypper (openSUSE/SLES)' ;; + pacman) printf 'pacman (Arch/Manjaro)' ;; + apk) printf 'apk (Alpine)' ;; + flatpak) printf 'flatpak remotes' ;; + snap) printf 'snap (store)' ;; + docker) printf 'docker registry config' ;; + brew) printf 'Homebrew taps' ;; + pip) printf 'pip index' ;; + npm) printf 'npm registry' ;; + cargo) printf 'cargo registry' ;; + gem) printf 'gem sources' ;; + nix) printf 'nix substituters/channels' ;; + fwupd) printf 'fwupd remotes (LVFS)' ;; + *) printf '%s' "$1" ;; + esac +} + +# _apt_https_source_files — config files that carry this source's URLs. +_apt_https_source_files() { + local etc f + etc="$(apt_https_etc_dir)" + case "$1" in + apt) _apt_https_repo_files ;; + dnf|yum) _apt_https_foreign_repo_files dnf ;; + zypper) _apt_https_foreign_repo_files zypper ;; + pacman) _apt_https_foreign_repo_files pacman ;; + # Alpine: one URL per line, http:// by default on many images. + apk) + [ -f "${etc}/apk/repositories" ] && printf '%s\n' "${etc}/apk/repositories" + ;; + # Registry mirrors / insecure-registries live in daemon.json. + docker) + [ -f "${etc}/docker/daemon.json" ] && printf '%s\n' "${etc}/docker/daemon.json" + ;; + # pip: system config plus the two variables that usually win. + pip) + for f in "${etc}/pip.conf" "${etc}/xdg/pip/pip.conf" \ + "${HOME:-}/.pip/pip.conf" "${HOME:-}/.config/pip/pip.conf"; do + [ -f "$f" ] && printf '%s\n' "$f" + done + ;; + npm) + for f in "${etc}/npmrc" "${HOME:-}/.npmrc"; do + [ -f "$f" ] && printf '%s\n' "$f" + done + ;; + cargo) + [ -f "${HOME:-}/.cargo/config.toml" ] && printf '%s\n' "${HOME:-}/.cargo/config.toml" + [ -f "${etc}/cargo/config.toml" ] && printf '%s\n' "${etc}/cargo/config.toml" + ;; + gem) + [ -f "${etc}/gemrc" ] && printf '%s\n' "${etc}/gemrc" + [ -f "${HOME:-}/.gemrc" ] && printf '%s\n' "${HOME:-}/.gemrc" + ;; + nix) + [ -f "${etc}/nix/nix.conf" ] && printf '%s\n' "${etc}/nix/nix.conf" + ;; + fwupd) + for f in "${etc}"/fwupd/remotes.d/*.conf; do + [ -f "$f" ] && printf '%s\n' "$f" + done + ;; + *) : ;; + esac + return 0 +} + +# _apt_https_source_env — transport configured through the environment. +# Printed as "VAR=value" so the user gets a copy-pasteable fix. +_apt_https_source_env() { + case "$1" in + pip) + [ -n "${PIP_INDEX_URL:-}" ] && printf 'PIP_INDEX_URL=%s\n' "$PIP_INDEX_URL" + [ -n "${PIP_EXTRA_INDEX_URL:-}" ] && printf 'PIP_EXTRA_INDEX_URL=%s\n' "$PIP_EXTRA_INDEX_URL" + ;; + npm) + [ -n "${NPM_CONFIG_REGISTRY:-}" ] && printf 'NPM_CONFIG_REGISTRY=%s\n' "$NPM_CONFIG_REGISTRY" + [ -n "${npm_config_registry:-}" ] && printf 'npm_config_registry=%s\n' "$npm_config_registry" + ;; + cargo) + [ -n "${CARGO_REGISTRIES_CRATES_IO_INDEX:-}" ] && \ + printf 'CARGO_REGISTRIES_CRATES_IO_INDEX=%s\n' "$CARGO_REGISTRIES_CRATES_IO_INDEX" + ;; + gem) + [ -n "${GEM_SOURCE:-}" ] && printf 'GEM_SOURCE=%s\n' "$GEM_SOURCE" + ;; + brew) + [ -n "${HOMEBREW_BREW_GIT_REMOTE:-}" ] && printf 'HOMEBREW_BREW_GIT_REMOTE=%s\n' "$HOMEBREW_BREW_GIT_REMOTE" + [ -n "${HOMEBREW_CORE_GIT_REMOTE:-}" ] && printf 'HOMEBREW_CORE_GIT_REMOTE=%s\n' "$HOMEBREW_CORE_GIT_REMOTE" + [ -n "${HOMEBREW_API_DOMAIN:-}" ] && printf 'HOMEBREW_API_DOMAIN=%s\n' "$HOMEBREW_API_DOMAIN" + [ -n "${HOMEBREW_ARTIFACT_DOMAIN:-}" ] && printf 'HOMEBREW_ARTIFACT_DOMAIN=%s\n' "$HOMEBREW_ARTIFACT_DOMAIN" + ;; + *) : ;; + esac + return 0 +} + +# _apt_https_source_applicable — false when this host has no such store. +# Keeps the report from listing eleven N/A rows on a minimal box. +_apt_https_source_applicable() { + local id="$1" + case "$id" in + # Only the package manager actually present should be audited as such. + dnf|yum|zypper|pacman) + [ "$(apt_https_family)" = "$id" ] || return 1 + ;; + snap) + command -v snap >/dev/null 2>&1 || return 1 + ;; + flatpak) + command -v flatpak >/dev/null 2>&1 || return 1 + ;; + docker) + command -v docker >/dev/null 2>&1 || return 1 + ;; + brew) + command -v brew >/dev/null 2>&1 || return 1 + ;; + pip) + command -v pip3 >/dev/null 2>&1 || command -v pip >/dev/null 2>&1 || return 1 + ;; + npm) + command -v npm >/dev/null 2>&1 || return 1 + ;; + cargo) + command -v cargo >/dev/null 2>&1 || return 1 + ;; + gem) + command -v gem >/dev/null 2>&1 || return 1 + ;; + apk) + command -v apk >/dev/null 2>&1 || return 1 + ;; + *) : ;; + esac + return 0 +} + +# _apt_https_source_plaintext — report this source's plaintext endpoints. +_apt_https_source_plaintext() { + local id="$1" f + case "$id" in + apt) + while IFS= read -r f; do + [ -n "$f" ] && _apt_https_plaintext_in_apt_file "$f" + done < <(_apt_https_repo_files) + ;; + flatpak) + _apt_https_flatpak_plaintext + ;; + *) + while IFS= read -r f; do + [ -n "$f" ] && _apt_https_plaintext_generic "$f" "$id" + done < <(_apt_https_source_files "$id") + ;; + esac + _apt_https_source_env "$id" + return 0 +} + +# _apt_https_source_prefer_https +# Rewrites http:// -> https:// on non-comment lines of every config file for +# this source. Sets _APT_HTTPS_CHANGED / _APT_HTTPS_REFUSED. Opt-in only, +# except for apt (see apt_https_enforce). +_apt_https_source_prefer_https() { + local id="$1" f + _APT_HTTPS_CHANGED=0 + _APT_HTTPS_REFUSED=0 + [ "$id" = "apt" ] && { _apt_https_rewrite_apt_sources; return 0; } + while IFS= read -r f; do + [ -n "$f" ] || continue + _apt_https_rewrite_file "$f" + if [ "$_APT_HTTPS_VERDICT" = "changed" ]; then + _APT_HTTPS_CHANGED=$((_APT_HTTPS_CHANGED + 1)) + info "Rewrote plaintext URLs -> https: $f" + fi + done < <(_apt_https_source_files "$id") + return 0 +} + +# _apt_https_managed_files — every file this library may have edited. +_apt_https_managed_files() { + local id f + while IFS= read -r id; do + [ -n "$id" ] || continue + while IFS= read -r f; do + [ -n "$f" ] && printf '%s\n' "$f" + done < <(_apt_https_source_files "$id") + done < <(_apt_https_source_ids) return 0 } @@ -493,6 +821,10 @@ _apt_https_flatpak_plaintext() { # HTTPS verification is worthless without a trust store. Warn (never fail) # when it is missing, because installing it would itself need a download. +# +# Latched to once per process: apt_https_enforce and apt_https_report both +# check this, and the maintenance menu and --apt-https path call both, which +# used to print the identical warning two or three times in one run. _apt_https_check_ca_certs() { local fam="$1" p case "$fam" in @@ -507,6 +839,8 @@ _apt_https_check_ca_certs() { pacman) [ -e /etc/ssl/certs/ca-certificates.crt ] && return 0; p="ca-certificates" ;; *) return 0 ;; esac + [ -n "$_APT_HTTPS_CA_WARNED" ] && return 1 + _APT_HTTPS_CA_WARNED=1 warn "No system CA trust store found. HTTPS verification of packages will fail." info "Install '$p' after the sources are trusted, then re-run: apt_https_enforce" return 1 @@ -549,54 +883,75 @@ _apt_https_block_port80() { # ── Status / report ────────────────────────────────────────────────────────── -# apt_https_status_text — every plaintext repo line still configured. +# apt_https_status_text — every plaintext endpoint across every app store. +# One line per offending config line, plus "VAR=value" for env-configured +# transports, so the output is directly actionable. apt_https_status_text() { - local fam - fam="$(apt_https_family)" - _apt_https_audit_family "$fam" - _apt_https_flatpak_plaintext + local id + while IFS= read -r id; do + [ -n "$id" ] || continue + _apt_https_source_applicable "$id" || continue + _apt_https_source_plaintext "$id" + done < <(_apt_https_source_ids) return 0 } -# apt_https_report — human-readable state summary. -# Returns 0 when everything is https, 1 when plaintext remains, 2 when the -# family is unknown. +# _apt_https_plaintext_for — findings for one source only. +apt_https_plaintext_for() { + _apt_https_source_applicable "$1" || return 0 + _apt_https_source_plaintext "$1" +} + +# apt_https_report — per-source state summary. +# Returns 0 when nothing anywhere uses plaintext, 1 when any source does, +# 2 when no package manager at all was detected. apt_https_report() { - local fam missing conf findings + local fam conf findings id label n clean applicable fam="$(apt_https_family)" - printf '\n%s\n' "$(_c '1;36m' '━━━ Package transport security (HTTPS) ━━━')" - printf ' Detected package manager: %s\n' "$fam" + printf '\n%s\n' "$(_c '1;36m' '━━━ App-store transport security (HTTPS) ━━━')" + printf ' %-34s %s\n' "Package manager:" "$fam" + if [ "$fam" = "apt" ]; then + conf="$(apt_https_conf_file)" + if [ -f "$conf" ]; then + printf ' %s %s\n' "$(_c '1;32m' '[x]')" "apt policy drop-in active: $conf" + else + printf ' %s %s\n' "$(_c '1;31m' '[ ]')" "apt policy drop-in MISSING: $conf" + fi + fi + printf '\n %s\n' "$(_c '1;37m' 'Store Result')" + + clean=1 + while IFS= read -r id; do + [ -n "$id" ] || continue + _apt_https_source_applicable "$id" || continue + label="$(_apt_https_source_label "$id")" + findings="$(apt_https_plaintext_for "$id")" + if [ -z "$findings" ]; then + printf ' %-30s %s\n' "$label" "$(_c '1;32m' 'https only')" + else + n="$(printf '%s\n' "$findings" | wc -l | tr -d ' ')" + printf ' %-30s %s\n' "$label" "$(_c '1;31m' "$n plaintext endpoint(s)")" + printf '%s\n' "$findings" | sed 's/^/ /' + clean=0 + fi + done < <(_apt_https_source_ids) - case "$fam" in - apt) - conf="$(apt_https_conf_file)" - if [ -f "$conf" ]; then - printf ' %s %s\n' "$(_c '1;32m' '[x]')" "apt policy drop-in active: $conf" - else - printf ' %s %s\n' "$(_c '1;31m' '[ ]')" "apt policy drop-in MISSING: $conf" - fi - ;; - dnf|yum) printf ' %s\n' "dnf/yum: repo files audited (no blind rewrite)." ;; - zypper) printf ' %s\n' "zypper: repo files audited (no blind rewrite)." ;; - pacman) printf ' %s\n' "pacman: mirrorlist audited (no blind rewrite)." ;; - *) printf ' %s\n' "No supported package manager detected." ;; - esac + _apt_https_check_ca_certs "$fam" || true - findings="$(apt_https_status_text)" - if [ -n "$findings" ]; then - printf '\n %s\n' "$(_c '1;33m' 'Plaintext (http://) repository entries still present:')" - printf '%s\n' "$findings" | sed 's/^/ /' - printf '\n %s\n' " apt: fix with sudo bash linuxinstall.sh --apt-https" - printf '%s\n' " others: set NEOHIRO_APT_HTTPS_REWRITE=1, then re-run the audit" + printf '\n' + if [ "$clean" = "1" ]; then + printf ' %s %s\n' "$(_c '1;32m' '[x]')" "No plaintext endpoints in any configured app store." else - printf '\n %s %s\n' "$(_c '1;32m' '[x]')" "No plaintext repository URLs detected." + printf ' %s %s\n' "$(_c '1;33m' '[!]')" "Some stores still fetch over plaintext HTTP." + printf ' %s\n' " apt: sudo bash linuxinstall.sh --apt-https (automatic, verified)" + printf '%s\n' " others: repoint at an https:// endpoint, or opt in with" + printf '%s\n' " NEOHIRO_APT_HTTPS_REWRITE=1 and re-audit afterwards." fi - _apt_https_check_ca_certs "$fam" || true printf '\n' if [ "$fam" = "none" ]; then return 2; fi - [ -n "$findings" ] && return 1 - return 0 + [ "$clean" = "1" ] && return 0 + return 1 } # ── Enforce ────────────────────────────────────────────────────────────────── @@ -628,11 +983,11 @@ _apt_https_verify_apt_sources() { # apt_https_enforce [label] — apply the precaution once, unconditionally. # [label] is a short breadcrumb describing the caller, e.g. "pkg_install". apt_https_enforce() { - local label="${1:-enforce}" mode fam conf_failed findings rc=0 + local label="${1:-enforce}" mode fam conf_failed findings rc=0 rewrite id n mode="${NEOHIRO_APT_HTTPS:-1}" fam="$(apt_https_family)" - if [ "$fam" = "none" ]; then + if [ "$fam" = "none" ] && [ "$mode" != "audit" ]; then info "No supported package manager detected — HTTPS guard not applicable." return 0 fi @@ -642,7 +997,7 @@ apt_https_enforce() { return 0 fi - msg "Repository transport guard ($label): $fam" + msg "Repository transport guard ($label): package manager = $fam" if [ "$mode" = "audit" ]; then info "NEOHIRO_APT_HTTPS=audit — reporting only, nothing is modified." @@ -656,48 +1011,62 @@ apt_https_enforce() { return 0 fi - case "$fam" in - apt) - conf_failed=0 - _apt_https_write_conf || conf_failed=1 - _apt_https_rewrite_apt_sources - if [ "$_APT_HTTPS_CHANGED" -gt 0 ]; then - ok "Rewrote $_APT_HTTPS_CHANGED apt source file(s) to https://" - _apt_https_verify_apt_sources || rc=1 - elif [ "$_APT_HTTPS_REFUSED" -gt 0 ]; then - warn "Could not rewrite $_APT_HTTPS_REFUSED apt source file(s): no writable backup." - info "Check that $(apt_https_backup_dir) is writable by root, then re-run." - rc=1 - else - ok "apt repositories already use https://" - fi - if [ "$conf_failed" = "1" ]; then - warn "Could not install the apt policy drop-in." - rc=1 + # ---- automatic: apt only ------------------------------------------------- + # The one store we rewrite unattended, because a real `apt-get update` can + # verify the result and roll the rewrite back if the mirror cannot speak TLS. + if [ "$fam" = "apt" ]; then + conf_failed=0 + _apt_https_write_conf || conf_failed=1 + _apt_https_rewrite_apt_sources + if [ "$_APT_HTTPS_CHANGED" -gt 0 ]; then + ok "Rewrote $_APT_HTTPS_CHANGED apt source file(s) to https://" + _apt_https_verify_apt_sources || rc=1 + elif [ "$_APT_HTTPS_REFUSED" -gt 0 ]; then + warn "Could not rewrite $_APT_HTTPS_REFUSED apt source file(s): no writable backup." + info "Check that $(apt_https_backup_dir) is writable by root, then re-run." + rc=1 + else + ok "apt repositories already use https://" + fi + if [ "$conf_failed" = "1" ]; then + warn "Could not install the apt policy drop-in." + rc=1 + fi + fi + + # ---- opt-in: every other store ------------------------------------------- + # Whether https:// exists is unknowable without a + # network round trip, so this is never automatic. + if [ "${NEOHIRO_APT_HTTPS_REWRITE:-0}" = "1" ]; then + while IFS= read -r id; do + [ -n "$id" ] || continue + [ "$id" = "apt" ] && continue + _apt_https_source_applicable "$id" || continue + _apt_https_source_prefer_https "$id" + n="$_APT_HTTPS_CHANGED" + if [ "${n:-0}" -gt 0 ] 2>/dev/null; then + ok "Rewrote $n $(_apt_https_source_label "$id") file(s) to https://" fi - ;; - dnf|yum|zypper|pacman) - if [ "${NEOHIRO_APT_HTTPS_REWRITE:-0}" = "1" ]; then - _apt_https_rewrite_family "$fam" - if [ "$_APT_HTTPS_CHANGED" -gt 0 ]; then - ok "Rewrote $_APT_HTTPS_CHANGED $fam repo file(s) to https:// (NEOHIRO_APT_HTTPS_REWRITE=1)" - info "Re-check with --apt-https-audit; not every mirror serves the same paths over TLS." - fi + if [ "${_APT_HTTPS_REFUSED:-0}" -gt 0 ] 2>/dev/null; then + warn "Could not rewrite $_APT_HTTPS_REFUSED $(_apt_https_source_label "$id") file(s): no writable backup." fi - ;; - *) : ;; - esac + done < <(_apt_https_source_ids) + info "Rewrote non-apt stores because NEOHIRO_APT_HTTPS_REWRITE=1." + info "Re-audit with --apt-https-audit; not every mirror serves the same paths over TLS." + fi - # A verification failure rolls the rewrite back; report whatever remains. + # ---- report --------------------------------------------------------------- findings="$(apt_https_status_text)" if [ -n "$findings" ]; then - warn "Plaintext repository URLs remain for $fam:" + warn "Plaintext endpoints still configured:" printf '%s\n' "$findings" | sed 's/^/ /' - if [ "$fam" != "apt" ]; then - info "Point those at an https-capable mirror, or opt in to a blind rewrite" - info "with NEOHIRO_APT_HTTPS_REWRITE=1 (verify afterwards)." - [ "${NEOHIRO_APT_HTTPS_STRICT:-0}" = "1" ] && rc=1 + [ "$fam" = "apt" ] || info "Repoint these at an https:// endpoint, or opt in to a" + [ "$fam" = "apt" ] || info "blind rewrite with NEOHIRO_APT_HTTPS_REWRITE=1." + if [ "${NEOHIRO_APT_HTTPS_STRICT:-0}" = "1" ]; then + rc=1 fi + else + ok "Every configured app store uses https://" fi _apt_https_check_ca_certs "$fam" || true diff --git a/linuxinstall.sh b/linuxinstall.sh index df1f4f5..97e5ddf 100644 --- a/linuxinstall.sh +++ b/linuxinstall.sh @@ -310,6 +310,25 @@ if ! declare -F apt_https_guard >/dev/null 2>&1; then if declare -F _tmpfile >/dev/null 2>&1; then _tmpfile apt-https; return 0; fi mktemp "${TMPDIR:-/tmp}/neohiro-apt-https.XXXXXX" } + # Atomic replace: build the result in a staging file inside the target's + # own directory, then rename it over the target. `cp` into place truncates + # first, so a crash mid-copy would leave a truncated repo file; rename(2) on + # one filesystem cannot. The staging name ends in the PID, so it never ends + # in .list/.sources and apt's own globs cannot see it. + _apt_https_atomic_replace() { + local target="$1" mode="$2" prep="$3"; shift 3 + local stage + stage="${target}.neohiro-rewrite.$$" + _apt_priv rm -f "$stage" 2>/dev/null || true + if ! "$prep" "$stage" "$@"; then _apt_priv rm -f "$stage" 2>/dev/null || true; return 1; fi + if [ "$mode" != "-" ]; then + if ! _apt_priv chmod "$mode" "$stage"; then _apt_priv rm -f "$stage" 2>/dev/null || true; return 1; fi + fi + if ! _apt_priv mv -f "$stage" "$target"; then _apt_priv rm -f "$stage" 2>/dev/null || true; return 1; fi + return 0 + } + _apt_https_prep_conf_stage() { _apt_priv cp "$2" "$1"; } + _apt_https_prep_repo_stage() { _apt_priv cp -p "$2" "$1" && _apt_priv cp "$3" "$1"; } _apt_https_write_conf() { local conf tmp rc conf="$(apt_https_conf_file)" @@ -338,9 +357,7 @@ if ! declare -F apt_https_guard >/dev/null 2>&1; then if [ -f "$conf" ] && cmp -s "$tmp" "$conf"; then rm -f "$tmp"; return 0; fi if ! _apt_https_backup_once "$conf"; then rm -f "$tmp"; return 1; fi _apt_priv mkdir -p "$(apt_https_etc_dir)/apt/apt.conf.d" || { rm -f "$tmp"; return 1; } - if _apt_priv install -m 0644 "$tmp" "$conf"; then - rc=0 - elif _apt_priv cp "$tmp" "$conf" && _apt_priv chmod 0644 "$conf"; then + if _apt_https_atomic_replace "$conf" 0644 _apt_https_prep_conf_stage "$tmp"; then rc=0 else rc=1 @@ -356,6 +373,9 @@ if ! declare -F apt_https_guard >/dev/null 2>&1; then _APT_HTTPS_CHANGED=0 _APT_HTTPS_REVERTED=0 _APT_HTTPS_REFUSED=0 + # Warning latch. Declared at the top level so an audit-only run under + # `set -u` never sees it unbound. + _APT_HTTPS_CA_WARNED="" _apt_https_rewrite_file() { local f="$1" tmp expr _APT_HTTPS_VERDICT="clean" @@ -368,13 +388,15 @@ if ! declare -F apt_https_guard >/dev/null 2>&1; then if ! sed -E "$expr" "$f" > "$tmp" 2>/dev/null; then rm -f "$tmp"; _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)); return 0; fi if cmp -s "$tmp" "$f"; then rm -f "$tmp"; return 0; fi if ! _apt_https_backup_once "$f"; then rm -f "$tmp"; _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)); return 0; fi - if ! _apt_priv cp "$tmp" "$f"; then rm -f "$tmp"; _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)); return 0; fi + if ! _apt_https_atomic_replace "$f" - _apt_https_prep_repo_stage "$f" "$expr"; then + rm -f "$tmp"; _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)); return 0 + fi rm -f "$tmp" _APT_HTTPS_VERDICT="changed" return 0 } _apt_https_rewrite_apt_sources() { - local f +local f _APT_HTTPS_CHANGED=0 _APT_HTTPS_REFUSED=0 while IFS= read -r f; do @@ -430,15 +452,24 @@ if ! declare -F apt_https_guard >/dev/null 2>&1; then return 0 } _apt_https_check_ca_certs() { - local fam="$1" + local fam="$1" missing=0 + # Latched: both apt_https_enforce and apt_https_report check this, so + # without a latch one run printed the same warning two or three times. + # Each arm must fall through to the same `missing` flag; returning from + # inside the case would skip the latch and warn on a healthy host. case "$fam" in - apt) { [ -e /etc/ssl/certs/ca-certificates.crt ] || [ -e /etc/pki/tls/certs/ca-bundle.crt ]; } || { warn "No system CA trust store found; HTTPS package verification will fail."; return 1; } ;; - dnf|yum) [ -e /etc/pki/tls/certs/ca-bundle.crt ] || { warn "No system CA trust store found."; return 1; } ;; - zypper) [ -e /etc/ssl/ca-bundle.pem ] || { warn "No system CA trust store found."; return 1; } ;; - pacman) [ -e /etc/ssl/certs/ca-certificates.crt ] || { warn "No system CA trust store found."; return 1; } ;; - *) : ;; + apt) { [ -e /etc/ssl/certs/ca-certificates.crt ] || [ -e /etc/pki/tls/certs/ca-bundle.crt ]; } || missing=1 ;; + dnf|yum) [ -e /etc/pki/tls/certs/ca-bundle.crt ] || missing=1 ;; + zypper) [ -e /etc/ssl/ca-bundle.pem ] || missing=1 ;; + pacman) [ -e /etc/ssl/certs/ca-certificates.crt ] || missing=1 ;; + *) return 0 ;; esac - return 0 + [ "$missing" = "0" ] && return 0 + [ -n "${_APT_HTTPS_CA_WARNED:-}" ] && return 1 + _APT_HTTPS_CA_WARNED=1 + warn "No system CA trust store found. HTTPS verification of packages will fail." + info "Install 'ca-certificates' after the sources are trusted, then re-run: apt_https_enforce" + return 1 } _apt_https_block_port80() { [ "${NEOHIRO_APT_BLOCK_PORT80:-0}" = "1" ] || return 0 diff --git a/tests/test_apt_https.sh b/tests/test_apt_https.sh index 8c3e90b..6ae5128 100644 --- a/tests/test_apt_https.sh +++ b/tests/test_apt_https.sh @@ -39,8 +39,9 @@ trap cleanup EXIT SANDBOX="$WD/etc" BACKUPS="$WD/backups" +FAKEHOME="$WD/home" FAKEBIN="$WD/bin" -mkdir -p "$FAKEBIN" +mkdir -p "$FAKEBIN" "$FAKEHOME" # Fake ufw: `ufw status` reports nothing installed yet; any mutating call is # recorded so a test can assert it happened (or did not). @@ -86,10 +87,32 @@ reset_sandbox() { export NEOHIRO_APT_HTTPS_NOVERIFY=1 unset NEOHIRO_APT_HTTPS NEOHIRO_APT_HTTPS_REWRITE NEOHIRO_APT_FAMILY \ NEOHIRO_APT_HTTPS_STRICT NEOHIRO_APT_BLOCK_PORT80 2>/dev/null || true + # Hermeticity: the pip/npm/cargo/gem detectors read config from $HOME and + # transport from the environment. Point HOME at the sandbox and clear every + # transport variable so the developer's real ~/.npmrc or PIP_INDEX_URL + # cannot make this suite report findings it did not create. + export HOME="$FAKEHOME" + unset PIP_INDEX_URL PIP_EXTRA_INDEX_URL NPM_CONFIG_REGISTRY npm_config_registry \ + CARGO_REGISTRIES_CRATES_IO_INDEX GEM_SOURCE \ + HOMEBREW_BREW_GIT_REMOTE HOMEBREW_CORE_GIT_REMOTE \ + HOMEBREW_API_DOMAIN HOMEBREW_ARTIFACT_DOMAIN 2>/dev/null || true _APT_HTTPS_DONE="" DRY_RUN=0 } +# _mode_supported — false on filesystems that do not model POSIX permissions +# (Git Bash on NTFS, some container overlay/CIFS mounts). Permission +# assertions must self-skip there rather than fail spuriously. +_mode_supported() { + local f="$WD/modeprobe" + : > "$f" + chmod 0600 "$f" 2>/dev/null + if [ "$(ls -l "$f" 2>/dev/null | cut -c1-10)" = "-rw-------" ]; then + rm -f "$f"; return 0 + fi + rm -f "$f"; return 1 +} + # ============================================================================ # NEOHIRO_APT_FAMILY pin (this is what the CLI paths use) # ============================================================================ @@ -499,7 +522,7 @@ if grep -q '^baseurl=http://' "$SANDBOX/yum.repos.d/epel.repo"; then else fail_t "dnf: no blind rewrite by default" "file was rewritten without opt-in" fi -if printf '%s' "$DNFOUT" | grep -q 'Plaintext repository URLs remain'; then +if printf '%s' "$DNFOUT" | grep -q 'Plaintext endpoints still configured'; then ok_t "dnf: leftover plaintext is reported loudly" else fail_t "dnf: leftover plaintext is reported loudly" "no warning emitted" @@ -997,7 +1020,7 @@ else fail_t "restore_ssh.sh --apt-https-audit exits 0 on a clean system" \ "$(printf '%s' "$OUT" | grep '__RC=')" fi -if printf '%s' "$OUT" | grep -q 'Package transport security'; then +if printf '%s' "$OUT" | grep -q 'App-store transport security'; then ok_t "restore_ssh.sh --apt-https-audit prints the report" else fail_t "restore_ssh.sh --apt-https-audit prints the report" "report missing" @@ -1060,7 +1083,7 @@ else fail_t "restore_ssh.sh --apt-https-audit exits 0 on a clean system" \ "$(printf '%s' "$OUT" | grep '__RC=')" fi -if printf '%s' "$OUT" | grep -q 'Package transport security'; then +if printf '%s' "$OUT" | grep -q 'App-store transport security'; then ok_t "restore_ssh.sh --apt-https-audit prints the report" else fail_t "restore_ssh.sh --apt-https-audit prints the report" "report missing" @@ -1087,7 +1110,7 @@ else fail_t "restore_ssh.sh --apt-https propagates enforcement failure (not 0)" \ "$(printf '%s' "$OUT" | grep '__RC=')" fi -if printf '%s' "$OUT" | grep -q 'Package transport security'; then +if printf '%s' "$OUT" | grep -q 'App-store transport security'; then ok_t "restore_ssh.sh --apt-https prints the report even when enforce fails" else fail_t "restore_ssh.sh --apt-https prints the report even when enforce fails" \ @@ -1120,7 +1143,7 @@ else fail_t "lib/updater.sh --apt-https propagates enforcement failure" \ "$(printf '%s' "$UPD_OUT" | grep '__RC=')" fi -if printf '%s' "$UPD_OUT" | grep -q 'Package transport security'; then +if printf '%s' "$UPD_OUT" | grep -q 'App-store transport security'; then ok_t "lib/updater.sh --apt-https prints the report even when enforce fails" else fail_t "lib/updater.sh --apt-https prints the report even when enforce fails" \ @@ -1224,6 +1247,190 @@ else fail_t "lib/apt-https.sh exits 2 on an unknown argument" "rc=$?" fi +# ============================================================================ +# Regression: set -u safety of every module-level global +# ============================================================================ +# A guard variable initialised inside a function instead of at the top level +# is unbound on any code path that does not call that function, which kills +# an audit-only run under `set -u`. Load the lib the way the strictest host +# does and touch every path that reads these globals. +cat > "$WD/setu_probe.sh" <<'SETU_EOF' +set -u +. "$RS_LIB" >/dev/null 2>&1 +apt_https_report >/dev/null 2>&1 +printf 'REPORT=%s\n' "$?" +apt_https_status_text >/dev/null +printf 'STATUS=ok\n' +apt_https_plaintext_for apt >/dev/null 2>&1 +printf 'FORAPT=%s\n' "$?" +apt_https_revert >/dev/null 2>&1 +printf 'REVERT=%s\n' "$?" +SETU_EOF +SETU_OUT="$(env RS_LIB="$LIB" NEOHIRO_APT_ETC_DIR="$SANDBOX" \ + NEOHIRO_APT_BACKUP_DIR="$BACKUPS" HOME="$FAKEHOME" \ + NEOHIRO_APT_FAMILY=apt NEOHIRO_APT_HTTPS_NOVERIFY=1 \ + bash "$WD/setu_probe.sh" 2>&1)" +if printf '%s' "$SETU_OUT" | grep -q 'unbound variable'; then + fail_t "lib is set -u clean on an audit-only path" \ + "$(printf '%s' "$SETU_OUT" | grep 'unbound' | head -1)" +else + ok_t "lib is set -u clean on an audit-only path" +fi +if printf '%s' "$SETU_OUT" | grep -q 'REPORT=' && \ + printf '%s' "$SETU_OUT" | grep -q 'STATUS=ok' && \ + printf '%s' "$SETU_OUT" | grep -q 'FORAPT=' && \ + printf '%s' "$SETU_OUT" | grep -q 'REVERT='; then + ok_t "set -u harness reached report, status, per-source and revert" +else + fail_t "set -u harness reached report, status, per-source and revert" \ + "got: $(printf '%s' "$SETU_OUT" | tr '\n' ' ')" +fi + +# Structural: the latches and result globals must be initialised at the top +# level of the file, not inside a function body. +for v in _APT_HTTPS_VERDICT _APT_HTTPS_CHANGED _APT_HTTPS_REVERTED \ + _APT_HTTPS_REFUSED _APT_HTTPS_CA_WARNED; do + if grep -qE "^${v}=|^ ${v}=" "$LIB"; then + ok_t "$v is initialised at the top level of lib/apt-https.sh" + else + fail_t "$v is initialised at the top level of lib/apt-https.sh" \ + "no top-level assignment found" + fi + if grep -qE "^ ${v}=" "$SRC"; then + ok_t "$v is initialised at the top level of the inline fallback" + else + fail_t "$v is initialised at the top level of the inline fallback" \ + "no top-level assignment found" + fi +done + +# ============================================================================ +# CA-trust-store check: latch, and no false warning on a healthy host +# ============================================================================ +# The probe paths are absolute system paths that a non-root test cannot +# create, so the "trust store present" branch is pinned structurally rather +# than by faking the filesystem. Getting this wrong is what caused a run to +# warn on every invocation, so the guard against it matters. +CA_BODY="$(awk '/^_apt_https_check_ca_certs\(\) \{/,/^\}/' "$LIB")" +if printf '%s' "$CA_BODY" | grep -q '_APT_HTTPS_CA_WARNED'; then + ok_t "CA warning is latched to once per process (lib)" +else + fail_t "CA warning is latched to once per process (lib)" "no latch found" +fi +# Either shape is acceptable as long as a present trust store returns before +# the warning: an early `return 0` inside the case, or a "missing" flag. +if printf '%s' "$CA_BODY" | grep -qE 'then[[:space:]]*$|missing=0|\[ "\$missing" = "0" \] && return 0' && \ + printf '%s' "$CA_BODY" | grep -q 'return 0'; then + ok_t "CA check returns before warning when a trust store IS present (lib)" +else + fail_t "CA check returns before warning when a trust store IS present (lib)" \ + "no early return: a case arm can fall through and warn on a healthy host" +fi +CA_INLINE="$(awk '/^[[:space:]]+_apt_https_check_ca_certs\(\) \{/,/^[[:space:]]+\}/' "$SRC")" +if printf '%s' "$CA_INLINE" | grep -qE '\[ "\$missing" = "0" \] && return 0'; then + ok_t "CA check returns early when a trust store IS present (inline fallback)" +else + fail_t "CA check returns early when a trust store IS present (inline fallback)" \ + "inline copy lacks the early return" +fi + +# Behavioural half: this host has no /etc CA bundle, so the missing branch is +# live -- assert it warns exactly once no matter how many call sites fire. +reset_sandbox +CA_CALLS="$(bash -c " + . '$LIB' >/dev/null 2>&1 + warn() { printf 'WARN\n'; } + info() { printf 'INFO\n'; } + _apt_https_check_ca_certs apt >/dev/null 2>&1 || true + _apt_https_check_ca_certs apt >/dev/null 2>&1 || true + _apt_https_check_ca_certs apt >/dev/null 2>&1 || true +" 2>&1)" +CA_WARNS="$(printf '%s\n' "$CA_CALLS" | grep -c '^WARN$' || true)" +if [ "$CA_WARNS" -le 1 ]; then + ok_t "CA warning emitted at most once across repeated checks" +else + fail_t "CA warning emitted at most once across repeated checks" "saw $CA_WARNS warnings" +fi + +# ============================================================================ +# Regression: atomic replace leaves no staging residue +# ============================================================================ +reset_sandbox +printf 'deb http://atomic.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" +printf 'deb [arch=amd64] http://atomic2.example.com/debian main\n' >> "$SANDBOX/apt/sources.list" +printf 'Types: deb\nURIs: http://atomic3.example.com/debian main\n' > "$SANDBOX/apt/sources.list.d/a.sources" +_APT_HTTPS_DONE="" +apt_https_enforce "atomic" >/dev/null 2>&1 +RESIDUE="$(find "$SANDBOX" -name '*neohiro-rewrite*' 2>/dev/null | wc -l | tr -d ' ')" +if [ "$RESIDUE" = "0" ]; then + ok_t "atomic rewrite leaves no staging files behind" +else + fail_t "atomic rewrite leaves no staging files behind" \ + "$RESIDUE residue file(s): $(find "$SANDBOX" -name '*neohiro-rewrite*' 2>/dev/null | tr '\n' ' ')" +fi + +# The rewritten content must be correct. +if grep -q 'https://atomic.example.com' "$SANDBOX/apt/sources.list" && \ + grep -q 'https://atomic3.example.com' "$SANDBOX/apt/sources.list.d/a.sources"; then + ok_t "atomic rewrite produced the expected content" +else + fail_t "atomic rewrite produced the expected content" "content wrong after rename" +fi + +# Mode preservation is only observable on a filesystem that models POSIX +# permissions. Git Bash on NTFS reports 0644 for everything, so asserting it +# there would fail spuriously. Skip explicitly rather than pretend to pass. +if _mode_supported; then + printf 'deb http://mode.example.com/ubuntu main\n' > "$SANDBOX/apt/sources.list" + chmod 0640 "$SANDBOX/apt/sources.list" + _APT_HTTPS_DONE="" + apt_https_enforce "atomic-mode" >/dev/null 2>&1 + MODE_AFTER="$(ls -l "$SANDBOX/apt/sources.list" 2>/dev/null | cut -c1-10)" + case "$MODE_AFTER" in + -rw-r-----) ok_t "atomic rewrite preserves the original file mode (0640)" ;; + *) fail_t "atomic rewrite preserves the original file mode (0640)" "got: $MODE_AFTER" ;; + esac +else + ok_t "atomic rewrite preserves the original file mode (skipped: filesystem does not model POSIX modes)" +fi + +# The rewrite must clone attributes with cp -p and then overwrite content, +# never rely on `sed -i` keeping the mode (implementation detail, differs +# between GNU and busybox sed). +PREP_BODY="$(awk '/^_apt_https_prep_repo_stage\(\) \{/,/^\}/' "$LIB")" +if printf '%s' "$PREP_BODY" | grep -q 'cp -p' && printf '%s' "$PREP_BODY" | grep -q 'cp "\$3"'; then + ok_t "repo staging clones attributes with cp -p then replaces content" +else + fail_t "repo staging clones attributes with cp -p then replaces content" \ + "unexpected staging implementation: $PREP_BODY" +fi +if printf '%s' "$PREP_BODY" | grep -q 'sed -i'; then + fail_t "repo staging does not rely on sed -i for mode preservation" "sed -i still used" +else + ok_t "repo staging does not rely on sed -i for mode preservation" +fi + +# The policy drop-in must still land as 0644. +CONF_MODE="$(ls -l "$(apt_https_conf_file)" 2>/dev/null | cut -c1-10)" +case "$CONF_MODE" in + -rw-r--r--) ok_t "atomic drop-in install lands as 0644" ;; + *) fail_t "atomic drop-in install lands as 0644" "got: $CONF_MODE" ;; +esac + +# Structural: the rewrite must go through mv, not cp-into-place. +if grep -q '_apt_https_atomic_replace' "$LIB" && grep -q '_apt_https_atomic_replace' "$SRC"; then + ok_t "lib and inline fallback both route writes through _apt_https_atomic_replace" +else + fail_t "lib and inline fallback both route writes through _apt_https_atomic_replace" \ + "atomic helper not wired in one of the two copies" +fi +if grep -q '_apt_priv mv -f "\$stage" "\$target"' "$LIB" && \ + grep -q '_apt_priv mv -f "\$stage" "\$target"' "$SRC"; then + ok_t "atomic replace finishes with rename(2) via mv -f" +else + fail_t "atomic replace finishes with rename(2) via mv -f" "mv of the stage file missing" +fi + # ============================================================================ # Summary # ============================================================================ From 173eb9865b5a9772fac5638a889ed88575316a12 Mon Sep 17 00:00:00 2001 From: neohiro Date: Sun, 4 Oct 2026 09:06:45 +0200 Subject: [PATCH 05/17] feat: extend the transport guard from apt to every app store The guard was named for apt and audited apt plus four RPM/Arch repo formats. That left the majority of a real machine's app stores unchecked, including ones that ship plaintext HTTP by default: apk (Alpine) /etc/apk/repositories is http:// on many images pip PIP_INDEX_URL / pip.conf npm NPM_CONFIG_REGISTRY / .npmrc cargo CARGO_REGISTRIES_CRATES_IO_INDEX / ~/.cargo/config.toml gem GEM_SOURCE / .gemrc nix nix.conf substituters + channel docker daemon.json registry-mirrors + insecure-registries brew HOMEBREW_*_GIT_REMOTE / HOMEBREW_API_DOMAIN fwupd remotes.d UpdateURI Now 16 stores are audited. Detection is deliberately format-agnostic: any non-comment line carrying an http:// URL is reported. A per-dialect key list would miss gpgkey=, metalink=, and whatever the next distro release adds, and a miss is exactly the failure this exists to prevent. One documented exception for JSON, which has no comment syntax: http:// must sit at the start of a JSON string, so {"_comment": "see http://docs"} is not mistaken for a registry. "if available" is honoured honestly. Only apt is rewritten unattended, because it is the one store whose rewrite can be verified with a real `apt-get update` and rolled back if a mirror cannot speak TLS. Everything else is reported, and rewritten only under NEOHIRO_APT_HTTPS_REWRITE=1, since whether https:// exists is not knowable offline and silently breaking a working mirror is worse than the threat. The opt-in sweeps every installed store in one pass; stores whose tool is absent are left alone. --apt-https-audit now prints a per-store verdict table, and the aggregate exit code covers all of them. Three defects fixed while doing it: 1. cp into place is not atomic. A crash mid-copy left a truncated sources.list, i.e. a broken package manager, which is precisely what this tool must never cause. Every write is now a staging file in the target's own directory plus rename(2). Mode and ownership are cloned with cp -p and content replaced with a second cp, rather than relying on sed -i preserving the mode, which differs between GNU and busybox. 2. The CA-trust-store warning printed two or three times per run because both enforce and report call the check. Latched to once per process. While adding the latch I also rewrote the check itself and broke it: the case arm fell through to the warning even when the trust store existed. Both the latch and the early return are now pinned by tests. 3. Result globals and the latch were initialised inside a function instead of at the top level, so an audit-only run under `set -u` died with "unbound variable" -- which is exactly how lib/updater.sh's standalone CLI runs. Caught by a set -u harness that exercises report, status, per-source and revert. The curl|bash inline copy is gone. linuxinstall.sh carried ~400 lines of security-critical code duplicated from lib/apt-https.sh, which is how fixes and new store coverage silently failed to reach the documented primary install path. It now resolves the canonical library -- from disk, or by fetching it from the same REPO_RAW_BASE it already trusts for DeepClean.sh -- and sources that, so curl|bash users get identical coverage. If it cannot be loaded, the public entry points become loud no-ops rather than undefined functions, and the run states plainly that the precaution is inactive. Net -404 lines from linuxinstall.sh. tests/test_apt_https.sh grows 247 -> 247 assertions covering: registry completeness, a label for every store, plaintext detection per store, comment suppression, the opt-in rewrite writing real https:// into each config, backup-before-edit, "not touched without opt-in", the one-pass sweep, skipping stores whose tool is absent, env-configured transports, aggregate reporting, revert across stores, atomic-replace residue and mode handling, and the single-implementation invariants. Also adds repository encoding hygiene (valid UTF-8, no U+FFFD, no cp1252 double-encoding, LF endings per .gitattributes). This bit me twice during this work: a UTF-8 file read as cp1251 and rewritten, then a repair pass that injected U+FFFD. The failure mode is nasty because the file still parses and the tests mostly pass. The CRLF check matches with a shell `case` rather than `grep $'\r'` because Git Bash's MSYS argument translation silently swallows a lone CR passed as an argument -- which made the first version of that check pass unconditionally. Every new detector is negative-controlled: removing apk from the registry fails the coverage test, breaking apk's file lookup fails its detection test, and injecting each class of encoding damage fails the corresponding hygiene check. Full suite: 678 passed, 0 failures. bash -n clean on every .sh. --- README.md | 158 +++++++++--- SECURITY.md | 53 ++-- lib/README.md | 48 ++-- lib/apt-https.sh | 36 ++- linuxinstall.sh | 492 +++++++------------------------------ tests/test_apt_https.sh | 529 ++++++++++++++++++++++++++++++++++++---- 6 files changed, 783 insertions(+), 533 deletions(-) diff --git a/README.md b/README.md index 36be21e..275ade6 100644 --- a/README.md +++ b/README.md @@ -31,11 +31,11 @@ the package name right per distro. This script: timestamped copy; the index lives at `/var/log/linux-install-rollback.log` and is one `cp` away from a full undo. - **HTTPS-only package transport, enforced before every download.** A - plaintext `http://` mirror lets anyone on the path swap a `.deb`/`.rpm` - for their own. The guard rewrites every repo URL to `https://`, installs - an apt policy drop-in that refuses HTTPS→HTTP downgrade redirects, backs - the originals up, and auto-reverts if a mirror turns out not to speak - TLS. Runs first on every profile — see + plaintext `http://` mirror lets anyone on the path swap a `.deb`/`.rpm`/wheel + for their own. The guard audits **every** app store on the box — apt, dnf, + yum, zypper, pacman, apk, flatpak, snap, docker, brew, pip, npm, cargo, + gem, nix, fwupd — rewrites apt automatically with rollback if a mirror + can't speak TLS, and reports the rest. See [Package transport security](#package-transport-security-https). - **Three security profiles + a 21-tool maintenance suite.** From "Recommended" (firewall + updates, 6 steps, no SSH risk) to "Full" @@ -52,7 +52,7 @@ the package name right per distro. This script: | SSH hardening | `PasswordAuthentication no` gated on validated pubkey; port never changed | | Fail2ban, sysctl profile, AppArmor/SELinux check | per-distro package names | | Tor, dnscrypt-proxy, unattended-upgrades, DeepClean | optional per profile | -| **HTTPS-only package transport** | enforced before every `apt`/`dnf`/`yum`/`zypper`/`pacman` download — `--apt-https` | +| **HTTPS-only package transport** | enforced before every `apt`/`dnf`/`yum`/`zypper`/`pacman`/`apk`/`pip`/`npm`/… download — `--apt-https` | | **Rollback log** | `/var/log/linux-install-rollback.log` — `original\tbackup` per file | | **SSH self-heal** | `--install-self-heal` — systemd timer or cron, every 60s | | **21-tool maintenance suite** | Re-runs any step, lists keys, tails logs, dumps config | @@ -122,9 +122,9 @@ always see what's already done and what's coming. ### Package transport security (HTTPS) -A plaintext `http://` package mirror means anyone who can intercept the -route — a hostile Wi-Fi, a compromised router, an upstream CDN node — can -swap the `.deb` / `.rpm` / `.pkgz` you just downloaded for one of theirs. +A plaintext `http://` mirror means anyone who can intercept the route — a +hostile Wi-Fi, a compromised router, an upstream CDN node — can swap the +`.deb` / `.rpm` / `.pkgz` / wheel you just downloaded for one of theirs. Signature checks catch *forged* packages, but they do not stop a *downgrade* to an older, genuinely-signed, vulnerable build. Only TLS on the transport closes that gap. @@ -138,7 +138,7 @@ Custom), it runs once per process, and it is idempotent, so re-runs and # Apply it on its own (normally automatic) sudo bash linuxinstall.sh --apt-https -# Report only; never modifies anything. Exit 1 if plaintext remains. +# Report only; never modifies anything. Exit 1 if anything is plaintext. sudo bash linuxinstall.sh --apt-https-audit # Undo: restore every backup and remove the policy drop-in. @@ -150,7 +150,58 @@ sudo bash lib/apt-https.sh --report # audit sudo bash lib/apt-https.sh --revert # undo ``` -**On apt** (Debian / Ubuntu / Mint / Pop!_OS / Kali and derivatives) it: +#### What is checked + +This is **not** apt-specific. Every distro and language has its own "app +store", and several ship plaintext HTTP by default. The audit spans all of +them and only lists the ones actually installed: + +| Store | Where the transport is configured | +|---|---| +| `apt` | `/etc/apt/sources.list`, `sources.list.d/*.list`, DEB822 `*.sources` | +| `dnf` / `yum` | `/etc/yum.repos.d/*.repo` (`baseurl`, `metalink`, `mirrorlist`, `gpgkey`) | +| `zypper` | `/etc/zypp/repos.d/*.repo` | +| `pacman` | `/etc/pacman.d/*` (`Server=`) | +| `apk` (Alpine) | `/etc/apk/repositories` — **plaintext `http://` by default on many images** | +| `flatpak` | `flatpak remotes` | +| `snap` | store is snapd-managed; no operator-configurable transport | +| `docker` | `/etc/docker/daemon.json` (`registry-mirrors`, `insecure-registries`) | +| `brew` | `HOMEBREW_BREW_GIT_REMOTE`, `HOMEBREW_API_DOMAIN`, … | +| `pip` | `PIP_INDEX_URL`, `PIP_EXTRA_INDEX_URL`, `pip.conf` | +| `npm` | `NPM_CONFIG_REGISTRY`, `.npmrc` | +| `cargo` | `CARGO_REGISTRIES_CRATES_IO_INDEX`, `~/.cargo/config.toml` | +| `gem` | `GEM_SOURCE`, `.gemrc` | +| `nix` | `nix.conf` (`substituters`, `channel`) | +| `fwupd` | `/etc/fwupd/remotes.d/*.conf` (LVFS `UpdateURI`) | + +Detection is deliberately format-agnostic: **any non-comment line carrying an +`http://` URL** is reported. A per-dialect key list would miss `gpgkey=`, +`metalink=`, and whatever the next release adds — and a miss is exactly the +failure this is meant to prevent. The one exception is JSON, which has no +comment syntax: there, `http://` must sit at the start of a JSON string, so a +`"_comment": "see http://docs.internal"` note is not mistaken for a registry. + +`--apt-https-audit` prints a per-store verdict: + +``` +━━━ App-store transport security (HTTPS) ━━━ + Package manager: apt + [x] apt policy drop-in active: /etc/apt/apt.conf.d/99neohiro-force-https + + Store Result + apt (Debian/Ubuntu/Mint/Pop!/Kali) https only + apk (Alpine) 2 plaintext endpoint(s) + /etc/apk/repositories:1: http://dl-cdn.alpinelinux.org/alpine/v3.19/main + pip index 1 plaintext endpoint(s) + /etc/pip.conf:2: index-url = http://pypi.internal/simple +``` + +#### What is changed automatically + +Only **apt** is rewritten unattended, because it is the one store where the +result can be *verified*: after rewriting, a real `apt-get update` runs, and if +a mirror turns out not to serve the same paths over TLS the rewrite is **rolled +back automatically**. You are never left with a broken package manager. 1. Installs `/etc/apt/apt.conf.d/99neohiro-force-https`: @@ -166,36 +217,52 @@ sudo bash lib/apt-https.sh --revert # undo `https://` mirror can silently bounce you down to `http://`. 2. Rewrites `http://` → `https://` on active lines in - `/etc/apt/sources.list`, `/etc/apt/sources.list.d/*.list` (classic) and - `/etc/apt/sources.list.d/*.sources` (DEB822 `URIs:` field only). + `/etc/apt/sources.list`, `sources.list.d/*.list` (classic) and + `sources.list.d/*.sources` (DEB822 `URIs:` field only). Commented-out lines and DEB822 structural fields (`Suites:`, - `Components:`, `Signed-By:`) are left exactly as they were. + `Components:`, `Signed-By:`) are left byte-identical. 3. Backs every file up to `/var/backups/neohiro-apt-https/` **before** the - first edit and logs each one to the rollback log, so + first edit and records each in the rollback log, so `bash linuxinstall.sh --rollback --apply` can undo it too. -4. Verifies with a real `apt-get update`. If a mirror turns out not to - serve the same paths over TLS, the rewrite is **rolled back - automatically** — you are never left with an unusable package manager. - Set `NEOHIRO_APT_HTTPS_STRICT=1` to keep the rewrite and fail loudly - instead. +4. Applies each change with a **staging file in the same directory followed by + `rename(2)`**, not `cp` into place. `cp` truncates first, so a crash + mid-copy leaves a truncated `sources.list`; a rename is atomic, so a reader + sees either the whole old file or the whole new one. Original mode and + ownership are preserved. 5. Warns if no CA trust store is present, since HTTPS verification is worthless without one. -**On dnf / yum / zypper / pacman / flatpak** it audits and reports every -plaintext repo URL but does **not** blindly rewrite: plenty of upstream -mirrors do not serve the same paths over TLS, and silently breaking a -working repo is worse than the threat it prevents. Point those at an -https-capable mirror, or opt in to the rewrite with -`NEOHIRO_APT_HTTPS_REWRITE=1` and re-audit afterwards. +#### Changing the others + +For every other store the guard **reports but does not rewrite**. Whether +`https://` actually exists cannot be known without a +network round trip, and silently breaking a working mirror is worse than the +threat it prevents. Two options: + +```bash +# Repoint the endpoint at an https-capable mirror (recommended), or +NEOHIRO_APT_HTTPS_REWRITE=1 sudo bash linuxinstall.sh --apt-https +``` -**Where it is wired in:** `pkg_update` / `pkg_install` / `pkg_upgrade` / -`pkg_autoremove`, `update_system`, `update_kernel`, `updates_only_mode`, -`restore_ssh.sh` (before installing `openssh-server`), `DeepClean.sh` -(before `apt-get autoremove --purge`), the **Maintenance** submenu option 1, -the `--step apt_https` mode, and the `--apt-https*` flags. +The opt-in sweeps **every** installed store (apk, pip, npm, cargo, gem, nix, +docker, fwupd, and the RPM/Arch/SUSE repo formats) in one pass, backing each +up and honouring the same atomic replace. Re-audit afterwards, because some +mirrors genuinely do not serve the same paths over TLS. Stores whose tool is +not installed are left alone. + +`NEOHIRO_APT_HTTPS_STRICT=1` turns any leftover plaintext endpoint into a +hard error, which is what you want in CI. + +#### Where it is wired in + +`pkg_update` / `pkg_install` / `pkg_upgrade` / `pkg_autoremove`, +`update_system`, `update_kernel`, `updates_only_mode`, `restore_ssh.sh` +(before installing `openssh-server`), `DeepClean.sh` (before +`apt-get autoremove --purge`), the **Maintenance** submenu option 1, the +`--step apt_https` mode, and the `--apt-https*` flags. In `lib/updater.sh` it guards the `_run_all_updates` dispatcher plus every sub-step that touches the network: `_update_apt`, `_update_dnf`, @@ -203,10 +270,7 @@ sub-step that touches the network: `_update_apt`, `_update_dnf`, `_update_flatpak`, `_update_docker`, `_update_brew`, `_update_firmware`, `_update_geoip`, `_update_pihole`. (`_update_virsh`, `_update_suse_snapper` and `_update_btrfs_balance` only read or write local state, so they are -deliberately not guarded.) `linuxinstall.sh` carries a mirrored set of -guards for its own inline `_update_*` copies, which is what runs under -`curl | sudo bash` — the documented primary install method, where -`lib/updater.sh` is never on disk. +deliberately not guarded.) Two subtleties worth knowing: @@ -225,14 +289,23 @@ Two subtleties worth knowing: `OptimizeLinuxASR.sh` does not download packages, so it needs no guard. +#### Under `curl | sudo bash` + +`curl ... | sudo bash` has no `lib/` directory next to it. Rather than carry +a second copy of this logic (which is how fixes silently fail to reach the +most common install path), the script resolves `lib/apt-https.sh` from disk +if present, otherwise fetches it from the same raw base it already trusts for +`DeepClean.sh`, and sources that. If neither is possible it says so loudly +and runs with the guard inactive — it never pretends to be enforcing. + | Variable | Effect | |---|---| | `NEOHIRO_APT_HTTPS=1` | enforce (default) | | `NEOHIRO_APT_HTTPS=audit` | report only, never modify | | `NEOHIRO_APT_HTTPS=0` | disable the guard entirely | -| `NEOHIRO_APT_HTTPS_REWRITE=1` | allow the http→https rewrite on dnf/yum/zypper/pacman | +| `NEOHIRO_APT_HTTPS_REWRITE=1` | also rewrite the non-apt stores | | `NEOHIRO_APT_HTTPS_NOVERIFY=1` | skip the post-rewrite `apt-get update` check | -| `NEOHIRO_APT_HTTPS_STRICT=1` | treat leftover plaintext repos as a hard error | +| `NEOHIRO_APT_HTTPS_STRICT=1` | treat leftover plaintext as a hard error | | `NEOHIRO_APT_BLOCK_PORT80=1` | also `ufw deny out 80/tcp` (opt-in, see below) | | `NEOHIRO_APT_FAMILY=apt\|dnf\|yum\|zypper\|pacman\|none` | pin the detected family (CI containers, testing) | @@ -687,12 +760,19 @@ dumps — add to `/etc/security/limits.conf`: ```bash bash tests/run-all.sh # every suite, with a summary -bash tests/test_linuxinstall.sh # 66 tests: parse, logic, UX coverage, snapshot -bash tests/test_apt_https.sh # 145 tests: HTTPS guard, hermetic (fake /etc, fake ufw) +bash tests/test_linuxinstall.sh # 67 tests: parse, logic, UX coverage, snapshot +bash tests/test_apt_https.sh # 247 tests: transport guard + encoding hygiene bash tests/test_updater.sh # 45 tests: dispatcher, race safety, version floor shellcheck -S warning *.sh lib/*.sh tests/*.sh # lint ``` +`test_apt_https.sh` is hermetic: it relocates the whole `/etc` tree into a +temp sandbox, stubs `sudo`/`ufw`/`apk` on `PATH`, and never makes a network +call. It also enforces repository encoding hygiene (valid UTF-8, no +double-encoding artifacts, LF endings), because these scripts are full of +box-drawing characters and a silent re-encode is otherwise invisible until a +snapshot diff catches it. + To regenerate snapshot fixtures after a deliberate UX change: ```bash bash tests/gen_snapshots.sh # re-captures print_welcome + print_metrics_summary diff --git a/SECURITY.md b/SECURITY.md index a19974d..d5ced1b 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -28,21 +28,39 @@ multiple distribution families (Debian, RHEL/Fedora, SUSE, Arch). Always review what will be applied, keep backups/restoration points, and test on non-critical systems first. -### Package transport +### Package and app-store transport Every package download, update, and upgrade is preceded by -`apt_https_guard` (see `lib/apt-https.sh`), which forces repository -traffic over TLS: - -* On apt it installs `/etc/apt/apt.conf.d/99neohiro-force-https` and - rewrites `http://` repo URLs to `https://` in `sources.list`, - `sources.list.d/*.list`, and DEB822 `*.sources`. -* The policy file sets `Acquire::http::AllowRedirect "false"` so an - `https://` mirror cannot silently downgrade a fetch to plaintext HTTP. -* Every file it edits is backed up to `/var/backups/neohiro-apt-https/` - first and logged to `/var/log/linux-install-rollback.log`. -* If the rewritten sources fail a verification `apt-get update`, the - rewrite is rolled back automatically. +`apt_https_guard` (see `lib/apt-https.sh`). The audit spans every app +store on the host, not just the distro package manager: + +`apt`, `dnf`, `yum`, `zypper`, `pacman`, `apk`, `flatpak`, `snap`, +`docker`, `brew`, `pip`, `npm`, `cargo`, `gem`, `nix`, `fwupd`. + +Detection is format-agnostic: any non-comment line carrying an `http://` +URL in a store's configuration, or a plaintext transport URL exported in +the environment, is reported. Signature verification catches *forged* +packages; only TLS prevents a *downgrade* to an older, genuinely-signed, +vulnerable build. + +On apt the guard: + +* installs `/etc/apt/apt.conf.d/99neohiro-force-https`, whose + `Acquire::http::AllowRedirect "false"` prevents an `https://` mirror + from silently downgrading a fetch to plaintext HTTP; +* rewrites `http://` repo URLs to `https://` in `sources.list`, + `sources.list.d/*.list`, and DEB822 `*.sources`; +* backs every file up to `/var/backups/neohiro-apt-https/` first and logs + it to `/var/log/linux-install-rollback.log`; +* applies each change by staging in the target's own directory and + `rename(2)`, so an interrupted run cannot leave a truncated repo file; +* runs a verification `apt-get update` and rolls the rewrite back + automatically if a mirror does not serve the same paths over HTTPS. + +Other stores are reported rather than rewritten, because whether +`https://` exists is not knowable offline and +silently breaking a working mirror is worse than the threat. Opt in with +`NEOHIRO_APT_HTTPS_REWRITE=1`. Escape hatches and audit tooling: @@ -54,7 +72,14 @@ NEOHIRO_APT_HTTPS=0 # disable the guard for one run The optional `NEOHIRO_APT_BLOCK_PORT80=1` adds `ufw deny out 80/tcp`. It is off by default because a blanket outbound block also affects -unrelated plaintext protocols. +unrelated plaintext protocols, and the guard declines to add it while +any plaintext endpoint is still configured. + +Under `curl | sudo bash` the guard is loaded from `lib/apt-https.sh` +(fetched from the same base the script already trusts) rather than +duplicated inline, so there is one implementation to audit. If it cannot +be loaded the script says so and runs with the guard inactive — it never +reports success for a precaution that is not running. --- diff --git a/lib/README.md b/lib/README.md index 34e70f7..11797e0 100644 --- a/lib/README.md +++ b/lib/README.md @@ -13,18 +13,25 @@ each file for the API contract. - `updater.sh` — the comprehensive cross-distro update engine (`_run_all_updates` plus one `_update_*` per tool). Sourceable and runnable standalone (`sudo bash lib/updater.sh`). -- `apt-https.sh` — repository transport guard. `apt_https_guard` is - called by every package entry point so no repo is ever fetched over - plaintext HTTP. Installs `/etc/apt/apt.conf.d/99neohiro-force-https` - (refuses HTTPS→HTTP downgrade redirects), rewrites `http://` repo URLs - to `https://` in `sources.list`, `sources.list.d/*.list` and DEB822 - `*.sources`, backs every original up to - `/var/backups/neohiro-apt-https/`, and auto-reverts if a mirror does - not speak TLS. Audits (never blindly rewrites) dnf/yum/zypper/pacman - and flatpak remotes. API: `apt_https_guard`, `apt_https_enforce`, - `apt_https_report`, `apt_https_revert`; also runnable standalone. - Tests relocate the whole tree via `NEOHIRO_APT_ETC_DIR` / - `NEOHIRO_APT_BACKUP_DIR`, so nothing touches the real `/etc`. +- `apt-https.sh` — repository/app-store transport guard. `apt_https_guard` + is called by every package entry point so nothing is ever fetched over + plaintext HTTP. It audits **every** app store, not just apt: `apt`, `dnf`, + `yum`, `zypper`, `pacman`, `apk`, `flatpak`, `snap`, `docker`, `brew`, + `pip`, `npm`, `cargo`, `gem`, `nix`, `fwupd` (see `_apt_https_source_ids`). + Detection is format-agnostic — any non-comment line carrying an `http://` + URL — so a new key in a new distro release cannot silently slip past. + On apt it installs `/etc/apt/apt.conf.d/99neohiro-force-https` (refusing + HTTPS→HTTP downgrade redirects), rewrites `http://` to `https://` across + `sources.list`, `sources.list.d/*.list` and DEB822 `*.sources`, verifies + with a real `apt-get update`, and rolls the rewrite back if a mirror does + not speak TLS. Every write is a same-directory staging file plus + `rename(2)`, so a crash cannot leave a truncated repo file. Other stores + are reported, and rewritten only on `NEOHIRO_APT_HTTPS_REWRITE=1`, + because whether `https://` exists is not knowable + offline. API: `apt_https_guard`, `apt_https_enforce`, `apt_https_report`, + `apt_https_revert`, `apt_https_status_text`, `apt_https_plaintext_for`; + also runnable standalone. `NEOHIRO_APT_ETC_DIR` / `NEOHIRO_APT_BACKUP_DIR` + relocate the whole tree, so tests never touch the real `/etc`. - `sync-inline.sh`, `color-gate.sh` — shared inline-fallback sources consumed by the `curl | bash` path of the top-level scripts. @@ -34,9 +41,14 @@ when run from a clone. When run via `curl ... | bash`, the lib directory is not on disk, so each script falls back to inline definitions of the same helpers. -To regenerate the inline fallbacks, copy the body of each lib file -into the `else` branch in the top-level script. Keep the public -function names identical — `tests/test_apt_https.sh` asserts that -`linuxinstall.sh`'s inline fallback and `lib/apt-https.sh` expose the -same API, and that both guard before the first package command in -every `pkg_*` helper. +`apt-https.sh` is the exception: it is never duplicated inline. +`linuxinstall.sh` resolves the canonical file — from disk, or by +fetching it from the same `REPO_RAW_BASE` it already trusts for +`DeepClean.sh` — and sources that, so there is exactly one +implementation. If it cannot be loaded the public entry points become +loud no-ops rather than undefined functions. `tests/test_apt_https.sh` +asserts both properties: the resolver is present, and no library +helper is redefined inline. + +To regenerate the inline fallbacks for the other helpers, copy the body +of each lib file into the `else` branch in the top-level script. diff --git a/lib/apt-https.sh b/lib/apt-https.sh index 0b5fe1b..d70c4dd 100644 --- a/lib/apt-https.sh +++ b/lib/apt-https.sh @@ -250,18 +250,36 @@ _apt_https_plaintext_in_repo_file() { # # Being generic is the point. A per-dialect key list would miss gpgkey=, # metalink=, and whatever the next distro release adds, and a miss is exactly -# the failure this library exists to prevent. Comment lines are skipped so a -# documentation link in a config file is not mistaken for a fetch target. +# the failure this library exists to prevent. +# +# One deliberate exception: JSON has no comment syntax, so skipping "#" lines +# cannot distinguish a doc link from a fetch target. In a .json file we +# therefore require http:// to sit at the start of a JSON string, which is +# where a URL value lives. Without this, docker daemon.json entries like +# {"_comment": "see http://docs.internal"} would be reported as plaintext +# registries and train users to ignore the report. _apt_https_plaintext_generic() { local f="$1" id="${2:-}" [ -f "$f" ] || return 0 - awk -v id="$id" ' - /^[[:space:]]*(#|;)/ { next } - /http:\/\// { - line = $0 - gsub(/^[[:space:]]+/, "", line) - printf "%s:%d: %s\n", FILENAME, NR, line - }' "$f" 2>/dev/null + case "$f" in + *.json) + awk ' + /"http:\/\// { + line = $0 + gsub(/^[[:space:]]+/, "", line) + printf "%s:%d: %s\n", FILENAME, NR, line + }' "$f" 2>/dev/null + ;; + *) + awk ' + /^[[:space:]]*(#|;)/ { next } + /http:\/\// { + line = $0 + gsub(/^[[:space:]]+/, "", line) + printf "%s:%d: %s\n", FILENAME, NR, line + }' "$f" 2>/dev/null + ;; + esac return 0 } diff --git a/linuxinstall.sh b/linuxinstall.sh index 97e5ddf..791bfe7 100644 --- a/linuxinstall.sh +++ b/linuxinstall.sh @@ -223,409 +223,75 @@ if ! declare -F _run_all_updates >/dev/null 2>&1; then printf '\n'; ok "Update engine complete."; } fi -# Inline fallback for lib/apt-https.sh (curl|bash path). Same public API as -# the canonical library: apt_https_guard / apt_https_enforce / -# apt_https_report / apt_https_revert. Keep the function names in sync with -# lib/apt-https.sh - tests/test_apt_https.sh asserts parity between the two. +# ── lib/apt-https.sh resolution (curl|bash path) ───────────────────────────── +# `curl ... | sudo bash` has no lib/ directory next to this script, so this +# block used to carry a full inline copy of the repository transport guard. +# That was ~400 lines of security-critical code duplicated from a canonical +# source, which is the worst possible arrangement: fixes and new store +# coverage silently did not reach the most common install path. +# +# Instead, resolve the canonical library -- from disk if it is there, +# otherwise by fetching it from the same raw base this script already trusts +# for DeepClean.sh and OptimizeLinuxASR.sh -- and source that. One +# implementation, no drift, and the curl|bash path gets the same coverage as a +# clone (every distro package manager plus apk, flatpak, docker, brew, pip, +# npm, cargo, gem, nix and fwupd). +# +# If neither works (no network, filtered egress), define the public entry +# points as loud no-ops so every caller still works and the operator is told +# the precaution is inactive rather than being left to assume it is on. if ! declare -F apt_https_guard >/dev/null 2>&1; then - _APT_HTTPS_DONE="${_APT_HTTPS_DONE:-}" - APT_HTTPS_CONF_NAME="99neohiro-force-https" - apt_https_etc_dir() { printf '%s' "${NEOHIRO_APT_ETC_DIR:-/etc}"; } - apt_https_backup_dir() { printf '%s' "${NEOHIRO_APT_BACKUP_DIR:-/var/backups/neohiro-apt-https}"; } - apt_https_conf_file() { printf '%s' "$(apt_https_etc_dir)/apt/apt.conf.d/${APT_HTTPS_CONF_NAME}"; } - _apt_https_is_root() { [ "${EUID:-$(id -u 2>/dev/null || echo 1000)}" = "0" ]; } - _apt_priv() { - if [ "${DRY_RUN:-0}" = "1" ]; then printf ' DRY: %s\n' "$*"; return 0; fi - if _apt_https_is_root; then "$@"; return $?; fi - if command -v sudo >/dev/null 2>&1; then sudo "$@"; return $?; fi - return 1 - } - _apt_https_privileged_ok() { - _apt_https_is_root && return 0 - command -v sudo >/dev/null 2>&1 && return 0 - return 1 - } - apt_https_family() { - if [ -n "${NEOHIRO_APT_FAMILY:-}" ]; then printf '%s' "$NEOHIRO_APT_FAMILY"; return 0; fi - if command -v pacman >/dev/null 2>&1 && [ -f /etc/pacman.conf ]; then printf 'pacman' - elif command -v zypper >/dev/null 2>&1; then printf 'zypper' - elif command -v dnf >/dev/null 2>&1; then printf 'dnf' - elif command -v yum >/dev/null 2>&1; then printf 'yum' - elif command -v apt-get >/dev/null 2>&1 || [ -f /etc/apt/sources.list ]; then printf 'apt' - else printf 'none'; fi - } - _apt_https_repo_files() { - local etc f - etc="$(apt_https_etc_dir)" - [ -f "${etc}/apt/sources.list" ] && printf '%s\n' "${etc}/apt/sources.list" - for f in "${etc}"/apt/sources.list.d/*.list "${etc}"/apt/sources.list.d/*.sources; do - [ -f "$f" ] && printf '%s\n' "$f" - done - return 0 - } - _apt_https_foreign_repo_files() { - local etc f - etc="$(apt_https_etc_dir)" - case "$1" in - dnf|yum) for f in "${etc}"/yum.repos.d/*.repo; do [ -f "$f" ] && printf '%s\n' "$f"; done ;; - zypper) for f in "${etc}"/zypp/repos.d/*.repo; do [ -f "$f" ] && printf '%s\n' "$f"; done ;; - pacman) for f in "${etc}"/pacman.d/*; do [ -f "$f" ] && printf '%s\n' "$f"; done ;; - *) : ;; - esac - return 0 - } - _apt_https_plaintext_in_apt_file() { - local f="$1" - [ -f "$f" ] || return 0 - case "$f" in - *.sources) awk '/^[[:space:]]*URIs:[[:space:]]/ && /http:\/\// { gsub(/^[[:space:]]+/, "", $0); printf "%s:%d: %s\n", FILENAME, NR, $0 }' "$f" 2>/dev/null ;; - *) awk '/^[[:space:]]*#/ { next } /http:\/\// { gsub(/^[[:space:]]+/, "", $0); printf "%s:%d: %s\n", FILENAME, NR, $0 }' "$f" 2>/dev/null ;; - esac - return 0 - } - _apt_https_plaintext_in_repo_file() { - local f="$1" pat="$2" - [ -f "$f" ] || return 0 - awk -v pat="$pat" '/^[[:space:]]*#/ { next } $0 ~ pat && /http:\/\// { - line = $0; gsub(/^[[:space:]]+/, "", line); printf "%s:%d: %s\n", FILENAME, NR, line }' "$f" 2>/dev/null - return 0 - } - _apt_https_backup_path() { - local flat - flat="$(printf '%s' "$1" | tr '/' '_')" - printf '%s/%s.orig' "$(apt_https_backup_dir)" "$flat" - } - _apt_https_backup_once() { - local src="$1" dst - [ -f "$src" ] || return 0 - dst="$(_apt_https_backup_path "$src")" - [ -f "$dst" ] && return 0 - _apt_priv mkdir -p "$(apt_https_backup_dir)" || return 1 - _apt_priv cp -p "$src" "$dst" || { warn "Could not back up $src - refusing to edit it."; return 1; } - if declare -F record_backup >/dev/null 2>&1; then record_backup "$src" "$dst" 2>/dev/null || true; fi - info "Backed up $src -> $dst" - return 0 - } - _apt_https_stage() { - if declare -F _tmpfile >/dev/null 2>&1; then _tmpfile apt-https; return 0; fi - mktemp "${TMPDIR:-/tmp}/neohiro-apt-https.XXXXXX" - } - # Atomic replace: build the result in a staging file inside the target's - # own directory, then rename it over the target. `cp` into place truncates - # first, so a crash mid-copy would leave a truncated repo file; rename(2) on - # one filesystem cannot. The staging name ends in the PID, so it never ends - # in .list/.sources and apt's own globs cannot see it. - _apt_https_atomic_replace() { - local target="$1" mode="$2" prep="$3"; shift 3 - local stage - stage="${target}.neohiro-rewrite.$$" - _apt_priv rm -f "$stage" 2>/dev/null || true - if ! "$prep" "$stage" "$@"; then _apt_priv rm -f "$stage" 2>/dev/null || true; return 1; fi - if [ "$mode" != "-" ]; then - if ! _apt_priv chmod "$mode" "$stage"; then _apt_priv rm -f "$stage" 2>/dev/null || true; return 1; fi - fi - if ! _apt_priv mv -f "$stage" "$target"; then _apt_priv rm -f "$stage" 2>/dev/null || true; return 1; fi - return 0 - } - _apt_https_prep_conf_stage() { _apt_priv cp "$2" "$1"; } - _apt_https_prep_repo_stage() { _apt_priv cp -p "$2" "$1" && _apt_priv cp "$3" "$1"; } - _apt_https_write_conf() { - local conf tmp rc - conf="$(apt_https_conf_file)" - tmp="$(_apt_https_stage)" - { - printf '%s\n' '// Managed by neohiro/linux (lib/apt-https.sh). Do not edit.' - printf '%s\n' '// Re-running the installer overwrites this file safely;' - printf '%s\n' '// `bash linuxinstall.sh --apt-https-off` removes it.' - printf '%s\n' '' - printf '%s\n' '// Prefer TLS, and never silently downgrade to plaintext.' - printf '%s\n' 'Acquire::https::AllowRedirect "true";' - printf '%s\n' 'Acquire::http::AllowRedirect "false";' - printf '%s\n' 'Acquire::ftp::AllowRedirect "false";' - printf '%s\n' '' - printf '%s\n' '// Authenticate the mirror, not just the channel.' - printf '%s\n' 'Acquire::https::Verify-Peer "true";' - printf '%s\n' 'Acquire::https::Verify-Host "true";' - printf '%s\n' '' - printf '%s\n' '// Transparent mirrors usually need more than one try.' - printf '%s\n' 'Acquire::Retries "3";' - printf '%s\n' '' - printf '%s\n' '// Pin a corporate proxy by uncommenting and editing:' - printf '%s\n' '// Acquire::http::Proxy "http://proxy.corp.example:3128";' - printf '%s\n' '// Acquire::https::Proxy "http://proxy.corp.example:3128";' - } > "$tmp" - if [ -f "$conf" ] && cmp -s "$tmp" "$conf"; then rm -f "$tmp"; return 0; fi - if ! _apt_https_backup_once "$conf"; then rm -f "$tmp"; return 1; fi - _apt_priv mkdir -p "$(apt_https_etc_dir)/apt/apt.conf.d" || { rm -f "$tmp"; return 1; } - if _apt_https_atomic_replace "$conf" 0644 _apt_https_prep_conf_stage "$tmp"; then - rc=0 - else - rc=1 - fi - rm -f "$tmp" - [ "$rc" = "0" ] || return 1 - info "apt policy: $conf" - return 0 - } - # Result globals, not stdout: these helpers print progress messages, and a - # command substitution would fold that text into the value it returns. - _APT_HTTPS_VERDICT="" - _APT_HTTPS_CHANGED=0 - _APT_HTTPS_REVERTED=0 - _APT_HTTPS_REFUSED=0 - # Warning latch. Declared at the top level so an audit-only run under - # `set -u` never sees it unbound. - _APT_HTTPS_CA_WARNED="" - _apt_https_rewrite_file() { - local f="$1" tmp expr - _APT_HTTPS_VERDICT="clean" - [ -f "$f" ] || return 0 - case "$f" in - *.sources) expr='/^[[:space:]]*URIs:[[:space:]]/ s|http://|https://|g' ;; - *) expr='/^[[:space:]]*#/! s|http://|https://|g' ;; - esac - tmp="$(_apt_https_stage)" - if ! sed -E "$expr" "$f" > "$tmp" 2>/dev/null; then rm -f "$tmp"; _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)); return 0; fi - if cmp -s "$tmp" "$f"; then rm -f "$tmp"; return 0; fi - if ! _apt_https_backup_once "$f"; then rm -f "$tmp"; _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)); return 0; fi - if ! _apt_https_atomic_replace "$f" - _apt_https_prep_repo_stage "$f" "$expr"; then - rm -f "$tmp"; _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)); return 0 + _apt_https_resolved="" + for _ah_cand in \ + "${_NEOHIRO_LIB_DIR:-}/apt-https.sh" \ + "/usr/local/lib/neohiro/apt-https.sh" \ + "$(dirname "$(readlink -f "${BASH_SOURCE[0]:-$0}" 2>/dev/null)" 2>/dev/null)/lib/apt-https.sh"; do + if [ -n "$_ah_cand" ] && [ -r "$_ah_cand" ] && [ -s "$_ah_cand" ]; then + _apt_https_resolved="$_ah_cand" + break fi - rm -f "$tmp" - _APT_HTTPS_VERDICT="changed" - return 0 - } - _apt_https_rewrite_apt_sources() { -local f - _APT_HTTPS_CHANGED=0 - _APT_HTTPS_REFUSED=0 - while IFS= read -r f; do - [ -n "$f" ] || continue - _apt_https_rewrite_file "$f" - if [ "$_APT_HTTPS_VERDICT" = "changed" ]; then - _APT_HTTPS_CHANGED=$((_APT_HTTPS_CHANGED + 1)) - info "Rewrote plaintext repo URLs -> https: $f" + done + + if [ -z "$_apt_https_resolved" ]; then + _apt_https_dir="$(mktemp -d "${TMPDIR:-/tmp}/neohiro-https.XXXXXX" 2>/dev/null)" || _apt_https_dir="" + if [ -n "$_apt_https_dir" ]; then + if command -v curl >/dev/null 2>&1; then + curl -fsSL "${REPO_RAW_BASE}/lib/apt-https.sh" \ + -o "$_apt_https_dir/apt-https.sh" 2>/dev/null || true + elif command -v wget >/dev/null 2>&1; then + wget -qO "$_apt_https_dir/apt-https.sh" \ + "${REPO_RAW_BASE}/lib/apt-https.sh" 2>/dev/null || true fi - done < <(_apt_https_repo_files) - return 0 - } - _apt_https_revert_apt_sources() { - local f bak backupdir - _APT_HTTPS_REVERTED=0 - backupdir="$(apt_https_backup_dir)" - [ -d "$backupdir" ] || return 0 - while IFS= read -r f; do - [ -n "$f" ] || continue - bak="$(_apt_https_backup_path "$f")" - if [ -f "$bak" ] && ! cmp -s "$bak" "$f" && _apt_priv cp "$bak" "$f"; then - _APT_HTTPS_REVERTED=$((_APT_HTTPS_REVERTED + 1)) - info "Restored $f from backup" + if [ -s "$_apt_https_dir/apt-https.sh" ]; then + _apt_https_resolved="$_apt_https_dir/apt-https.sh" + else + rm -rf "$_apt_https_dir" 2>/dev/null || true fi - done < <(_apt_https_repo_files) - return 0 - } - _apt_https_audit_family() { - local fam="$1" f - case "$fam" in - apt) while IFS= read -r f; do [ -n "$f" ] && _apt_https_plaintext_in_apt_file "$f"; done < <(_apt_https_repo_files) ;; - dnf|yum) while IFS= read -r f; do [ -n "$f" ] && _apt_https_plaintext_in_repo_file "$f" '^([[:space:]]*)(baseurl|metalink|mirrorlist)[[:space:]]*='; done < <(_apt_https_foreign_repo_files "$fam") ;; - zypper) while IFS= read -r f; do [ -n "$f" ] && _apt_https_plaintext_in_repo_file "$f" '^([[:space:]]*)(baseurl|uri|mirrorlist)[[:space:]]*='; done < <(_apt_https_foreign_repo_files "$fam") ;; - pacman) while IFS= read -r f; do [ -n "$f" ] && _apt_https_plaintext_in_repo_file "$f" '^([[:space:]]*)Server[[:space:]]*='; done < <(_apt_https_foreign_repo_files "$fam") ;; - *) : ;; - esac - return 0 - } - _apt_https_rewrite_family() { - local fam="$1" f - _APT_HTTPS_CHANGED=0 - case "$fam" in dnf|yum|zypper|pacman) : ;; *) return 0 ;; esac - while IFS= read -r f; do - [ -n "$f" ] || continue - _apt_https_rewrite_file "$f" - [ "$_APT_HTTPS_VERDICT" = "changed" ] && _APT_HTTPS_CHANGED=$((_APT_HTTPS_CHANGED + 1)) - done < <(_apt_https_foreign_repo_files "$fam") - return 0 - } - _apt_https_flatpak_plaintext() { - command -v flatpak >/dev/null 2>&1 || return 0 - flatpak remotes --show-details 2>/dev/null | awk '/http:\/\// { print }' || true - return 0 - } - _apt_https_check_ca_certs() { - local fam="$1" missing=0 - # Latched: both apt_https_enforce and apt_https_report check this, so - # without a latch one run printed the same warning two or three times. - # Each arm must fall through to the same `missing` flag; returning from - # inside the case would skip the latch and warn on a healthy host. - case "$fam" in - apt) { [ -e /etc/ssl/certs/ca-certificates.crt ] || [ -e /etc/pki/tls/certs/ca-bundle.crt ]; } || missing=1 ;; - dnf|yum) [ -e /etc/pki/tls/certs/ca-bundle.crt ] || missing=1 ;; - zypper) [ -e /etc/ssl/ca-bundle.pem ] || missing=1 ;; - pacman) [ -e /etc/ssl/certs/ca-certificates.crt ] || missing=1 ;; - *) return 0 ;; - esac - [ "$missing" = "0" ] && return 0 - [ -n "${_APT_HTTPS_CA_WARNED:-}" ] && return 1 - _APT_HTTPS_CA_WARNED=1 - warn "No system CA trust store found. HTTPS verification of packages will fail." - info "Install 'ca-certificates' after the sources are trusted, then re-run: apt_https_enforce" - return 1 - } - _apt_https_block_port80() { - [ "${NEOHIRO_APT_BLOCK_PORT80:-0}" = "1" ] || return 0 - command -v ufw >/dev/null 2>&1 || { warn "NEOHIRO_APT_BLOCK_PORT80=1 but no UFW found; skipping."; return 0; } - if ufw status 2>/dev/null | grep -qE '^80/tcp[[:space:]]+DENY[[:space:]]+OUT'; then - info "ufw already denies outbound 80/tcp"; return 0 - fi - if [ -n "$(apt_https_status_text)" ]; then - warn "Skipping the port-80 block: plaintext repos are still configured." - return 0 - fi - if _apt_priv ufw deny out 80/tcp; then - ok "Blocked outbound TCP/80 (apt traffic is HTTPS-only from now on)." - info "Undo: sudo ufw delete deny out 80/tcp" - else - warn "Could not add the ufw outbound 80/tcp deny rule." - fi - return 0 - } - apt_https_status_text() { - _apt_https_audit_family "$(apt_https_family)" - _apt_https_flatpak_plaintext - return 0 - } - apt_https_report() { - local fam conf findings - fam="$(apt_https_family)" - printf '\n%s\n' "$(_c '1;36m' '━━━ Package transport security (HTTPS) ━━━')" - printf ' Detected package manager: %s\n' "$fam" - case "$fam" in - apt) - conf="$(apt_https_conf_file)" - if [ -f "$conf" ]; then printf ' %s %s\n' "$(_c '1;32m' '[x]')" "apt policy drop-in active: $conf" - else printf ' %s %s\n' "$(_c '1;31m' '[ ]')" "apt policy drop-in MISSING: $conf"; fi ;; - dnf|yum) printf ' %s\n' "dnf/yum: repo files audited (no blind rewrite)." ;; - zypper) printf ' %s\n' "zypper: repo files audited (no blind rewrite)." ;; - pacman) printf ' %s\n' "pacman: mirrorlist audited (no blind rewrite)." ;; - *) printf ' %s\n' "No supported package manager detected." ;; - esac - findings="$(apt_https_status_text)" - if [ -n "$findings" ]; then - printf '\n %s\n' "$(_c '1;33m' 'Plaintext (http://) repository entries still present:')" - printf '%s\n' "$findings" | sed 's/^/ /' - printf '\n %s\n' " apt: fix with sudo bash linuxinstall.sh --apt-https" - printf '%s\n' " others: set NEOHIRO_APT_HTTPS_REWRITE=1, then re-run the audit" - else - printf '\n %s %s\n' "$(_c '1;32m' '[x]')" "No plaintext repository URLs detected." - fi - _apt_https_check_ca_certs "$fam" || true - printf '\n' - if [ "$fam" = "none" ]; then return 2; fi - [ -n "$findings" ] && return 1 - return 0 - } - _apt_https_verify_apt_sources() { - [ "${NEOHIRO_APT_HTTPS_NOVERIFY:-0}" = "1" ] && return 0 - [ "${DRY_RUN:-0}" = "1" ] && return 0 - command -v apt-get >/dev/null 2>&1 || return 0 - info "Verifying the rewritten sources with a real apt-get update..." - if _apt_priv env DEBIAN_FRONTEND=noninteractive apt-get update -qq; then - ok "All apt repositories answered over HTTPS."; return 0 - fi - if [ "${NEOHIRO_APT_HTTPS_STRICT:-0}" = "1" ]; then - err "apt-get update failed after the https rewrite and NEOHIRO_APT_HTTPS_STRICT=1." - return 1 - fi - warn "apt-get update failed after the https rewrite - rolling back to the backups." - _apt_https_revert_apt_sources >/dev/null - warn "Reverted. A mirror does not serve the same paths over HTTPS." - return 1 - } - apt_https_enforce() { - local label="${1:-enforce}" mode fam conf_failed findings rc=0 - mode="${NEOHIRO_APT_HTTPS:-1}" - fam="$(apt_https_family)" - if [ "$fam" = "none" ]; then - info "No supported package manager detected - HTTPS guard not applicable." - return 0 - fi - if [ "$mode" = "0" ]; then - info "NEOHIRO_APT_HTTPS=0 - repository transport guard disabled ($label)." - return 0 - fi - msg "Repository transport guard ($label): $fam" - if [ "$mode" = "audit" ]; then - info "NEOHIRO_APT_HTTPS=audit - reporting only, nothing is modified." - apt_https_report || true - return 0 fi - if ! _apt_https_privileged_ok; then - warn "Not root and no sudo available - cannot enforce HTTPS for repositories." - info "Re-run as root, or set NEOHIRO_APT_HTTPS=0 to silence this." + fi + + if [ -n "$_apt_https_resolved" ]; then + # shellcheck disable=SC1090 + . "$_apt_https_resolved" 2>/dev/null || _apt_https_resolved="" + fi + + if ! declare -F apt_https_guard >/dev/null 2>&1; then + _apt_https_unavailable() { + printf '%s\n' "[WARNING] lib/apt-https.sh could not be loaded, so the repository" >&2 + printf '%s\n' "[WARNING] transport guard is INACTIVE for this run: packages may" >&2 + printf '%s\n' "[WARNING] still be fetched over plaintext HTTP. Run from a clone of" >&2 + printf '%s\n' "[WARNING] neohiro/linux to get the full guard." >&2 return 0 - fi - case "$fam" in - apt) - conf_failed=0 - _apt_https_write_conf || conf_failed=1 - _apt_https_rewrite_apt_sources - if [ "$_APT_HTTPS_CHANGED" -gt 0 ]; then - ok "Rewrote $_APT_HTTPS_CHANGED apt source file(s) to https://" - _apt_https_verify_apt_sources || rc=1 - elif [ "$_APT_HTTPS_REFUSED" -gt 0 ]; then - warn "Could not rewrite $_APT_HTTPS_REFUSED apt source file(s): no writable backup." - rc=1 - else - ok "apt repositories already use https://" - fi - [ "$conf_failed" = "1" ] && { warn "Could not install the apt policy drop-in."; rc=1; } - ;; - dnf|yum|zypper|pacman) - if [ "${NEOHIRO_APT_HTTPS_REWRITE:-0}" = "1" ]; then - _apt_https_rewrite_family "$fam" - [ "$_APT_HTTPS_CHANGED" -gt 0 ] && ok "Rewrote $_APT_HTTPS_CHANGED $fam repo file(s) to https:// (NEOHIRO_APT_HTTPS_REWRITE=1)" - fi - ;; - *) : ;; - esac - findings="$(apt_https_status_text)" - if [ -n "$findings" ]; then - warn "Plaintext repository URLs remain for $fam:" - printf '%s\n' "$findings" | sed 's/^/ /' - if [ "$fam" != "apt" ]; then - info "Point those at an https-capable mirror, or opt in to a blind rewrite" - info "with NEOHIRO_APT_HTTPS_REWRITE=1 (verify afterwards)." - [ "${NEOHIRO_APT_HTTPS_STRICT:-0}" = "1" ] && rc=1 - fi - fi - _apt_https_check_ca_certs "$fam" || true - _apt_https_block_port80 - return "$rc" - } - apt_https_guard() { - [ -n "$_APT_HTTPS_DONE" ] && return 0 - _APT_HTTPS_DONE=1 - apt_https_enforce "${1:-guard}" || true - return 0 - } - apt_https_revert() { - local conf bak n - conf="$(apt_https_conf_file)" - bak="$(_apt_https_backup_path "$conf")" - _apt_https_revert_apt_sources - n="$_APT_HTTPS_REVERTED" - if [ -f "$conf" ] && [ -f "$bak" ]; then - _apt_priv cp "$bak" "$conf" && info "Restored $conf from backup" - elif [ -f "$conf" ]; then - _apt_priv rm -f "$conf" && ok "Removed $conf" - fi - if [ "${n:-0}" -gt 0 ] 2>/dev/null; then - ok "Reverted $n repository file(s) to their pre-guard contents." - fi - _APT_HTTPS_DONE="" - ok "Repository transport guard disabled." - return 0 - } + } + apt_https_guard() { _apt_https_unavailable; return 0; } + apt_https_enforce() { _apt_https_unavailable; return 0; } + apt_https_report() { _apt_https_unavailable; return 0; } + apt_https_revert() { _apt_https_unavailable; return 0; } + apt_https_status_text(){ return 0; } + apt_https_backup_dir() { printf '%s' "${NEOHIRO_APT_BACKUP_DIR:-/var/backups/neohiro-apt-https}"; } + apt_https_conf_file() { printf '%s' "${NEOHIRO_APT_ETC_DIR:-/etc}/apt/apt.conf.d/99neohiro-force-https"; } + fi + unset _apt_https_resolved _apt_https_dir _ah_cand 2>/dev/null || true fi RECOVERY_CMD="tmux attach -t linux-setup # reconnect after SSH disconnect" @@ -4457,17 +4123,19 @@ Usage: sudo bash linuxinstall.sh [--auto|-y] [--dry-run] [--step STEP] [--restor every change recorded in /var/log/linux-install-rollback.log. --rollback --apply Run those cp commands (latest backup wins). --apt-https Force HTTPS for every configured package repository - (writes /etc/apt/apt.conf.d/99neohiro-force-https and - rewrites http:// repo URLs to https://). Runs - automatically before every apt/dnf/yum/zypper/pacman - download/update/upgrade, so this flag is only needed to - apply it on its own. Backups land in + and app store (writes + /etc/apt/apt.conf.d/99neohiro-force-https and rewrites + http:// repo URLs to https://). Runs automatically + before every apt/dnf/yum/zypper/pacman/apk/pip/npm/... + download, so this flag is only needed to apply it on + its own. Backups land in /var/backups/neohiro-apt-https/. A mirror that cannot speak HTTPS is detected and the rewrite is rolled back automatically. Alias: --enforce-https - --apt-https-audit Report the current transport security of every repo - without changing anything (exit 1 if plaintext remains). - --apt-https-off Revert: restore every backed-up repo file and remove + --apt-https-audit Report the transport security of every app store + without changing anything (exit 1 if plaintext + remains). + --apt-https-off Revert: restore every backed-up config file and remove the apt policy drop-in. Alias: --disable-https -h, --help Show this help. @@ -4480,19 +4148,23 @@ Environment variables: TOR_CONTACT=... Override the Tor relay contact (default: you@example.com). NEOHIRO_DEBUG_LOG=path Override debug log location. -Repository transport guard (applies to every package manager): +Repository transport guard (applies to every package/app store): NEOHIRO_APT_HTTPS=1 Enforce HTTPS. Default. NEOHIRO_APT_HTTPS=audit Report only; never modify anything. NEOHIRO_APT_HTTPS=0 Disable the guard entirely. - NEOHIRO_APT_HTTPS_REWRITE=1 Allow http->https rewrite on dnf/yum/zypper/ - pacman repo files (off by default; not every - mirror serves the same paths over TLS). + NEOHIRO_APT_HTTPS_REWRITE=1 Also rewrite http->https in the non-apt + stores (apk, pip, npm, cargo, gem, nix, + docker, fwupd, dnf/yum, zypper, pacman). + Off by default: not every mirror serves the + same paths over TLS. NEOHIRO_APT_HTTPS_NOVERIFY=1 Skip the post-rewrite `apt-get update` check. - NEOHIRO_APT_HTTPS_STRICT=1 Treat leftover plaintext repos as a hard error. + NEOHIRO_APT_HTTPS_STRICT=1 Treat leftover plaintext as a hard error. NEOHIRO_APT_BLOCK_PORT80=1 Also add `ufw deny out 80/tcp` so no process can open a plaintext package connection at all. Off by default because a blanket outbound block also affects unrelated protocols. + NEOHIRO_APT_FAMILY=apt|dnf|zypper|pacman|none + Pin the detected package manager (CI images). USAGE exit 0 ;; diff --git a/tests/test_apt_https.sh b/tests/test_apt_https.sh index 6ae5128..a1142f5 100644 --- a/tests/test_apt_https.sh +++ b/tests/test_apt_https.sh @@ -124,10 +124,12 @@ else fail_t "NEOHIRO_APT_FAMILY pins the detected family" "got: ${PINNED:-}" fi -if grep -q 'NEOHIRO_APT_FAMILY' "$SRC" && grep -q 'NEOHIRO_APT_FAMILY' "$LIB"; then - ok_t "NEOHIRO_APT_FAMILY is honoured by the inline fallback too" +# Single source of truth: the family pin lives only in the library. The +# curl|bash path sources that library, so there is no second copy to drift. +if grep -q 'NEOHIRO_APT_FAMILY' "$LIB"; then + ok_t "NEOHIRO_APT_FAMILY is implemented in the single canonical library" else - fail_t "NEOHIRO_APT_FAMILY is honoured by the inline fallback too" "missing in one of the two copies" + fail_t "NEOHIRO_APT_FAMILY is implemented in the single canonical library" "not found in lib" fi # ============================================================================ @@ -808,27 +810,60 @@ else fi # ============================================================================ -# Parity: linuxinstall.sh's inline fallback must expose the same API +# Single implementation: the curl|bash path must LOAD the canonical library # ============================================================================ -INLINE_FNS="apt_https_guard apt_https_enforce apt_https_report apt_https_revert \ -apt_https_family apt_https_status_text apt_https_etc_dir apt_https_backup_dir apt_https_conf_file \ -_apt_https_repo_files _apt_https_foreign_repo_files _apt_https_plaintext_in_apt_file \ -_apt_https_plaintext_in_repo_file _apt_https_backup_path _apt_https_backup_once _apt_https_stage \ -_apt_https_write_conf _apt_https_rewrite_file _apt_https_rewrite_apt_sources \ -_apt_https_revert_apt_sources _apt_https_audit_family _apt_https_rewrite_family \ -_apt_https_flatpak_plaintext _apt_https_check_ca_certs _apt_https_block_port80 _apt_priv \ -_apt_https_verify_apt_sources" -MISSING="" -for fn in $INLINE_FNS; do - grep -qE "^[[:space:]]+${fn}\(\)" "$SRC" || MISSING="$MISSING inline:$fn" - grep -qE "^${fn}\(\)" "$LIB" || MISSING="$MISSING lib:$fn" +# linuxinstall.sh used to carry a ~400-line inline copy of the guard. That is +# the worst arrangement for security-critical code: a fix in lib/apt-https.sh +# silently did not reach `curl | sudo bash`, the documented primary install +# method. It now resolves and sources the canonical library instead. +if grep -q '_apt_https_resolved' "$SRC" && \ + grep -q 'REPO_RAW_BASE}/lib/apt-https.sh' "$SRC"; then + ok_t "curl|bash path resolves lib/apt-https.sh instead of duplicating it" +else + fail_t "curl|bash path resolves lib/apt-https.sh instead of duplicating it" \ + "resolver block missing" +fi + +# No helper from the library may be redefined inline any more. If one creeps +# back in, the two copies can drift again -- which is the bug this guards. +INLINE_DUPLICATES="" +for fn in _apt_https_plaintext_in_apt_file _apt_https_atomic_replace \ + _apt_https_prep_repo_stage _apt_https_rewrite_file \ + _apt_https_source_files _apt_https_source_plaintext \ + _apt_https_source_prefer_https _apt_https_managed_files \ + _apt_https_plaintext_generic; do + if grep -qE "^[[:space:]]+${fn}\(\) *\{" "$SRC"; then + INLINE_DUPLICATES="$INLINE_DUPLICATES $fn" + fi done -if [ -z "$MISSING" ]; then - ok_t "inline fallback and lib/apt-https.sh expose the same API" +if [ -z "$INLINE_DUPLICATES" ]; then + ok_t "no library helper is redefined inline in linuxinstall.sh" +else + fail_t "no library helper is redefined inline in linuxinstall.sh" \ + "duplicated again:$INLINE_DUPLICATES" +fi + +# When the library genuinely cannot be loaded, every public entry point must +# still exist so no caller hits an undefined function. +if grep -q 'apt_https_guard() { _apt_https_unavailable' "$SRC" && \ + grep -q 'apt_https_enforce() { _apt_https_unavailable' "$SRC" && \ + grep -q 'apt_https_report() { _apt_https_unavailable' "$SRC" && \ + grep -q 'apt_https_revert() { _apt_https_unavailable' "$SRC" && \ + grep -q 'apt_https_status_text(){ return 0; }' "$SRC"; then + ok_t "degraded mode still defines every public entry point" else - fail_t "inline fallback and lib/apt-https.sh expose the same API" "missing:$MISSING" + fail_t "degraded mode still defines every public entry point" \ + "a stub is missing; a caller would hit an undefined function" fi +# The helpers that callers outside this file use must be stubbed too. +for fn in apt_https_backup_dir apt_https_conf_file; do + if grep -qE "^[[:space:]]+${fn}\(\) *\{" "$SRC"; then + ok_t "degraded mode stubs $fn (used by _step_apt_https)" + else + fail_t "degraded mode stubs $fn (used by _step_apt_https)" "not stubbed" + fi +done # ============================================================================ # _step_apt_https exit-status plumbing # ============================================================================ @@ -1199,12 +1234,14 @@ if printf '%s' "$GUARD_BODY" | grep -q 'return 0'; then else fail_t "apt_https_guard always returns 0 (lib)" "no unconditional return 0" fi -INLINE_GUARD="$(awk '/^[[:space:]]+apt_https_guard\(\) \{/,/^[[:space:]]+\}/' "$SRC")" -if printf '%s' "$INLINE_GUARD" | grep -q 'apt_https_enforce .* || true'; then - ok_t "apt_https_guard swallows enforcement failure (inline fallback)" +# With the library loaded the guarantee comes from the lib's own guard; with it +# unavailable, the degraded stub must also return 0. Either way a package +# operation is never blocked by the precaution itself. +if grep -q 'apt_https_guard() { _apt_https_unavailable; return 0; }' "$SRC"; then + ok_t "degraded apt_https_guard also swallows failure (inline resolver)" else - fail_t "apt_https_guard swallows enforcement failure (inline fallback)" \ - "enforce call is not guarded with '|| true'" + fail_t "degraded apt_https_guard also swallows failure (inline resolver)" \ + "stub could return non-zero and abort a package operation" fi # Behaviour: a guard call must return 0 even when enforcement fails hard. @@ -1296,11 +1333,13 @@ for v in _APT_HTTPS_VERDICT _APT_HTTPS_CHANGED _APT_HTTPS_REVERTED \ fail_t "$v is initialised at the top level of lib/apt-https.sh" \ "no top-level assignment found" fi - if grep -qE "^ ${v}=" "$SRC"; then - ok_t "$v is initialised at the top level of the inline fallback" + # The inline path no longer declares these: it sources the library, which + # owns every global. A duplicate here would reintroduce the drift hazard. + if grep -qE "^[[:space:]]+${v}=" "$SRC"; then + fail_t "$v is NOT duplicated into the inline resolver" \ + "declared in both places again - they can drift" else - fail_t "$v is initialised at the top level of the inline fallback" \ - "no top-level assignment found" + ok_t "$v is not duplicated into the inline resolver (single owner)" fi done @@ -1326,13 +1365,8 @@ else fail_t "CA check returns before warning when a trust store IS present (lib)" \ "no early return: a case arm can fall through and warn on a healthy host" fi -CA_INLINE="$(awk '/^[[:space:]]+_apt_https_check_ca_certs\(\) \{/,/^[[:space:]]+\}/' "$SRC")" -if printf '%s' "$CA_INLINE" | grep -qE '\[ "\$missing" = "0" \] && return 0'; then - ok_t "CA check returns early when a trust store IS present (inline fallback)" -else - fail_t "CA check returns early when a trust store IS present (inline fallback)" \ - "inline copy lacks the early return" -fi +# Single owner: the CA check lives only in the library, so the structural +# guard above is the whole invariant -- no inline copy to keep in sync. # Behavioural half: this host has no /etc CA bundle, so the missing branch is # live -- assert it warns exactly once no matter how many call sites fire. @@ -1417,19 +1451,428 @@ case "$CONF_MODE" in *) fail_t "atomic drop-in install lands as 0644" "got: $CONF_MODE" ;; esac -# Structural: the rewrite must go through mv, not cp-into-place. -if grep -q '_apt_https_atomic_replace' "$LIB" && grep -q '_apt_https_atomic_replace' "$SRC"; then - ok_t "lib and inline fallback both route writes through _apt_https_atomic_replace" +# Structural: every write is a staging rename, never cp-into-place, and the +# logic lives in exactly one place. +if grep -q '_apt_https_atomic_replace' "$LIB"; then + ok_t "all writes route through _apt_https_atomic_replace" else - fail_t "lib and inline fallback both route writes through _apt_https_atomic_replace" \ - "atomic helper not wired in one of the two copies" + fail_t "all writes route through _apt_https_atomic_replace" "helper not wired into the lib" fi -if grep -q '_apt_priv mv -f "\$stage" "\$target"' "$LIB" && \ - grep -q '_apt_priv mv -f "\$stage" "\$target"' "$SRC"; then +if grep -q '_apt_priv mv -f "\$stage" "\$target"' "$LIB"; then ok_t "atomic replace finishes with rename(2) via mv -f" else fail_t "atomic replace finishes with rename(2) via mv -f" "mv of the stage file missing" fi +if grep -q '_apt_https_atomic_replace' "$SRC"; then + fail_t "atomic replace is not reimplemented in the inline resolver" "duplicate copy" +else + ok_t "atomic replace is not reimplemented in the inline resolver (single owner)" +fi + +# ============================================================================ +# App-store coverage: every store must be CHECKED and USED +# ============================================================================ +# The precaution is not apt-specific. Each distro and language has its own +# "app store", and several ship plaintext HTTP by default. These tests drive +# the per-source detectors directly (they do not gate on applicability) so +# every store is exercised on any host. +APT_SOURCES="apt dnf yum zypper pacman apk flatpak snap docker brew pip npm cargo gem nix fwupd" +REGISTERED="$(_apt_https_source_ids | tr '\n' ' ')" + +# 1) Every store we claim to cover is registered. +MISSING_SOURCES="" +for s in $APT_SOURCES; do + case " $REGISTERED " in *" $s "*) : ;; *) MISSING_SOURCES="$MISSING_SOURCES $s" ;; esac +done +if [ -z "$MISSING_SOURCES" ]; then + ok_t "all app stores are registered: $(printf '%s' "$REGISTERED" | tr '\n' ' ')" +else + fail_t "all app stores are registered" "missing:$MISSING_SOURCES" +fi + +# 2) Every registered store has a label (a missing label means a blank row in +# the report, i.e. a store the user cannot identify). +NO_LABEL="" +for s in $REGISTERED; do + [ -n "$s" ] || continue + [ "$(_apt_https_source_label "$s")" = "$s" ] && NO_LABEL="$NO_LABEL $s" +done +if [ -z "$NO_LABEL" ]; then + ok_t "every registered store has a human-readable label" +else + fail_t "every registered store has a human-readable label" "fallback label used:$NO_LABEL" +fi + +# fake_tool — put an executable stub for on PATH so a store's +# applicability gate passes on a host that does not actually run that distro. +# Used to exercise the gate-driven sweep for apk/brew/docker. +fake_tool() { + local n="$1" + mkdir -p "$FAKEBIN" + printf '#!/bin/sh\nexit 0\n' > "$FAKEBIN/$n" + chmod +x "$FAKEBIN/$n" 2>/dev/null || true +} +unfake_tool() { rm -f "$FAKEBIN/$1" 2>/dev/null || true; } + +# 3) Per-store detection: write a plaintext config for each store and assert +# the detector finds it. A store that silently finds nothing is the exact +# failure this library exists to prevent. +seed_store_config() { + case "$1" in + apk) mkdir -p "$SANDBOX/apk"; printf '%s\n' \ + 'http://dl-cdn.alpinelinux.org/alpine/v3.19/main' \ + 'https://dl-cdn.alpinelinux.org/alpine/v3.19/community' > "$SANDBOX/apk/repositories" ;; + docker) mkdir -p "$SANDBOX/docker"; printf '%s\n' \ + '{ "registry-mirrors": ["http://mirror.internal:5000"], "insecure-registries": ["reg.local:5000"] }' > "$SANDBOX/docker/daemon.json" ;; + pip) mkdir -p "$FAKEHOME/.config/pip"; printf '%s\n' \ + '[global]' 'index-url = http://pypi.internal/simple' > "$FAKEHOME/.config/pip/pip.conf" ;; + npm) printf '%s\n' 'registry=http://npm.internal/' > "$FAKEHOME/.npmrc" ;; + cargo) mkdir -p "$FAKEHOME/.cargo"; printf '%s\n' \ + '[source.crates-io]' 'registry = "sparse+http://index.internal/"' > "$FAKEHOME/.cargo/config.toml" ;; + gem) printf '%s\n' '---' ':sources:' '- http://rubygems.internal/' > "$FAKEHOME/.gemrc" ;; + nix) mkdir -p "$SANDBOX/nix"; printf '%s\n' \ + 'substituters = http://cache.nixos.org' 'channel = http://nixos.org/channels' > "$SANDBOX/nix/nix.conf" ;; + fwupd) mkdir -p "$SANDBOX/fwupd/remotes.d"; printf '%s\n' \ + '[lvfs]' 'UpdateURI=http://fwupd.lvfs.org/fwupd-stable.xml.gz' > "$SANDBOX/fwupd/remotes.d/lvfs.conf" ;; + dnf|yum) mkdir -p "$SANDBOX/yum.repos.d"; printf '%s\n' \ + '[e]' 'name=E' 'baseurl=http://mirror.internal/epel/' 'gpgcheck=1' > "$SANDBOX/yum.repos.d/e.repo" ;; + zypper) mkdir -p "$SANDBOX/zypp/repos.d"; printf '%s\n' \ + '[r]' 'name=R' 'baseurl=http://download.internal/oss/' > "$SANDBOX/zypp/repos.d/r.repo" ;; + pacman) mkdir -p "$SANDBOX/pacman.d"; printf '%s\n' \ + '## Worldwide' 'Server=http://mirror.internal/archlinux/$repo/os/$arch' > "$SANDBOX/pacman.d/mirrorlist" ;; + *) return 1 ;; + esac + return 0 +} + +for s in apk docker pip npm cargo gem nix fwupd; do + reset_sandbox + seed_store_config "$s" + FOUND="$(_apt_https_source_plaintext "$s" 2>/dev/null)" + if printf '%s' "$FOUND" | grep -q 'http://'; then + ok_t "$s: plaintext endpoint is detected" + else + fail_t "$s: plaintext endpoint is detected" "detector returned nothing for $s" + fi +done + +for s in dnf zypper pacman; do + reset_sandbox + TEST_FAMILY="$s" + seed_store_config "$s" + FOUND="$(_apt_https_source_plaintext "$s" 2>/dev/null)" + if printf '%s' "$FOUND" | grep -q 'http://'; then + ok_t "$s: plaintext endpoint is detected" + else + fail_t "$s: plaintext endpoint is detected" "detector returned nothing for $s" + fi +done +TEST_FAMILY=apt + +# 4) Comment lines must not be treated as fetch targets in the generic +# detector, or every config with a doc link would look dirty. +for s in apk docker npm gem nix fwupd; do + reset_sandbox + seed_store_config "$s" + # Prepend a comment that mentions a plaintext URL. + FIRST_FILE="$(_apt_https_source_files "$s" | head -1)" + if [ -n "$FIRST_FILE" ] && [ -f "$FIRST_FILE" ]; then + case "$FIRST_FILE" in + *.json) printf '%s\n' '{ "_comment": "see http://docs.internal for details", "registry-mirrors": ["https://ok.internal"] }' > "$FIRST_FILE" ;; + *) printf '%s\n%s\n' '# docs: http://docs.internal/see-me' "$(cat "$FIRST_FILE")" > "$FIRST_FILE" ;; + esac + if [ "$FIRST_FILE" = "$FAKEHOME/.npmrc" ]; then + printf '%s\n%s\n' '; docs: http://docs.internal/see-me' "$(cat "$FIRST_FILE")" > "$FIRST_FILE" + fi + CLEAN="$(_apt_https_source_plaintext "$s" 2>/dev/null)" + if [ -z "$CLEAN" ] || ! printf '%s' "$CLEAN" | grep -q 'docs.internal'; then + ok_t "$s: commented plaintext URLs are not flagged" + else + fail_t "$s: commented plaintext URLs are not flagged" "false positive: $CLEAN" + fi + else + fail_t "$s: commented plaintext URLs are not flagged" "no config file located for $s" + fi +done + +# 5) Opt-in rewrite must convert every file-based store, not just apt. +for s in apk docker pip npm cargo gem nix fwupd; do + reset_sandbox + seed_store_config "$s" + BEFORE="$(_apt_https_source_plaintext "$s" 2>/dev/null)" + _apt_https_source_prefer_https "$s" + AFTER="$(_apt_https_source_plaintext "$s" 2>/dev/null)" + if [ -n "$BEFORE" ] && [ -z "$AFTER" ] && [ "${_APT_HTTPS_CHANGED:-0}" -gt 0 ]; then + ok_t "$s: prefer-https rewrite removes the plaintext endpoint" + else + fail_t "$s: prefer-https rewrite removes the plaintext endpoint" \ + "before=[$(printf '%s' "$BEFORE" | tr '\n' '|')] after=[$(printf '%s' "$AFTER" | tr '\n' '|')] changed=${_APT_HTTPS_CHANGED:-?}" + fi + # And the rewritten value must actually be https, not just "no longer http". + ANY_HTTPS=0 + while IFS= read -r f; do + [ -n "$f" ] && grep -q 'https://' "$f" 2>/dev/null && ANY_HTTPS=1 + done < <(_apt_https_source_files "$s") + if [ "$ANY_HTTPS" = "1" ]; then + ok_t "$s: rewrite actually wrote https:// into the config" + else + fail_t "$s: rewrite actually wrote https:// into the config" "no https:// present" + fi + # Backup must exist so the change is revertible. + BK="$(find "$BACKUPS" -name '*.orig' 2>/dev/null | wc -l | tr -d ' ')" + if [ "$BK" -gt 0 ]; then + ok_t "$s: rewrite backed the config up first" + else + fail_t "$s: rewrite backed the config up first" "no .orig in $BACKUPS" + fi +done + +# 6) "if available": non-apt stores must NOT be rewritten unattended. +for s in apk docker pip npm cargo gem nix fwupd; do + reset_sandbox + seed_store_config "$s" + _APT_HTTPS_DONE="" + apt_https_enforce "no-optin-$s" >/dev/null 2>&1 + STILL="$(_apt_https_source_plaintext "$s" 2>/dev/null)" + if [ -n "$STILL" ]; then + ok_t "$s: left untouched without NEOHIRO_APT_HTTPS_REWRITE=1" + else + fail_t "$s: left untouched without NEOHIRO_APT_HTTPS_REWRITE=1" "rewritten unattended" + fi +done + +# 7) With the opt-in, apt_https_enforce must sweep every store, not just apt. +# apk and brew are stubbed onto PATH so their applicability gate passes: +# the gate deliberately skips stores that are not installed. +fake_tool apk; fake_tool npm +reset_sandbox +seed_store_config apk +seed_store_config nix +seed_store_config npm +_APT_HTTPS_DONE="" +NEOHIRO_APT_HTTPS_REWRITE=1 apt_https_enforce "sweep" >/dev/null 2>&1 +LEFT=0 +for s in apk nix npm; do + [ -n "$(_apt_https_source_plaintext "$s" 2>/dev/null)" ] && LEFT=$((LEFT + 1)) +done +if [ "$LEFT" = "0" ]; then + ok_t "opt-in sweep cleans apk, nix and npm in one pass" +else + fail_t "opt-in sweep cleans apk, nix and npm in one pass" "$LEFT store(s) still plaintext" +fi + +# 7b) The sweep must NOT touch a store whose tool is absent, even with the +# opt-in set: rewriting config for software that is not installed is +# out of scope and can surprise a later manual install. +unfake_tool apk +reset_sandbox +seed_store_config apk +_APT_HTTPS_DONE="" +NEOHIRO_APT_HTTPS_REWRITE=1 apt_https_enforce "sweep-uninstalled" >/dev/null 2>&1 +if [ -n "$(_apt_https_source_plaintext apk 2>/dev/null)" ]; then + ok_t "opt-in sweep skips a store whose tool is not installed" +else + fail_t "opt-in sweep skips a store whose tool is not installed" \ + "rewrote config for an absent tool" +fi +fake_tool apk + +env_src_id() { + case "$1" in + PIP_INDEX_URL|PIP_EXTRA_INDEX_URL) printf 'pip' ;; + NPM_CONFIG_REGISTRY|npm_config_registry) printf 'npm' ;; + CARGO_REGISTRIES_CRATES_IO_INDEX) printf 'cargo' ;; + GEM_SOURCE) printf 'gem' ;; + HOMEBREW_*) printf 'brew' ;; + *) printf '' ;; + esac +} + +# 8) Env-configured transports are reported. Exercised through +# _apt_https_source_env directly, because the aggregate only reports a +# store whose tool is installed -- and no host runs all of them. +for pair in "PIP_INDEX_URL=http://pypi.internal/simple" \ + "PIP_EXTRA_INDEX_URL=http://extra.internal/simple" \ + "NPM_CONFIG_REGISTRY=http://npm.internal/" \ + "npm_config_registry=http://npm2.internal/" \ + "CARGO_REGISTRIES_CRATES_IO_INDEX=http://crates.internal/" \ + "GEM_SOURCE=http://rubygems.internal/" \ + "HOMEBREW_API_DOMAIN=http://brew.internal" \ + "HOMEBREW_ARTIFACT_DOMAIN=http://brew-artifacts.internal" \ + "HOMEBREW_BREW_GIT_REMOTE=http://git.internal/brew.git"; do + reset_sandbox + VAR="${pair%%=*}" + VAL="${pair#*=}" + SRC="$(env_src_id "$VAR")" + # shellcheck disable=SC2086 + OUT="$(env "$VAR=$VAL" bash -c ". '$LIB' >/dev/null 2>&1; _apt_https_source_env $SRC" 2>/dev/null)" + if printf '%s' "$OUT" | grep -q "$VAR=$VAL"; then + ok_t "env transport detected: $VAR" + else + fail_t "env transport detected: $VAR" "got: $(printf '%s' "$OUT" | tr '\n' '|')" + fi +done + +# 8b) An unset variable must produce nothing (no empty VAR= noise). +reset_sandbox +if [ -z "$(_apt_https_source_env pip 2>/dev/null)" ]; then + ok_t "no env noise when transport variables are unset" +else + fail_t "no env noise when transport variables are unset" \ + "got: $(_apt_https_source_env pip 2>/dev/null | tr '\n' '|')" +fi + +# 9) Revert must restore every store it touched. +reset_sandbox +seed_store_config apk +seed_store_config nix +_APT_HTTPS_DONE="" +NEOHIRO_APT_HTTPS_REWRITE=1 apt_https_enforce "revert-all" >/dev/null 2>&1 +if [ -n "$(_apt_https_source_plaintext apk 2>/dev/null)" ]; then + fail_t "precondition: apk was rewritten before revert" "still https" +else + ok_t "precondition: apk was rewritten before revert" +fi +apt_https_revert >/dev/null 2>&1 +if [ -n "$(_apt_https_source_plaintext apk 2>/dev/null)" ] && \ + [ -n "$(_apt_https_source_plaintext nix 2>/dev/null)" ]; then + ok_t "revert restores non-apt stores too, not just apt" +else + fail_t "revert restores non-apt stores too, not just apt" \ + "apk=[$(_apt_https_source_plaintext apk 2>/dev/null)] nix=[$(_apt_https_source_plaintext nix 2>/dev/null)]" +fi + +# 10) The aggregate must span every store, and report rc=1 when any is dirty. +reset_sandbox +seed_store_config gem +if apt_https_status_text 2>/dev/null | grep -q 'rubygems.internal'; then + ok_t "apt_https_status_text aggregates non-apt stores" +else + fail_t "apt_https_status_text aggregates non-apt stores" "gem endpoint missing from aggregate" +fi +apt_https_report >/dev/null 2>&1 +if [ $? -eq 1 ]; then + ok_t "report exits 1 when any store is plaintext" +else + fail_t "report exits 1 when any store is plaintext" "rc=$?" +fi +REPORT_OUT="$(apt_https_report 2>&1)" +if printf '%s' "$REPORT_OUT" | grep -q 'gem sources'; then + ok_t "report names the offending store" +else + fail_t "report names the offending store" "no 'gem sources' row" +fi + +# 11) Applicability gating: a store with no tooling on this host must not be +# listed as a dirty row, so the report stays readable on a minimal box. +reset_sandbox +_missing_tools=0 +for tool in docker flatpak snap apk brew; do + command -v "$tool" >/dev/null 2>&1 || _missing_tools=$((_missing_tools + 1)) +done +if [ "$_missing_tools" -gt 0 ]; then + ok_t "applicability gate can be exercised (host is missing $_missing_tools store tool(s))" +else + ok_t "applicability gate: all store tools present on this host" +fi +reset_sandbox +if _apt_https_source_applicable docker 2>/dev/null; then + if command -v docker >/dev/null 2>&1; then + ok_t "docker marked applicable when the binary exists" + else + fail_t "docker marked applicable when the binary exists" "reported applicable with no docker" + fi +else + ok_t "docker marked not-applicable when the binary is absent" +fi + +# ============================================================================ +# Repository encoding hygiene +# ============================================================================ +# These scripts are full of box-drawing and em-dash characters, so they are +# one careless editor, IDE, or shell one-liner away from being silently +# re-encoded. That failure mode is nasty: the file still parses, the tests +# mostly pass, and only the snapshot or a diff comparison notices. It bit this +# work twice (a UTF-8 file read as cp1252 and rewritten, then a repair pass +# that introduced U+FFFD), so it gets an automated guard. +# +# Pure bash + coreutils on purpose: the CI matrix includes a busybox-based +# bash:3.2-alpine image where python may be absent. +if command -v iconv >/dev/null 2>&1; then + BAD_UTF8="" + while IFS= read -r f; do + iconv -f UTF-8 -t UTF-8 "$f" >/dev/null 2>&1 || BAD_UTF8="$BAD_UTF8 $f" + done </dev/null && FFFD_FILES="$FFFD_FILES $f" + if LC_ALL=C grep -q "$MOJI_A" "$f" 2>/dev/null || LC_ALL=C grep -q "$MOJI_C" "$f" 2>/dev/null; then + MOJI_FILES="$MOJI_FILES $f" + fi +done < Date: Sun, 4 Oct 2026 10:21:59 +0200 Subject: [PATCH 06/17] fix: collision-proof backup names, single family probe, actionable report Sign-off review of the multi-store guard. Three concrete defects, one latent hazard, and a report that could not actually be acted on. 1. Backup filenames collided. _flattening a path is not injective_: /a/b/c and /a_b/c both became _a_b_c.orig. A collision means `--apt-https-off` restores one file with another file's contents -- a silent corruption of a package manager or a pip/npm config, discovered at the worst possible moment. Backup names now carry a cksum of the real path. cksum is POSIX and stable across shells, runs and machines, which matters because these backups get inspected and moved during an incident. Falls back to the flattened name alone if cksum is somehow unavailable. 2. The detected family was re-probed 4x per pass. _apt_https_source_applicable called apt_https_family itself, and the report and status loops call it once per store. Each call is up to five `command -v` PATH scans, so a single audit did ~40 of them. Beyond the waste, it was a consistency hazard: if PATH changed mid-run, a store could be "applicable" in one check and absent from the next, so the same file could be reported twice or not at all. Callers now compute the family once and pass it in. Measured with a probe-counting override: 4 probes per status_text -> 1, and the report now makes a single consistent decision per run. 3. Unset HOME built paths at the filesystem root. `"${HOME:-}/.pip/pip.conf"` degrades to "/.pip/pip.conf" when HOME is empty, so the per-user store detectors probed paths directly under /. Harmless today because those files do not exist, but it is wrong and would start matching the moment anything created them. Per-user config is now only consulted when HOME is non-empty. 4. The report told you something was wrong but not what to do. It said "repoint at an https:// endpoint" in the abstract. For flatpak and snap that advice is actively misleading: there is no config file to rewrite, so the opt-in sweep cannot fix them and no amount of editing helps. Every dirty store now prints a one-line remediation with the real command or the real config key -- including `flatpak remote-modify --url=https://...`, and an honest note that the snapd store cannot be repointed at all. Also fixed a misstatement in the report footer: it described NEOHIRO_APT_HTTPS_REWRITE=1 as an "apt only" rewrite, when that flag sweeps every installed store. apt is the one rewritten *without* the flag. And corrected a stale comment that still said "the three functions below" now that the rewrite path has grown a staging helper, two prep functions and a collision-proof backup path. Two tests were pinning the old backup filename by reimplementing the naming scheme. They now ask the library where the backup is, so a future change to the scheme cannot break them for the wrong reason. tests/test_apt_https.sh 247 -> 253. New coverage: family probed exactly once per pass, applicability check does not re-probe when given a hint, backup names distinct for paths that flatten alike, single path component, and deterministic across calls. Both fixes are negative-controlled -- ignoring the family hint fails the probe-count test, and disabling the checksum fails the collision test. Full suite: 684 passed, 0 failures. bash -n clean on every .sh. Skipped, noted rather than guessed: - Whether `Acquire::Retries` / `AllowRedirect` behave identically on apt 1.x and 2.x. Verifying that needs a matrix container per apt release; I have no such environment here, and guessing at apt's parser is worse than leaving the long-standing keys alone. - Making flatpak remotes auto-rewritable. It would mean shelling out to `flatpak remote-modify`, mutating running-session state during a package operation. That is a behaviour change with real blast radius, not a review fix, so it wants its own decision and its own tests. --- README.md | 2 +- lib/apt-https.sh | 132 ++++++++++++++++++++++++++++++---------- tests/test_apt_https.sh | 87 +++++++++++++++++++++++++- 3 files changed, 184 insertions(+), 37 deletions(-) diff --git a/README.md b/README.md index 275ade6..ad352e4 100644 --- a/README.md +++ b/README.md @@ -761,7 +761,7 @@ dumps — add to `/etc/security/limits.conf`: ```bash bash tests/run-all.sh # every suite, with a summary bash tests/test_linuxinstall.sh # 67 tests: parse, logic, UX coverage, snapshot -bash tests/test_apt_https.sh # 247 tests: transport guard + encoding hygiene +bash tests/test_apt_https.sh # 253 tests: transport guard + encoding hygiene bash tests/test_updater.sh # 45 tests: dispatcher, race safety, version floor shellcheck -S warning *.sh lib/*.sh tests/*.sh # lint ``` diff --git a/lib/apt-https.sh b/lib/apt-https.sh index d70c4dd..e3f8f80 100644 --- a/lib/apt-https.sh +++ b/lib/apt-https.sh @@ -289,11 +289,28 @@ _APT_HTTPS_GENERIC_RE='/^[[:space:]]*(#|;)/! s|http://|https://|g' # ── Backups ────────────────────────────────────────────────────────────────── -# _apt_https_backup_path — deterministic, path-flattened backup name. +# _apt_https_backup_path — deterministic, collision-resistant name. +# +# The path is flattened so it is a legal filename inside the backup +# directory, but flattening alone is NOT injective: /a/b/c and /a_b/c both +# become _a_b_c. A collision here means reverting one file restores another +# file's contents, which is a silent, hard-to-trace corruption of a package +# manager. So append a checksum of the real path. +# +# cksum (POSIX) is used rather than a shell hash so the value is stable +# across shells, runs, and machines -- these backups may be inspected or +# moved during an incident. _apt_https_backup_path() { - local flat - flat="$(printf '%s' "$1" | tr '/' '_')" - printf '%s/%s.orig' "$(apt_https_backup_dir)" "$flat" + local flat sum + flat="$(printf '%s' "$1" | tr -c 'A-Za-z0-9._-' '_')" + sum="$(printf '%s' "$1" | cksum 2>/dev/null | tr -d ' ' | cut -d' ' -f1)" + if [ -n "$sum" ]; then + printf '%s/%s-%s.orig' "$(apt_https_backup_dir)" "$flat" "$sum" + else + # cksum unavailable: the flattened name alone is still better than + # hashing nothing, and this is recorded so it is not mistaken for a bug. + printf '%s/%s.orig' "$(apt_https_backup_dir)" "$flat" + fi } # _apt_https_backup_once — copy aside the first time we touch it. @@ -440,12 +457,13 @@ _apt_https_prep_repo_stage() { # ── apt source rewriting ───────────────────────────────────────────────────── # -# The three functions below report their result through globals -# (_APT_HTTPS_VERDICT / _APT_HTTPS_CHANGED / _APT_HTTPS_REVERTED) instead of -# stdout. They emit progress messages of their own, and a command -# substitution would fold that text into the value it returns -- which then -# lands inside an arithmetic expansion and turns a word like "Restored" into -# a variable lookup. Globals keep the two channels separate. +# The rewriters below report their result through globals +# (_APT_HTTPS_VERDICT / _APT_HTTPS_CHANGED / _APT_HTTPS_REVERTED / +# _APT_HTTPS_REFUSED) instead of stdout. They emit progress messages of +# their own, and a command substitution would fold that text into the value +# it returns -- which then lands inside an arithmetic expansion and turns a +# word like "Restored" into a variable lookup. Globals keep the message +# channel and the value channel separate. _APT_HTTPS_VERDICT="" _APT_HTTPS_CHANGED=0 @@ -601,8 +619,9 @@ _apt_https_source_label() { # _apt_https_source_files — config files that carry this source's URLs. _apt_https_source_files() { - local etc f + local etc f home etc="$(apt_https_etc_dir)" + home="${HOME:-}" case "$1" in apt) _apt_https_repo_files ;; dnf|yum) _apt_https_foreign_repo_files dnf ;; @@ -618,23 +637,31 @@ _apt_https_source_files() { ;; # pip: system config plus the two variables that usually win. pip) - for f in "${etc}/pip.conf" "${etc}/xdg/pip/pip.conf" \ - "${HOME:-}/.pip/pip.conf" "${HOME:-}/.config/pip/pip.conf"; do + for f in "${etc}/pip.conf" "${etc}/xdg/pip/pip.conf"; do [ -f "$f" ] && printf '%s\n' "$f" done + # Per-user config only makes sense with a real HOME. Without this guard + # an unset HOME builds "/.config/pip/pip.conf" -- a path at the + # filesystem root, which would be probed and could match. + if [ -n "$home" ]; then + for f in "${home}/.pip/pip.conf" "${home}/.config/pip/pip.conf"; do + [ -f "$f" ] && printf '%s\n' "$f" + done + fi ;; npm) - for f in "${etc}/npmrc" "${HOME:-}/.npmrc"; do - [ -f "$f" ] && printf '%s\n' "$f" - done + [ -f "${etc}/npmrc" ] && printf '%s\n' "${etc}/npmrc" + [ -n "$home" ] && [ -f "${home}/.npmrc" ] && printf '%s\n' "${home}/.npmrc" ;; cargo) - [ -f "${HOME:-}/.cargo/config.toml" ] && printf '%s\n' "${HOME:-}/.cargo/config.toml" + if [ -n "$home" ] && [ -f "${home}/.cargo/config.toml" ]; then + printf '%s\n' "${home}/.cargo/config.toml" + fi [ -f "${etc}/cargo/config.toml" ] && printf '%s\n' "${etc}/cargo/config.toml" ;; gem) [ -f "${etc}/gemrc" ] && printf '%s\n' "${etc}/gemrc" - [ -f "${HOME:-}/.gemrc" ] && printf '%s\n' "${HOME:-}/.gemrc" + [ -n "$home" ] && [ -f "${home}/.gemrc" ] && printf '%s\n' "${home}/.gemrc" ;; nix) [ -f "${etc}/nix/nix.conf" ] && printf '%s\n' "${etc}/nix/nix.conf" @@ -679,14 +706,23 @@ _apt_https_source_env() { return 0 } -# _apt_https_source_applicable — false when this host has no such store. -# Keeps the report from listing eleven N/A rows on a minimal box. +# _apt_https_source_applicable [family] +# False when this host has no such store, so the report does not list eleven +# N/A rows on a minimal box. +# +# The detected family is passed in by callers rather than re-probed here. +# apt_https_family costs up to five `command -v` PATH scans, and the report +# loop called it once per RPM/Arch store -- twice over, since the report and +# the per-source query each ran the loop. That is wasted work and a latent +# inconsistency: if PATH changed mid-run a store could be "applicable" in one +# pass and missing from the next, and the same file could be reported twice. _apt_https_source_applicable() { - local id="$1" + local id="$1" fam="${2:-}" case "$id" in # Only the package manager actually present should be audited as such. dnf|yum|zypper|pacman) - [ "$(apt_https_family)" = "$id" ] || return 1 + [ -n "$fam" ] || fam="$(apt_https_family)" + [ "$fam" = "$id" ] || return 1 ;; snap) command -v snap >/dev/null 2>&1 || return 1 @@ -720,6 +756,33 @@ _apt_https_source_applicable() { return 0 } +# _apt_https_source_hint — one short line on how to fix this store. +# +# Printed only for stores that actually have a plaintext endpoint. Without it +# the report says "repoint at https" and leaves the reader to work out where +# that setting lives -- and for flatpak and snap there is no config file this +# library can rewrite at all, so the generic advice is actively misleading. +_apt_https_source_hint() { + case "$1" in + apt) printf 'rewrite automatically: sudo bash linuxinstall.sh --apt-https' ;; + dnf|yum) printf 'edit baseurl=/metalink=/mirrorlist= under /etc/yum.repos.d/*.repo' ;; + zypper) printf 'edit baseurl=/uri= under /etc/zypp/repos.d/*.repo' ;; + pacman) printf 'edit Server= lines in /etc/pacman.d/mirrorlist' ;; + apk) printf 'rewrite with NEOHIRO_APT_HTTPS_REWRITE=1, or edit /etc/apk/repositories' ;; + docker) printf 'set registry-mirrors to https:// in /etc/docker/daemon.json, drop insecure-registries, then restart docker' ;; + brew) printf 'export HOMEBREW_API_DOMAIN / HOMEBREW_BREW_GIT_REMOTE with an https:// URL' ;; + pip) printf 'export PIP_INDEX_URL=https://... (and PIP_EXTRA_INDEX_URL) or fix pip.conf' ;; + npm) printf 'npm config set registry https://registry.npmjs.org/' ;; + cargo) printf 'set the crates-io registry to sparse+https:// in ~/.cargo/config.toml' ;; + gem) printf 'gem sources --remove && gem sources --add https://rubygems.org/' ;; + nix) printf 'use substituters = https://cache.nixos.org in nix.conf' ;; + # No operator-editable transport config; a URL rewrite cannot fix these. + flatpak) printf 'flatpak remote-modify --url=https://... (no config file to rewrite)' ;; + snap) printf 'snapd-managed store; cannot be repointed without a custom snapd build' ;; + *) : ;; + esac +} + # _apt_https_source_plaintext — report this source's plaintext endpoints. _apt_https_source_plaintext() { local id="$1" f @@ -905,18 +968,19 @@ _apt_https_block_port80() { # One line per offending config line, plus "VAR=value" for env-configured # transports, so the output is directly actionable. apt_https_status_text() { - local id + local id fam + fam="$(apt_https_family)" while IFS= read -r id; do [ -n "$id" ] || continue - _apt_https_source_applicable "$id" || continue + _apt_https_source_applicable "$id" "$fam" || continue _apt_https_source_plaintext "$id" done < <(_apt_https_source_ids) return 0 } -# _apt_https_plaintext_for — findings for one source only. +# _apt_https_plaintext_for [family] — findings for one source only. apt_https_plaintext_for() { - _apt_https_source_applicable "$1" || return 0 + _apt_https_source_applicable "$1" "${2:-}" || return 0 _apt_https_source_plaintext "$1" } @@ -924,7 +988,7 @@ apt_https_plaintext_for() { # Returns 0 when nothing anywhere uses plaintext, 1 when any source does, # 2 when no package manager at all was detected. apt_https_report() { - local fam conf findings id label n clean applicable + local fam conf findings id label n clean fam="$(apt_https_family)" printf '\n%s\n' "$(_c '1;36m' '━━━ App-store transport security (HTTPS) ━━━')" printf ' %-34s %s\n' "Package manager:" "$fam" @@ -941,15 +1005,16 @@ apt_https_report() { clean=1 while IFS= read -r id; do [ -n "$id" ] || continue - _apt_https_source_applicable "$id" || continue + _apt_https_source_applicable "$id" "$fam" || continue label="$(_apt_https_source_label "$id")" - findings="$(apt_https_plaintext_for "$id")" + findings="$(apt_https_plaintext_for "$id" "$fam")" if [ -z "$findings" ]; then printf ' %-30s %s\n' "$label" "$(_c '1;32m' 'https only')" else n="$(printf '%s\n' "$findings" | wc -l | tr -d ' ')" printf ' %-30s %s\n' "$label" "$(_c '1;31m' "$n plaintext endpoint(s)")" printf '%s\n' "$findings" | sed 's/^/ /' + printf ' %s\n' "$(_apt_https_source_hint "$id")" clean=0 fi done < <(_apt_https_source_ids) @@ -961,9 +1026,10 @@ apt_https_report() { printf ' %s %s\n' "$(_c '1;32m' '[x]')" "No plaintext endpoints in any configured app store." else printf ' %s %s\n' "$(_c '1;33m' '[!]')" "Some stores still fetch over plaintext HTTP." - printf ' %s\n' " apt: sudo bash linuxinstall.sh --apt-https (automatic, verified)" - printf '%s\n' " others: repoint at an https:// endpoint, or opt in with" - printf '%s\n' " NEOHIRO_APT_HTTPS_REWRITE=1 and re-audit afterwards." + printf ' %s\n' " Use the per-store hint above. apt needs nothing: --apt-https" + printf '%s\n' " rewrites it unattended (and rolls back if a mirror cannot speak" + printf '%s\n' " TLS). For the others, repoint the endpoint, or opt in to a blanket" + printf '%s\n' " rewrite with NEOHIRO_APT_HTTPS_REWRITE=1 and re-audit afterwards." fi printf '\n' @@ -1059,7 +1125,7 @@ apt_https_enforce() { while IFS= read -r id; do [ -n "$id" ] || continue [ "$id" = "apt" ] && continue - _apt_https_source_applicable "$id" || continue + _apt_https_source_applicable "$id" "$fam" || continue _apt_https_source_prefer_https "$id" n="$_APT_HTTPS_CHANGED" if [ "${n:-0}" -gt 0 ] 2>/dev/null; then diff --git a/tests/test_apt_https.sh b/tests/test_apt_https.sh index a1142f5..43070c9 100644 --- a/tests/test_apt_https.sh +++ b/tests/test_apt_https.sh @@ -213,9 +213,10 @@ else fail_t "classic: bracketed options preserved during rewrite" "options mangled" fi -BAK_REL="$(printf '%s' "$SANDBOX" | tr '/' '_')" -BAK_REL="${BAK_REL#/}" -BAK="$BACKUPS/${BAK_REL}_apt_sources.list.orig" +# Ask the library where it keeps the backup instead of reimplementing the +# naming. Reconstructing it here meant every change to the backup-name scheme +# broke this test for the wrong reason. +BAK="$(_apt_https_backup_path "$SANDBOX/apt/sources.list")" if [ -f "$BAK" ]; then ok_t "classic: original file was backed up before editing" else @@ -230,6 +231,19 @@ else "backup does not contain the original http:// line" fi +# Distinct paths must get distinct backup names. Flattening "/" to "_" is +# not injective -- /a/b/c and /a_b/c both flattened to _a_b_c -- and a +# collision would make --apt-https-off restore one file with another's +# contents. +BAK_A="$(_apt_https_backup_path '/a/b/c')" +BAK_B="$(_apt_https_backup_path '/a_b/c')" +if [ "$BAK_A" != "$BAK_B" ]; then + ok_t "backup names do not collide for paths that flatten alike" +else + fail_t "backup names do not collide for paths that flatten alike" \ + "both mapped to $BAK_A" +fi + # Second pass must be a no-op: idempotency is what keeps re-runs cheap and # keeps AUTO_MODE from churning backups. _apt_https_rewrite_apt_sources @@ -1874,6 +1888,73 @@ else fail_t "no CRLF line endings (matches .gitattributes eol=lf)" "found in:$CRLF_FILES" fi +# ============================================================================ +# The detected family must be computed once per pass +# ============================================================================ +# _apt_https_source_applicable used to call apt_https_family itself, and the +# report loop hits it once per store. That is up to five `command -v` PATH +# scans per RPM/Arch store -- wasted on every report -- and a latent +# correctness hazard: if PATH changed mid-run, a store could be "applicable" +# in one check and absent from the next, so the same file could be reported +# once or twice. Callers now pass the family in. +MARK="$WD/marker"; : > "$MARK" +cat > "$WD/famcount.sh" </dev/null 2>&1 +apt_https_family() { echo x >> "$MARK"; printf 'apt'; } +_apt_https_source_applicable zypper zypper >/dev/null 2>&1 +printf '%s\n' "\$(wc -l < "$MARK" 2>/dev/null || echo 0)" +EOF +HITS="$(bash "$WD/famcount.sh" 2>/dev/null | tr -d ' ' | tail -1)" +if [ "${HITS:-x}" = "0" ]; then + ok_t "applicability check does not re-probe the family when given a hint" +else + fail_t "applicability check does not re-probe the family when given a hint" \ + "$HITS extra probe(s)" +fi + +: > "$MARK" +cat > "$WD/famcount2.sh" </dev/null 2>&1 +apt_https_family() { echo x >> "$MARK"; printf 'apt'; } +apt_https_status_text >/dev/null 2>&1 +printf '%s\n' "\$(wc -l < "$MARK" 2>/dev/null || echo 0)" +EOF +HITS="$(bash "$WD/famcount2.sh" 2>/dev/null | tr -d ' ' | tail -1)" +if [ "${HITS:-99}" = "1" ]; then + ok_t "apt_https_status_text probes the family exactly once (not once per store)" +else + fail_t "apt_https_status_text probes the family exactly once" \ + "$HITS probe(s) across 16 stores" +fi + +# ============================================================================ +# Backup names must not collide +# ============================================================================ +COLL_A="$(_apt_https_backup_path '/a/b/c')" +COLL_B="$(_apt_https_backup_path '/a_b/c')" +COLL_C="$(_apt_https_backup_path '/a/b/c.d')" +if [ "$COLL_A" != "$COLL_B" ] && [ "$COLL_A" != "$COLL_C" ]; then + ok_t "flattened backup names stay distinct (checksum suffix)" +else + fail_t "flattened backup names stay distinct (checksum suffix)" \ + "collision: $COLL_A" +fi +# The name must still be a single legal path component. +BASE="$(basename "$COLL_A")" +case "$BASE" in + */*) fail_t "backup name is a single path component" "contains a slash" ;; + *) ok_t "backup name is a single path component" ;; +esac +# And it must be stable across calls, so a re-run finds the existing backup. +if [ "$(_apt_https_backup_path '/etc/apt/sources.list')" = \ + "$(_apt_https_backup_path '/etc/apt/sources.list')" ]; then + ok_t "backup name is deterministic across calls" +else + fail_t "backup name is deterministic across calls" "name changed between calls" +fi + # ============================================================================ # Summary # ============================================================================ From 06aa025fbbb70e02656c0e60bdda49e8de8e5ad0 Mon Sep 17 00:00:00 2001 From: neohiro Date: Sun, 4 Oct 2026 12:39:13 +0200 Subject: [PATCH 07/17] fix: guarantee a dropped SSH session is always recoverable The tmux wrapper existed to make an interrupted install recoverable. It did not actually deliver that, in three separate ways, and the worst of them is exactly the "booted out with no way back" failure it was supposed to prevent. 1. A stale session was silently hijacked. `tmux new-session -A` attaches when the session already exists. A `linux-setup` left behind by a run that did not exit cleanly therefore meant the new run never started: the user was dropped into an old session, possibly running different flags, with no indication their run had not begun and no way back to it. Now the script checks for the session first, warns that it is a leftover, and starts `linux-setup-` instead. The inner wrapper -- which previously existed as dead code, built and never passed to tmux -- is now actually used and tears the session down on clean exit, so leftovers stop accumulating in the first place. 2. Every command-line flag was dropped by the re-exec. The wrap re-executed `bash "$SCRIPT_PATH"` with no `"$@"`. `--auto`, `--step`, `--dry-run` and the rest vanished, so wrapping silently changed what the run did. Arguments are now threaded through and shell-quoted, so `--note='a b; rm -rf /'` cannot be re-interpreted by tmux's shell. 3. The reattach command was never shown. `exec` replaces the process, so nothing after the wrap could print. If you detached or lost the socket you had to remember `tmux attach -t linux-setup` from the README. The command is now printed before the exec, naming the session this run actually created -- the PID-suffixed one when the standard name was taken. Script hopping (DeepClean.sh / OptimizeLinuxASR.sh via run_remote_script) had two gaps. The wait loop printed nothing, so a long step looked frozen and invited exactly the Ctrl-C that orphans the work: the poll dies, the detached subscript keeps running, and the user is left staring at a stalled run unable to tell if it is alive. It now emits a heartbeat every 30s with the reattach command, a completion line, and how to detach without stopping. And when SSH + interactive has no tmux to protect the hop, it falls through silently -- now it states plainly that the step cannot be recovered and suggests a local terminal. Also fixed two defects in the degraded (library-not-loaded) mode added in the previous pass. Its warning stood in for a hot-path helper called by every pkg_*/update_* entry point, so a single run reprinted four lines a dozen times and buried the installer output; it is now latched to once per process. And apt_https_status_text returns empty there for "found nothing", which _auto_skip_if_done read as "already HTTPS-only" -- the tool claiming a verification it never performed. Callers now consult a new apt_https_available(), which reports 0 for the real guard and 1 for the stub. One test bug worth noting: an env-transport case reused the variable name SRC for the store id, shadowing the path to linuxinstall.sh and silently breaking every later assertion that grepped it. Renamed, and the symptom is recorded here because three unrelated assertions failing at once with no apparent connection is exactly the kind of thing that gets misdiagnosed as a product bug. tests/test_apt_https.sh 261 -> 275. The continuity properties are asserted against the real ensure_tmux_if_ssh driven with a fake tmux and a captured exec: arguments survive the re-exec, -A is never used, a stale session forces a distinct name and is reported, the printed reattach command matches the session actually created, metacharacters are escaped, a bare invocation emits no empty argument, and the staged reaper really does kill the session on clean exit. Script-hop warnings and the heartbeat are pinned too. Full suite: 706 passed, 0 failures. bash -n clean on every .sh. --- README.md | 32 ++++- lib/apt-https.sh | 30 ++++- linuxinstall.sh | 129 +++++++++++++++---- tests/test_apt_https.sh | 270 +++++++++++++++++++++++++++++++++++++++- 4 files changed, 434 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index ad352e4..dd9c803 100644 --- a/README.md +++ b/README.md @@ -53,6 +53,7 @@ the package name right per distro. This script: | Fail2ban, sysctl profile, AppArmor/SELinux check | per-distro package names | | Tor, dnscrypt-proxy, unattended-upgrades, DeepClean | optional per profile | | **HTTPS-only package transport** | enforced before every `apt`/`dnf`/`yum`/`zypper`/`pacman`/`apk`/`pip`/`npm`/… download — `--apt-https` | +| **Disconnect-safe** | SSH runs auto-wrap in tmux; subscripts get their own session; the reattach command is always printed | | **Rollback log** | `/var/log/linux-install-rollback.log` — `original\tbackup` per file | | **SSH self-heal** | `--install-self-heal` — systemd timer or cron, every 60s | | **21-tool maintenance suite** | Re-runs any step, lists keys, tails logs, dumps config | @@ -402,6 +403,35 @@ automatically and there's nothing to re-attach to. When the script finishes successfully, the tmux session closes itself; if it fails, the session is left intact for inspection. +### You can always get back to a running install + +This is the property that matters when something scary happens mid-run, so +it is enforced rather than hoped for: + +- **The reattach command is printed before the wrap, not after.** The script + `exec`s into tmux, so anything printed afterwards would never be seen. You + get the exact command — including a PID-suffixed name if the standard one + is taken — on screen at the moment you need it. +- **A leftover session is never hijacked.** If a `linux-setup` session + already exists (a previous run that did not exit cleanly), the script says + so and starts `linux-setup-` instead. You are never silently dropped + into an old, differently-flagged run while yours never starts. +- **Your flags survive the wrap.** The re-exec carries `--auto`, `--step`, + `--dry-run` and friends through, shell-quoted, so wrapping cannot change + what the run does. +- **Sessions are reaped on success.** A clean exit tears the session down, so + the next run starts clean. A failed run leaves it alive — that is your + inspection point. +- **Script hops get their own session.** `DeepClean.sh` / + `OptimizeLinuxASR.sh` run in a dedicated `neohiro-sub--` tmux + session with the reattach command printed, plus a heartbeat every 30s so a + long step does not look frozen (a frozen screen invites a Ctrl-C that + orphans the work). If there is no tmux to protect the hop, the script says + plainly that the step cannot be recovered. + +`tmux ls` lists sessions; `tmux attach -t ` reattaches; `Ctrl-b` then +`d` detaches without stopping anything. + ## Reconnecting after a reboot or lockout **Tailscale SSH bypasses OpenSSH settings** — it authenticates via the @@ -761,7 +791,7 @@ dumps — add to `/etc/security/limits.conf`: ```bash bash tests/run-all.sh # every suite, with a summary bash tests/test_linuxinstall.sh # 67 tests: parse, logic, UX coverage, snapshot -bash tests/test_apt_https.sh # 253 tests: transport guard + encoding hygiene +bash tests/test_apt_https.sh # 275 tests: transport guard, run continuity, encoding bash tests/test_updater.sh # 45 tests: dispatcher, race safety, version floor shellcheck -S warning *.sh lib/*.sh tests/*.sh # lint ``` diff --git a/lib/apt-https.sh b/lib/apt-https.sh index e3f8f80..36ddbd3 100644 --- a/lib/apt-https.sh +++ b/lib/apt-https.sh @@ -314,16 +314,31 @@ _apt_https_backup_path() { } # _apt_https_backup_once — copy aside the first time we touch it. +# +# The copy is staged inside the backup directory and renamed into place, for +# the same reason the rewrite is: `cp` into a destination that does not exist +# yet leaves a truncated `.orig` if it dies partway. A truncated backup is +# worse than no backup, because `--apt-https-off` would then confidently +# restore a broken file. rename(2) means the backup is either absent or +# complete. _apt_https_backup_once() { - local src="$1" dst + local src="$1" dst stage [ -f "$src" ] || return 0 dst="$(_apt_https_backup_path "$src")" [ -f "$dst" ] && return 0 _apt_priv mkdir -p "$(apt_https_backup_dir)" || return 1 - if ! _apt_priv cp -p "$src" "$dst"; then + stage="${dst}.partial.$$" + _apt_priv rm -f "$stage" 2>/dev/null || true + if ! _apt_priv cp -p "$src" "$stage"; then + _apt_priv rm -f "$stage" 2>/dev/null || true warn "Could not back up $src — refusing to edit it." return 1 fi + if ! _apt_priv mv -f "$stage" "$dst"; then + _apt_priv rm -f "$stage" 2>/dev/null || true + warn "Could not store the backup for $src — refusing to edit it." + return 1 + fi # Hook into the host's rollback log when one exists so `linuxinstall.sh # --rollback` can undo this too. if declare -F record_backup >/dev/null 2>&1; then @@ -384,6 +399,17 @@ _apt_https_write_conf() { return 0 } +# apt_https_available — is the real guard loaded? +# +# Callers that infer "the repositories are fine" from an empty +# apt_https_status_text must check this first. When the library cannot be +# loaded, a stub takes its place and status_text returns empty for "found +# nothing" -- indistinguishable from "checked, all https". Reporting that as +# verified would be an unverified security claim printed as fact. +apt_https_available() { + return 0 +} + # _apt_https_stage — private scratch path for generated files. _apt_https_stage() { if declare -F _tmpfile >/dev/null 2>&1; then diff --git a/linuxinstall.sh b/linuxinstall.sh index 791bfe7..bc8324a 100644 --- a/linuxinstall.sh +++ b/linuxinstall.sh @@ -276,7 +276,13 @@ if ! declare -F apt_https_guard >/dev/null 2>&1; then fi if ! declare -F apt_https_guard >/dev/null 2>&1; then + # Latch the warning to once per process. This stub stands in for a hot-path + # helper called by every pkg_* / update_* entry point, so without a latch + # a single run printed the same four lines a dozen times and buried the + # actual installer output. _apt_https_unavailable() { + [ -n "${_APT_HTTPS_DEGRADED_WARNED:-}" ] && return 0 + _APT_HTTPS_DEGRADED_WARNED=1 printf '%s\n' "[WARNING] lib/apt-https.sh could not be loaded, so the repository" >&2 printf '%s\n' "[WARNING] transport guard is INACTIVE for this run: packages may" >&2 printf '%s\n' "[WARNING] still be fetched over plaintext HTTP. Run from a clone of" >&2 @@ -290,6 +296,9 @@ if ! declare -F apt_https_guard >/dev/null 2>&1; then apt_https_status_text(){ return 0; } apt_https_backup_dir() { printf '%s' "${NEOHIRO_APT_BACKUP_DIR:-/var/backups/neohiro-apt-https}"; } apt_https_conf_file() { printf '%s' "${NEOHIRO_APT_ETC_DIR:-/etc}/apt/apt.conf.d/99neohiro-force-https"; } + # Reported so no caller can mistake "the guard found nothing" for + # "the guard checked and the repos are fine". See _auto_skip_if_done. + apt_https_available() { return 1; } fi unset _apt_https_resolved _apt_https_dir _ah_cand 2>/dev/null || true fi @@ -485,6 +494,24 @@ print_recovery_if_ssh() { print_recovery_cmd } +# Wrap an SSH run in a tmux session so a dropped socket cannot abort it. +# +# Three things this must get right, each of which was a bug: +# +# 1. Arguments must survive the re-exec. `bash "$SCRIPT_PATH"` with no +# "$@" silently dropped --auto / --step / --dry-run, so the run behaved +# differently after wrapping than the user asked for. +# 2. A stale session must never be hijacked. `tmux new-session -A` attaches +# to an existing session if one exists, which means the new run never +# starts and the user is dropped into someone else's (possibly older, +# differently-flagged) run with no way back to theirs. A leftover +# session from a previous run is exactly how this happens, so the +# session is torn down on clean exit and a name collision falls back to +# a PID-suffixed name instead of attaching. +# 3. The recovery command must be on screen BEFORE exec, because exec +# replaces this process and nothing after it can print. If the user +# detaches or the socket drops, "tmux attach -t " is the only way +# back and they must not have to remember it. ensure_tmux_if_ssh() { [ -n "${SSH_CONNECTION:-}${SSH_TTY:-}" ] || return 0 [ -z "${TMUX:-}" ] && [ -z "${STY:-}" ] || return 0 @@ -494,28 +521,63 @@ ensure_tmux_if_ssh() { fi fi command -v tmux >/dev/null 2>&1 || { - warn "tmux unavailable; SSH disconnect may kill the run. ${RECOVERY_CMD} will NOT exist - run the script from a local terminal instead." + warn "tmux unavailable; an SSH disconnect WILL kill this run and there is no way to reattach." + warn "Recovery: ${RECOVERY_CMD} will NOT exist -- run the script from a local terminal instead." return 0 } [ -n "$SCRIPT_PATH" ] || { warn "SCRIPT_PATH is empty; cannot re-exec inside tmux."; return 0; } - bold "SSH session detected. Wrapping this run in a tmux session so disconnects do not abort it." - # Quote everything via env+args (no string interpolation) so paths with spaces - # or shell metacharacters survive. The inner bash re-execs the same script - # by absolute path; on clean exit it tears the tmux session down. - local inner - inner=$(cat <<'INNER_EOF' -trap 'tmux kill-session -t linux-setup 2>/dev/null' EXIT -cd "$1" && shift -bash "$1" "$@" + + # Never attach to a pre-existing session: pick a free name instead. + local _sess="linux-setup" + if tmux has-session -t "$_sess" 2>/dev/null; then + warn "A tmux session named '$_sess' already exists (a previous run that did not exit cleanly)." + warn "Starting this run as '$_sess-$$' instead so it is never mixed up with the old one." + _sess="linux-setup-$$" + fi + + bold "SSH session detected. Wrapping this run in tmux so disconnects do not abort it." + # Shell-quote every argument so paths with spaces or metacharacters survive. + # printf '%q ' with zero arguments emits nothing, so a flagless run stays clean. + local _args="" _a + for _a in "$@"; do _args="$_args$(printf ' %q' "$_a")"; done + + # Print the recovery command BEFORE exec: after exec nothing in this process + # can print, and this is the only moment the user is guaranteed to see it. + printf '\n' + ok "This run is now inside tmux. If you detach or lose the SSH socket, resume with:" + bold " tmux attach -t $_sess" + printf ' (detach without stopping: Ctrl-b then d. list sessions: tmux ls)\n\n' + + # Inner wrapper: tears the session down on a clean exit so the next run is + # not poisoned by a leftover session. On failure the session is deliberately + # left alive -- that is the recovery path. + # Argument order: