feat(clp): speed up logtype-insights with a one-command bootstrap and model2vec clustering

CONTRIBUTING.md

@@ -26,7 +26,7 @@ plugins/clp/.codex-plugin/plugin.json
 plugins/clp/bin/
     Restricted-passthrough bash wrappers (clp-s-*) and shared lib/clp-common.sh,
     plus local helpers that are not clp-s passthroughs (structurize.py,
-    logtype-cache).
+    logtype-cache, logtype-insights-bootstrap, logtype-cluster).
 plugins/clp/skills-claude/
     Claude Code skills: compress, compress-folder, search, logtype-insights,
     decompress, claude-code-trajectory, clpp-compress, clpp-search, dev,

LOCAL_TESTING.md

@@ -47,16 +47,21 @@ scripts/validate-codex-plugin.sh ./plugins/clp
 Check shell wrapper syntax and style:
 
 ```bash
-for f in plugins/clp/bin/clp-s-*; do
+for f in plugins/clp/bin/clp-s-* plugins/clp/bin/logtype-cache \
+         plugins/clp/bin/logtype-insights-bootstrap \
+         plugins/clp/bin/logtype-cluster; do
   bash -n "$f"
 done
+python3 -m py_compile plugins/clp/bin/logtype-cluster.py
 
 shellcheck \
   plugins/clp/bin/clp-s-list-sessions \
   plugins/clp/bin/clp-s-compress-session \
   plugins/clp/bin/clp-s-compress-folder \
   plugins/clp/bin/clp-s-search-kql \
   plugins/clp/bin/clp-s-decompress \
+  plugins/clp/bin/logtype-insights-bootstrap \
+  plugins/clp/bin/logtype-cluster \
   plugins/clp/bin/lib/clp-common.sh
 ```
 
@@ -190,6 +195,25 @@ jq -s '{schema:{message:"message"},
 "$LC" list
 ```
 
+Exercise the `logtype-insights` helper scripts. The bootstrap wraps the
+schema sample, the dictionary dump, and the cache probe in one command; on
+clp-core 0.13+ one call suffices, on older builds it prints
+`FALLBACK=TEMPLATIZE_NEEDS_MESSAGE` — re-run adding `--message message`:
+
+```bash
+./plugins/clp/bin/logtype-insights-bootstrap \
+  --cache-dir /tmp/smoke-lt-cache --out-dir /tmp/smoke-bootstrap "$ARCHIVE"
+# Expect DIST lines, LOGTYPE_COUNT>0, CACHE_MODE=UPTODATE (cache primed above).
+
+# Clusterer: one-time setup (network — installs model2vec into a plugin venv
+# and downloads the embedding model), then cluster the baseline:
+./plugins/clp/bin/logtype-cluster setup
+./plugins/clp/bin/logtype-cluster cluster \
+  --input /tmp/smoke-bootstrap/logtypes.ndjson
+# Expect CLUSTERS<=TEMPLATES and one {"id","count","representative"} line per
+# cluster; /tmp/logtype-clusters.json holds the memberships for `expand`.
+```
+
 Note that the message field is a CLP-string: `message:term` returns 0 by
 design. Match message content by projecting the field and grepping it, and use
 the scalar fields for KQL:

README.md

@@ -37,7 +37,8 @@ plugins/clp/.codex-plugin/plugin.json
     Codex plugin manifest.
 plugins/clp/bin/
     Restricted-passthrough bash wrappers for clp-s, plus local helpers
-    (structurize.py, logtype-cache) that are not clp-s passthroughs.
+    (structurize.py, logtype-cache, logtype-insights-bootstrap,
+    logtype-cluster) that are not clp-s passthroughs.
 plugins/clp/skills-claude/
     Claude Code skills: compress, compress-folder, search, logtype-insights,
     decompress, claude-code-trajectory.

plugins/clp/README.md

@@ -79,13 +79,22 @@ boundary (flag allowlist, path validation, env hardening):
 The wrappers prefer `CLP_S_BIN`, then plugin-local `bin/clp-s`, then
 plugin-local `.clp-core/bin/clp-s`, then `clp-s` on `PATH`.
 
-Local helpers (not `clp-s` passthroughs — they never invoke the binary):
+Local helpers (not `clp-s` passthroughs — they invoke `clp-s` only through
+the wrappers above, or not at all):
 
 - `bin/structurize.py` — converts unstructured text logs to structured JSONL.
   Used by `clp-s-compress-folder --structurize`; not called directly.
 - `bin/logtype-cache` — persistent cache of the `logtype-insights`
   classification, with incremental update when an archive grows. See
   [Logtype Cache](#logtype-cache).
+- `bin/logtype-insights-bootstrap` — one-command bootstrap for the
+  `logtype-insights` skill: schema-discovery sample, per-field value
+  distributions, logtype dictionary dump (with the templatize fallback for
+  binaries that predate the shapes API), and the classification-cache probe,
+  summarized as grep-able `KEY=VALUE` lines.
+- `bin/logtype-cluster` (+ `logtype-cluster.py`) — groups semantically similar
+  logtypes with model2vec static embeddings so the LLM classifies one
+  representative per cluster. See [Logtype Cluster](#logtype-cluster).
 
 ## Session Workflow
 
@@ -287,6 +296,19 @@ The skill is app-agnostic — it discovers the schema (timestamp/severity/logger
 message field names) from a sample record, so it works on structurized text
 archives and native-JSON archives alike.
 
+The skill's mechanical preamble is packaged as one command:
+
+```bash
+./plugins/clp/bin/logtype-insights-bootstrap /tmp/archive
+```
+
+It samples records for schema discovery, prints per-field value
+distributions, dumps + normalizes the dictionary (falling back to
+templatization on binaries that predate the shapes API — re-run with
+`--message <field>` when it asks), probes the classification cache, and prints
+a grep-able `KEY=VALUE` summary (`LOGTYPE_COUNT=`, `FALLBACK=`, `CACHE_MODE=`,
+`TO_CLASSIFY=`, output-file paths).
+
 Note that the message field is stored as a CLP-string, so KQL **cannot** match
 message content: `message:term` and `message:*term*` always return 0. Retrieve
 message content by projecting the field and grepping it; the scalar fields
@@ -338,6 +360,36 @@ that read or write the cache (`diff`, `get`, `put`, `put-merged`, `list`,
 `show`). `normalize`, `count`, and `key` only transform/hash the input and do
 not accept it.
 
+### Logtype Cluster
+
+Classification cost scales with the number of templates the LLM must label.
+`bin/logtype-cluster` shrinks that: it embeds the to-classify logtypes with a
+lightweight model2vec static model (numpy-only, no torch) and greedily groups
+them at a cosine-similarity threshold, so the LLM classifies one
+representative per cluster (by cluster id) and `expand` propagates the
+category to every member mechanically — byte-exact, because the LLM never
+echoes logtype strings.
+
+```bash
+LTC=./plugins/clp/bin/logtype-cluster
+"$LTC" setup                                   # one-time: venv + model2vec + model download
+"$LTC" cluster --input /tmp/logtypes-to-classify.ndjson
+"$LTC" expand --clusters /tmp/logtype-clusters.json \
+  --classification /tmp/logtype-class.json     # id-based assignments from the LLM
+```
+
+- `setup` creates a venv at `~/.config/yscope-clp-plugin/venvs/logtype-cluster`
+  and pins the HuggingFace model cache to
+  `~/.config/yscope-clp-plugin/huggingface` (unless `HF_HOME` is already set).
+  Needs network once; afterwards `cluster` runs offline.
+- Model: `minishlab/potion-base-8M` (override with `--model` or
+  `$CLP_LOG_CLUSTER_MODEL`). Threshold: cosine 0.80 (override with
+  `--threshold` or `$CLP_LOG_CLUSTER_THRESHOLD`; raise to 0.85–0.90 to split
+  more, lower to merge more).
+- `expand` is stdlib-only (no venv needed) and validates that every cluster id
+  is assigned exactly once before writing anything (exit 2 otherwise), which
+  protects the logtype cache from partial classifications.
+
 ## Query Starters
 
 For session-log analysis (which tools fired, what failed, how long a turn

plugins/clp/bin/logtype-cluster

@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# logtype-cluster — launcher for the logtype-insights pre-clustering tool.
+#
+# `setup` creates a dedicated venv under the plugin config dir and installs
+# model2vec (the lightweight static-embedding model used to merge semantically
+# similar logtype templates before LLM classification), then pre-downloads the
+# embedding model so later runs work offline. Every other subcommand is
+# forwarded to logtype-cluster.py, preferring the venv python when it exists.
+
+CLP_PLUGIN_BIN_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)"
+# shellcheck disable=SC1091
+source "${CLP_PLUGIN_BIN_DIR}/lib/clp-common.sh"
+
+VENV_DIR="$(clp_config_dir)/venvs/logtype-cluster"
+DEFAULT_MODEL="${CLP_LOG_CLUSTER_MODEL:-minishlab/potion-base-8M}"
+# Keep the embedding-model download under the plugin config dir instead of the
+# user-wide HF cache, unless the user already pinned HF_HOME themselves.
+export HF_HOME="${HF_HOME:-$(clp_config_dir)/huggingface}"
+
+if [[ "${1:-}" == "setup" ]]; then
+  echo "Creating venv: $VENV_DIR"
+  python3 -m venv "$VENV_DIR"
+  "$VENV_DIR/bin/pip" install --quiet --disable-pip-version-check "model2vec>=0.4,<1"
+  echo "Pre-downloading embedding model: $DEFAULT_MODEL (HF_HOME=$HF_HOME)"
+  "$VENV_DIR/bin/python" - "$DEFAULT_MODEL" <<'PY'
+import sys
+from model2vec import StaticModel
+model = StaticModel.from_pretrained(sys.argv[1])
+dims = model.encode(["setup probe"]).shape
+print(f"model ready: {sys.argv[1]} (embedding shape {dims})")
+PY
+  echo "Setup complete. 'logtype-cluster cluster' now works offline."
+  exit 0
+fi
+
+PY_BIN="$VENV_DIR/bin/python"
+[[ -x "$PY_BIN" ]] || PY_BIN="$(command -v python3)"
+export LOGTYPE_CLUSTER_SETUP_CMD="${CLP_PLUGIN_BIN_DIR}/logtype-cluster setup"
+exec "$PY_BIN" "${CLP_PLUGIN_BIN_DIR}/logtype-cluster.py" "$@"