From 7b55444a10120449cac08a4e6d7d2f0d54c033e7 Mon Sep 17 00:00:00 2001 From: Robin Li Date: Mon, 14 Sep 2026 15:44:00 -0400 Subject: [PATCH] feat(clp): speed up logtype-insights with a one-command bootstrap and model2vec clustering MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cuts the skill's two dominant costs — kick-start latency and classification wall time: - bin/logtype-insights-bootstrap: one command for schema sampling, per-field value distributions, the logtype dictionary dump, and the classification-cache probe (KEY=VALUE summary). Handles the templatize fallback for pre-shapes-API binaries, converting the frequency table to the canonical NDJSON so the cache probe works there too. SIGPIPE-safe on high-distinct fields. - bin/logtype-cluster (setup/cluster/expand): model2vec (potion-base-8M) pre-clustering so the classifier labels one representative per cluster by id; expand re-attaches member logtypes byte-exact from the cluster file, which keeps GROWTH cache merging safe by construction. - Classification runs in ONE haiku-default subagent (sonnet retry on validation failure) instead of pasting every template into sonnet. - skills-claude/logtype-insights: 621-line monolith split into a thin SKILL.md plus on-demand references (logtype-baseline, logtype-classify, logtype-insight). skills-codex stays a monolith per convention, trimmed and using the same scripts. Both keep the mandatory scoped semantic cross-check from #5. - Both variants now narrate every step to the user, including the fallback and retry branches. - Docs: helper lists (README, CONTRIBUTING), LOCAL_TESTING smoke block, release-testing walkthrough, dev preflight. --- CONTRIBUTING.md | 2 +- LOCAL_TESTING.md | 26 +- README.md | 3 +- plugins/clp/README.md | 54 +- plugins/clp/bin/logtype-cluster | 41 + plugins/clp/bin/logtype-cluster.py | 277 +++++++ plugins/clp/bin/logtype-insights-bootstrap | 204 +++++ plugins/clp/skills-claude/dev/SKILL.md | 7 +- .../skills-claude/logtype-insights/SKILL.md | 699 +++--------------- .../references/logtype-baseline.md | 132 ++++ .../references/logtype-classify.md | 184 +++++ .../references/logtype-insight.md | 131 ++++ .../skills-codex/logtype-insights/SKILL.md | 529 +++++-------- release-testing/TESTING.md | 25 +- 14 files changed, 1390 insertions(+), 924 deletions(-) create mode 100755 plugins/clp/bin/logtype-cluster create mode 100755 plugins/clp/bin/logtype-cluster.py create mode 100755 plugins/clp/bin/logtype-insights-bootstrap create mode 100644 plugins/clp/skills-claude/references/logtype-baseline.md create mode 100644 plugins/clp/skills-claude/references/logtype-classify.md create mode 100644 plugins/clp/skills-claude/references/logtype-insight.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 097345b..7ea569b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -26,7 +26,7 @@ plugins/clp/.codex-plugin/plugin.json plugins/clp/bin/ Restricted-passthrough bash wrappers (clp-s-*) and shared lib/clp-common.sh, plus local helpers that are not clp-s passthroughs (structurize.py, - logtype-cache). + logtype-cache, logtype-insights-bootstrap, logtype-cluster). plugins/clp/skills-claude/ Claude Code skills: compress, compress-folder, search, logtype-insights, decompress, claude-code-trajectory, clpp-compress, clpp-search, dev, diff --git a/LOCAL_TESTING.md b/LOCAL_TESTING.md index 6fc2f33..2fba700 100644 --- a/LOCAL_TESTING.md +++ b/LOCAL_TESTING.md @@ -47,9 +47,12 @@ scripts/validate-codex-plugin.sh ./plugins/clp Check shell wrapper syntax and style: ```bash -for f in plugins/clp/bin/clp-s-*; do +for f in plugins/clp/bin/clp-s-* plugins/clp/bin/logtype-cache \ + plugins/clp/bin/logtype-insights-bootstrap \ + plugins/clp/bin/logtype-cluster; do bash -n "$f" done +python3 -m py_compile plugins/clp/bin/logtype-cluster.py shellcheck \ plugins/clp/bin/clp-s-list-sessions \ @@ -57,6 +60,8 @@ shellcheck \ plugins/clp/bin/clp-s-compress-folder \ plugins/clp/bin/clp-s-search-kql \ plugins/clp/bin/clp-s-decompress \ + plugins/clp/bin/logtype-insights-bootstrap \ + plugins/clp/bin/logtype-cluster \ plugins/clp/bin/lib/clp-common.sh ``` @@ -190,6 +195,25 @@ jq -s '{schema:{message:"message"}, "$LC" list ``` +Exercise the `logtype-insights` helper scripts. The bootstrap wraps the +schema sample, the dictionary dump, and the cache probe in one command; on +clp-core 0.13+ one call suffices, on older builds it prints +`FALLBACK=TEMPLATIZE_NEEDS_MESSAGE` — re-run adding `--message message`: + +```bash +./plugins/clp/bin/logtype-insights-bootstrap \ + --cache-dir /tmp/smoke-lt-cache --out-dir /tmp/smoke-bootstrap "$ARCHIVE" +# Expect DIST lines, LOGTYPE_COUNT>0, CACHE_MODE=UPTODATE (cache primed above). + +# Clusterer: one-time setup (network — installs model2vec into a plugin venv +# and downloads the embedding model), then cluster the baseline: +./plugins/clp/bin/logtype-cluster setup +./plugins/clp/bin/logtype-cluster cluster \ + --input /tmp/smoke-bootstrap/logtypes.ndjson +# Expect CLUSTERS<=TEMPLATES and one {"id","count","representative"} line per +# cluster; /tmp/logtype-clusters.json holds the memberships for `expand`. +``` + Note that the message field is a CLP-string: `message:term` returns 0 by design. Match message content by projecting the field and grepping it, and use the scalar fields for KQL: diff --git a/README.md b/README.md index a54e25e..2b9d049 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,8 @@ plugins/clp/.codex-plugin/plugin.json Codex plugin manifest. plugins/clp/bin/ Restricted-passthrough bash wrappers for clp-s, plus local helpers - (structurize.py, logtype-cache) that are not clp-s passthroughs. + (structurize.py, logtype-cache, logtype-insights-bootstrap, + logtype-cluster) that are not clp-s passthroughs. plugins/clp/skills-claude/ Claude Code skills: compress, compress-folder, search, logtype-insights, decompress, claude-code-trajectory. diff --git a/plugins/clp/README.md b/plugins/clp/README.md index 23320b4..35ba22d 100644 --- a/plugins/clp/README.md +++ b/plugins/clp/README.md @@ -79,13 +79,22 @@ boundary (flag allowlist, path validation, env hardening): The wrappers prefer `CLP_S_BIN`, then plugin-local `bin/clp-s`, then plugin-local `.clp-core/bin/clp-s`, then `clp-s` on `PATH`. -Local helpers (not `clp-s` passthroughs — they never invoke the binary): +Local helpers (not `clp-s` passthroughs — they invoke `clp-s` only through +the wrappers above, or not at all): - `bin/structurize.py` — converts unstructured text logs to structured JSONL. Used by `clp-s-compress-folder --structurize`; not called directly. - `bin/logtype-cache` — persistent cache of the `logtype-insights` classification, with incremental update when an archive grows. See [Logtype Cache](#logtype-cache). +- `bin/logtype-insights-bootstrap` — one-command bootstrap for the + `logtype-insights` skill: schema-discovery sample, per-field value + distributions, logtype dictionary dump (with the templatize fallback for + binaries that predate the shapes API), and the classification-cache probe, + summarized as grep-able `KEY=VALUE` lines. +- `bin/logtype-cluster` (+ `logtype-cluster.py`) — groups semantically similar + logtypes with model2vec static embeddings so the LLM classifies one + representative per cluster. See [Logtype Cluster](#logtype-cluster). ## Session Workflow @@ -287,6 +296,19 @@ The skill is app-agnostic — it discovers the schema (timestamp/severity/logger message field names) from a sample record, so it works on structurized text archives and native-JSON archives alike. +The skill's mechanical preamble is packaged as one command: + +```bash +./plugins/clp/bin/logtype-insights-bootstrap /tmp/archive +``` + +It samples records for schema discovery, prints per-field value +distributions, dumps + normalizes the dictionary (falling back to +templatization on binaries that predate the shapes API — re-run with +`--message ` when it asks), probes the classification cache, and prints +a grep-able `KEY=VALUE` summary (`LOGTYPE_COUNT=`, `FALLBACK=`, `CACHE_MODE=`, +`TO_CLASSIFY=`, output-file paths). + Note that the message field is stored as a CLP-string, so KQL **cannot** match message content: `message:term` and `message:*term*` always return 0. Retrieve message content by projecting the field and grepping it; the scalar fields @@ -338,6 +360,36 @@ that read or write the cache (`diff`, `get`, `put`, `put-merged`, `list`, `show`). `normalize`, `count`, and `key` only transform/hash the input and do not accept it. +### Logtype Cluster + +Classification cost scales with the number of templates the LLM must label. +`bin/logtype-cluster` shrinks that: it embeds the to-classify logtypes with a +lightweight model2vec static model (numpy-only, no torch) and greedily groups +them at a cosine-similarity threshold, so the LLM classifies one +representative per cluster (by cluster id) and `expand` propagates the +category to every member mechanically — byte-exact, because the LLM never +echoes logtype strings. + +```bash +LTC=./plugins/clp/bin/logtype-cluster +"$LTC" setup # one-time: venv + model2vec + model download +"$LTC" cluster --input /tmp/logtypes-to-classify.ndjson +"$LTC" expand --clusters /tmp/logtype-clusters.json \ + --classification /tmp/logtype-class.json # id-based assignments from the LLM +``` + +- `setup` creates a venv at `~/.config/yscope-clp-plugin/venvs/logtype-cluster` + and pins the HuggingFace model cache to + `~/.config/yscope-clp-plugin/huggingface` (unless `HF_HOME` is already set). + Needs network once; afterwards `cluster` runs offline. +- Model: `minishlab/potion-base-8M` (override with `--model` or + `$CLP_LOG_CLUSTER_MODEL`). Threshold: cosine 0.80 (override with + `--threshold` or `$CLP_LOG_CLUSTER_THRESHOLD`; raise to 0.85–0.90 to split + more, lower to merge more). +- `expand` is stdlib-only (no venv needed) and validates that every cluster id + is assigned exactly once before writing anything (exit 2 otherwise), which + protects the logtype cache from partial classifications. + ## Query Starters For session-log analysis (which tools fired, what failed, how long a turn diff --git a/plugins/clp/bin/logtype-cluster b/plugins/clp/bin/logtype-cluster new file mode 100755 index 0000000..dd46651 --- /dev/null +++ b/plugins/clp/bin/logtype-cluster @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +set -euo pipefail + +# logtype-cluster — launcher for the logtype-insights pre-clustering tool. +# +# `setup` creates a dedicated venv under the plugin config dir and installs +# model2vec (the lightweight static-embedding model used to merge semantically +# similar logtype templates before LLM classification), then pre-downloads the +# embedding model so later runs work offline. Every other subcommand is +# forwarded to logtype-cluster.py, preferring the venv python when it exists. + +CLP_PLUGIN_BIN_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" +# shellcheck disable=SC1091 +source "${CLP_PLUGIN_BIN_DIR}/lib/clp-common.sh" + +VENV_DIR="$(clp_config_dir)/venvs/logtype-cluster" +DEFAULT_MODEL="${CLP_LOG_CLUSTER_MODEL:-minishlab/potion-base-8M}" +# Keep the embedding-model download under the plugin config dir instead of the +# user-wide HF cache, unless the user already pinned HF_HOME themselves. +export HF_HOME="${HF_HOME:-$(clp_config_dir)/huggingface}" + +if [[ "${1:-}" == "setup" ]]; then + echo "Creating venv: $VENV_DIR" + python3 -m venv "$VENV_DIR" + "$VENV_DIR/bin/pip" install --quiet --disable-pip-version-check "model2vec>=0.4,<1" + echo "Pre-downloading embedding model: $DEFAULT_MODEL (HF_HOME=$HF_HOME)" + "$VENV_DIR/bin/python" - "$DEFAULT_MODEL" <<'PY' +import sys +from model2vec import StaticModel +model = StaticModel.from_pretrained(sys.argv[1]) +dims = model.encode(["setup probe"]).shape +print(f"model ready: {sys.argv[1]} (embedding shape {dims})") +PY + echo "Setup complete. 'logtype-cluster cluster' now works offline." + exit 0 +fi + +PY_BIN="$VENV_DIR/bin/python" +[[ -x "$PY_BIN" ]] || PY_BIN="$(command -v python3)" +export LOGTYPE_CLUSTER_SETUP_CMD="${CLP_PLUGIN_BIN_DIR}/logtype-cluster setup" +exec "$PY_BIN" "${CLP_PLUGIN_BIN_DIR}/logtype-cluster.py" "$@" diff --git a/plugins/clp/bin/logtype-cluster.py b/plugins/clp/bin/logtype-cluster.py new file mode 100755 index 0000000..3d99e8f --- /dev/null +++ b/plugins/clp/bin/logtype-cluster.py @@ -0,0 +1,277 @@ +#!/usr/bin/env python3 +"""logtype-cluster.py — merge semantically similar logtypes before classification. + +Invoked via the `logtype-cluster` bash launcher, which prefers the plugin venv +(created by `logtype-cluster setup`) and pins HF_HOME. Two subcommands: + + cluster Embed the input logtypes with a model2vec static model and group + them by greedy leader clustering at a cosine threshold. The LLM then + classifies only the cluster representatives, by cluster id. + expand Propagate the LLM's per-cluster category assignments to every member + logtype verbatim (stdlib-only; never re-generates logtype strings, + so cache GROWTH matching stays byte-exact by construction). + +Exit codes: 0 ok, 1 input problem, 2 missing dependency or validation failure, +3 usage error. +""" + +import argparse +import json +import os +import signal +import sys + +DEFAULT_MODEL = os.environ.get("CLP_LOG_CLUSTER_MODEL", "minishlab/potion-base-8M") +DEFAULT_THRESHOLD = os.environ.get("CLP_LOG_CLUSTER_THRESHOLD", "0.80") +SETUP_CMD = os.environ.get("LOGTYPE_CLUSTER_SETUP_CMD", "logtype-cluster setup") + + +class Parser(argparse.ArgumentParser): + def error(self, message): + self.print_usage(sys.stderr) + print(f"error: {message}", file=sys.stderr) + sys.exit(3) + + +def fail(code, *lines): + for line in lines: + print(f"error: {line}", file=sys.stderr) + sys.exit(code) + + +def load_logtypes(path): + """Read {"logtype": ...} NDJSON; return sorted unique logtype strings.""" + logtypes = set() + skipped = 0 + try: + with open(path, encoding="utf-8") as fh: + for line in fh: + line = line.strip() + if not line.startswith("{"): + continue + try: + obj = json.loads(line) + except json.JSONDecodeError: + skipped += 1 + continue + value = obj.get("logtype") if isinstance(obj, dict) else None + if isinstance(value, str) and value: + logtypes.add(value) + else: + skipped += 1 + except OSError as exc: + fail(1, f"cannot read --input file: {exc}") + if skipped: + print(f"warning: skipped {skipped} line(s) without a usable \"logtype\" field", + file=sys.stderr) + if not logtypes: + fail(1, f"no logtypes found in {path}") + return sorted(logtypes) + + +def cmd_cluster(args): + try: + threshold = float(args.threshold) + except ValueError: + fail(3, f"--threshold must be a number, got: {args.threshold}") + if not 0.0 < threshold <= 1.0: + fail(3, f"--threshold must be in (0, 1], got: {threshold}") + + templates = load_logtypes(args.input) + + try: + import numpy as np + from model2vec import StaticModel + except ImportError as exc: + fail(2, f"missing dependency ({exc.name or exc})", + f"run once to install it: {SETUP_CMD}") + + try: + model = StaticModel.from_pretrained(args.model) + except Exception as exc: # network/cache errors surface as various types + fail(2, f"could not load embedding model {args.model!r}: {exc}", + f"first use needs network access; run: {SETUP_CMD}") + + embeddings = np.asarray(model.encode(templates), dtype=np.float64) + norms = np.linalg.norm(embeddings, axis=1, keepdims=True) + norms[norms == 0] = 1.0 + embeddings = embeddings / norms + + # Greedy leader clustering against running centroids. Input order is the + # sorted template list, so cluster contents are deterministic. + sums, counts, members = [], [], [] + for i, vec in enumerate(embeddings): + if sums: + centroids = np.stack(sums) + centroid_norms = np.linalg.norm(centroids, axis=1) + centroid_norms[centroid_norms == 0] = 1.0 + sims = (centroids @ vec) / centroid_norms + best = int(np.argmax(sims)) + if sims[best] >= threshold: + sums[best] = sums[best] + vec + counts[best] += 1 + members[best].append(i) + continue + sums.append(vec.copy()) + counts.append(1) + members.append([i]) + + clusters = [] + for total, idxs in zip(sums, members): + centroid = total / np.linalg.norm(total) if np.linalg.norm(total) else total + sims = embeddings[idxs] @ centroid + rep = idxs[int(np.argmax(sims))] + clusters.append({ + "representative": templates[rep], + "members": [templates[i] for i in idxs], + "count": len(idxs), + }) + clusters.sort(key=lambda c: (-c["count"], c["representative"])) + for n, cluster in enumerate(clusters, 1): + cluster["id"] = f"c{n}" + + result = { + "model": args.model, + "threshold": threshold, + "template_count": len(templates), + "clusters": [ + {"id": c["id"], "representative": c["representative"], + "members": c["members"], "count": c["count"]} + for c in clusters + ], + } + with open(args.output, "w", encoding="utf-8") as fh: + json.dump(result, fh, ensure_ascii=False, indent=1) + fh.write("\n") + + print(f"CLUSTERS={len(clusters)}") + print(f"TEMPLATES={len(templates)}") + print(f"MODEL={args.model}") + print(f"THRESHOLD={threshold}") + print(f"OUTPUT={args.output}") + # Paste-ready NDJSON for the classification prompt: id + member count + + # representative only — the LLM never sees (or echoes) member logtypes. + for c in clusters: + print(json.dumps({"id": c["id"], "count": c["count"], + "representative": c["representative"]}, + ensure_ascii=False)) + + +def load_json_file(path, what): + try: + with open(path, encoding="utf-8") as fh: + return json.load(fh) + except OSError as exc: + fail(1, f"cannot read {what} file: {exc}") + except json.JSONDecodeError as exc: + fail(2, f"{what} file is not valid JSON: {exc}") + + +def cmd_expand(args): + clusters_doc = load_json_file(args.clusters, "--clusters") + classification = load_json_file(args.classification, "--classification") + + clusters = clusters_doc.get("clusters") if isinstance(clusters_doc, dict) else None + if not isinstance(clusters, list) or not clusters: + fail(2, f"--clusters file has no \"clusters\" list: {args.clusters}") + if not isinstance(classification, dict): + fail(2, "--classification file must be a JSON object") + + assignments = classification.get("assignments") + if not isinstance(assignments, list): + fail(2, "classification is missing the \"assignments\" list " + "(expected [{\"id\": \"c1\", \"category\": \"...\"}, ...])") + + cluster_ids = [c.get("id") for c in clusters] + known = set(cluster_ids) + categories = {} + problems = [] + for n, entry in enumerate(assignments, 1): + if not isinstance(entry, dict): + problems.append(f"assignment #{n} is not an object") + continue + cid, category = entry.get("id"), entry.get("category") + if not isinstance(cid, str) or not cid: + problems.append(f"assignment #{n} has no \"id\"") + elif cid not in known: + problems.append(f"assignment #{n}: unknown cluster id {cid!r}") + elif cid in categories: + problems.append(f"cluster id {cid!r} assigned more than once") + elif not isinstance(category, str) or not category.strip(): + problems.append(f"assignment #{n} ({cid}) has an empty \"category\"") + else: + categories[cid] = category.strip() + missing = [cid for cid in cluster_ids if cid not in categories] + if missing: + problems.append(f"{len(missing)} cluster id(s) not assigned: " + + ", ".join(missing)) + if problems: + for p in problems: + print(f"error: {p}", file=sys.stderr) + print("error: fix the classification JSON and re-run expand — nothing " + "was written, do NOT store this in the cache", file=sys.stderr) + sys.exit(2) + + for key, kind in (("taxonomy", list), ("query_plan", list), ("schema", dict)): + if key in classification and not isinstance(classification[key], kind): + fail(2, f"classification \"{key}\" must be a {kind.__name__}") + if "templates" in classification: + print("warning: classification contains \"templates\"; ignoring it — " + "templates are rebuilt from the cluster members", file=sys.stderr) + + templates = [] + for cluster in clusters: + cat = categories[cluster["id"]] + for member in cluster.get("members", []): + templates.append({"logtype": member, "category": cat}) + + expanded = {} + if "schema" in classification: + expanded["schema"] = classification["schema"] + expanded["taxonomy"] = classification.get("taxonomy", []) + expanded["templates"] = templates + expanded["query_plan"] = classification.get("query_plan", []) + + with open(args.output, "w", encoding="utf-8") as fh: + json.dump(expanded, fh, ensure_ascii=False, indent=1) + fh.write("\n") + + print(f"TEMPLATES={len(templates)}") + print(f"CATEGORIES={len(set(categories.values()))}") + print(f"OUTPUT={args.output}") + + +def main(): + signal.signal(signal.SIGPIPE, signal.SIG_DFL) + parser = Parser(prog="logtype-cluster", description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + sub = parser.add_subparsers(dest="command", required=True) + + p_cluster = sub.add_parser("cluster", + help="group similar logtypes; print representatives") + p_cluster.add_argument("--input", required=True, + help='{"logtype": ...} NDJSON (e.g. /tmp/logtypes-to-classify.ndjson)') + p_cluster.add_argument("--model", default=DEFAULT_MODEL, + help=f"model2vec model (default: {DEFAULT_MODEL})") + p_cluster.add_argument("--threshold", default=DEFAULT_THRESHOLD, + help=f"cosine similarity threshold in (0,1] (default: {DEFAULT_THRESHOLD})") + p_cluster.add_argument("--output", default="/tmp/logtype-clusters.json", + help="clusters JSON path (default: /tmp/logtype-clusters.json)") + p_cluster.set_defaults(func=cmd_cluster) + + p_expand = sub.add_parser("expand", + help="propagate per-cluster categories to all members") + p_expand.add_argument("--clusters", required=True, + help="clusters JSON produced by `cluster`") + p_expand.add_argument("--classification", required=True, + help='LLM output JSON with "assignments" (per cluster id)') + p_expand.add_argument("--output", default="/tmp/logtype-expanded.json", + help="expanded classification path (default: /tmp/logtype-expanded.json)") + p_expand.set_defaults(func=cmd_expand) + + args = parser.parse_args() + args.func(args) + + +if __name__ == "__main__": + main() diff --git a/plugins/clp/bin/logtype-insights-bootstrap b/plugins/clp/bin/logtype-insights-bootstrap new file mode 100755 index 0000000..53361c3 --- /dev/null +++ b/plugins/clp/bin/logtype-insights-bootstrap @@ -0,0 +1,204 @@ +#!/usr/bin/env bash +set -euo pipefail + +CLP_PLUGIN_BIN_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" +# shellcheck disable=SC1091 +source "${CLP_PLUGIN_BIN_DIR}/lib/clp-common.sh" + +usage() { + cat <<'EOF' +Usage: + logtype-insights-bootstrap [options] ARCHIVES_DIR + +Mechanical bootstrap for the logtype-insights skill. One call performs the +schema-discovery sample, per-field value distributions, the logtype dictionary +dump (with the templatize fallback for binaries that predate the shapes API), +and the classification-cache probe — then prints a compact KEY=VALUE summary. + +Options: + --message FIELD Message field name (clp-string). Required only when the + shapes dump is empty and the templatize fallback must run + (the summary says FALLBACK=TEMPLATIZE_NEEDS_MESSAGE). + --out-dir DIR Where the intermediate files are written (default: /tmp). + --cache-dir DIR Classification cache dir (default: logtype-cache default). + --sample-cap N Max records sampled for the value distributions + (default: 20000, or $CLP_BOOTSTRAP_SAMPLE_CAP). + -h, --help Show this help. + +Summary keys printed on stdout (grep-able): + ARCHIVE= archives dir as passed (the search wrapper resolves it) + SAMPLE= one full JSON record (reveals the field names) + DIST field=... distinct=N sampled=N values="v"(count),... + value distribution per top-level scalar field; low + distinct => severity/logger-like, high => message-like + LOGTYPE_COUNT= distinct logtypes in the archive (report to the user) + FALLBACK= SHAPES_OK | TEMPLATIZE_USED | TEMPLATIZE_NEEDS_MESSAGE + CACHE_MODE= UPTODATE | GROWTH | NEW (from logtype-cache diff) + APP_KEY= BASE_KEY= cache keys (BASE_KEY only on GROWTH) + TO_CLASSIFY= number of templates that need classifying (0 on UPTODATE) + LOGTYPES_FILE= TO_CLASSIFY_FILE= FREQS_FILE= + paths of the produced files + CLASSIFICATION_FILE= cached plan, fetched for you (UPTODATE only) + BASE_CLASSIFICATION_FILE= base entry, fetched for you (GROWTH only) + +Exit codes: 0 summary produced (all FALLBACK/CACHE_MODE outcomes), 1 archive or +dump failure, 2 usage error. +EOF +} + +message_field="" +out_dir="/tmp" +cache_dir="" +sample_cap="${CLP_BOOTSTRAP_SAMPLE_CAP:-20000}" +archives_dir="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --message) + [[ $# -ge 2 ]] || { echo "error: --message requires a value" >&2; exit 2; } + message_field="$2"; shift 2 ;; + --out-dir) + [[ $# -ge 2 ]] || { echo "error: --out-dir requires a value" >&2; exit 2; } + out_dir="$2"; shift 2 ;; + --cache-dir) + [[ $# -ge 2 ]] || { echo "error: --cache-dir requires a value" >&2; exit 2; } + cache_dir="$2"; shift 2 ;; + --sample-cap) + [[ $# -ge 2 ]] || { echo "error: --sample-cap requires a value" >&2; exit 2; } + sample_cap="$2"; shift 2 ;; + -h|--help) usage; exit 0 ;; + -*) + echo "error: unknown option: $1" >&2; usage >&2; exit 2 ;; + *) + if [[ -z "$archives_dir" ]]; then archives_dir="$1"; shift + else echo "error: unexpected argument: $1" >&2; usage >&2; exit 2; fi ;; + esac +done + +[[ -n "$archives_dir" ]] || { usage >&2; exit 2; } +[[ -d "$archives_dir" ]] || { echo "error: not a directory: $archives_dir" >&2; exit 1; } +[[ "$sample_cap" =~ ^[0-9]+$ && "$sample_cap" -gt 0 ]] \ + || { echo "error: --sample-cap must be a positive integer" >&2; exit 2; } +mkdir -p "$out_dir" + +SEARCH="${CLP_PLUGIN_BIN_DIR}/clp-s-search-kql" +CACHE_BIN="${CLP_PLUGIN_BIN_DIR}/logtype-cache" +cache_args=() +[[ -n "$cache_dir" ]] && cache_args=(--cache-dir "$cache_dir") + +logtypes_file="${out_dir}/logtypes.ndjson" +to_classify_file="${out_dir}/logtypes-to-classify.ndjson" +freqs_file="${out_dir}/logtype-freqs.txt" +err_file="${out_dir}/logtypes.err" +diff_file="${out_dir}/lt-diff.out" +sample_file="${out_dir}/logtype-sample.ndjson" + +# --- 1. Sample records for schema discovery + value distributions ------------ +# head closes the pipe early, so the search stops after $sample_cap records +# even on huge archives; the resulting SIGPIPE is expected, not an error. +set +o pipefail +"$SEARCH" "$archives_dir" '*' 2>/dev/null | grep '^{' | head -n "$sample_cap" > "$sample_file" +set -o pipefail + +if [[ ! -s "$sample_file" ]]; then + echo "error: no records returned from the archive (is $archives_dir a CLP archive dir?)" >&2 + exit 1 +fi + +sampled_records="$(wc -l < "$sample_file" | tr -d '[:space:]')" +sample_record="$(head -1 "$sample_file")" + +# Candidate schema fields: top-level scalar keys of the sample record (cap 8). +mapfile -t dist_fields < <(printf '%s\n' "$sample_record" \ + | jq -r 'to_entries[] | select(.value | type=="string" or type=="number" or type=="boolean") | .key' \ + | head -8) + +echo "ARCHIVE=$archives_dir" +echo "SAMPLE=$sample_record" +for field in "${dist_fields[@]}"; do + dist="$(jq -r --arg f "$field" '.[$f] // empty | tostring' "$sample_file" \ + | sort | uniq -c | sort -rn)" + distinct="$(printf '%s\n' "$dist" | grep -c . || true)" + # Top 8 values as "value"(count), values truncated to 60 chars. Herestring, + # not a pipe, feeding head: on high-distinct fields (e.g. timestamp) head + # exits after 8 of ~20k lines and a SIGPIPE'd printf would kill the script + # under set -e + pipefail (exit 141). + values="$(head -8 <<<"$dist" \ + | sed -E 's/^[[:space:]]*([0-9]+)[[:space:]](.*)$/\2\t\1/' \ + | awk -F'\t' '{v=$1; if (length(v)>60) v=substr(v,1,60)"…"; printf "%s\"%s\"(%s)", (NR>1?", ":""), v, $2}')" + echo "DIST field=$field distinct=$distinct sampled=$sampled_records values=$values" +done + +# --- 2. Logtype dictionary dump (shapes API; templatize fallback) ------------ +fallback="SHAPES_OK" +# normalize exits 1 on empty input (pre-shapes-API binaries) — the emptiness +# check below decides between the templatize fallback and a hard error. +set +o pipefail +"$SEARCH" "$archives_dir" 'stats.log_shapes' 2>"$err_file" \ + | grep '^{' \ + | "$CACHE_BIN" normalize > "$logtypes_file" 2>/dev/null || true +set -o pipefail + +if ! grep -q '^{' "$logtypes_file" 2>/dev/null; then + # The binary predates the shapes API (or the dump failed). Build an + # approximate baseline by projecting the message field and templatizing + # variable runs — O(records), but yields templates AND counts in one pass. + if [[ -z "$message_field" ]]; then + echo "LOGTYPE_COUNT=0" + echo "FALLBACK=TEMPLATIZE_NEEDS_MESSAGE" + echo "LOGTYPES_FILE=$logtypes_file" + echo "HINT=stats.log_shapes emitted no NDJSON (see $err_file). Re-run with" \ + "--message (the clp-string message field from SAMPLE/DIST, e.g." \ + "'message' or 'msg') to build the templatize fallback baseline." + exit 0 + fi + set +o pipefail + "$SEARCH" --projection "$message_field" "$archives_dir" '*' 2>/dev/null \ + | grep '^{' | jq -r --arg f "$message_field" '.[$f] // empty' \ + | sed -E 's/\{[^}]+\}/<*>/g; s/0x[0-9a-fA-F]+/<*>/g; s/\b[0-9]+\b/<*>/g' \ + | sort | uniq -c | sort -rn > "$freqs_file" + # Convert " COUNT template" lines to the canonical {"logtype":...} NDJSON + # (jq -R JSON-escapes; normalize dedups + sorts) so the cache probe works. + sed -E 's/^[[:space:]]*[0-9]+ //' "$freqs_file" \ + | jq -Rc 'select(length>0) | {logtype:.}' \ + | "$CACHE_BIN" normalize > "$logtypes_file" 2>/dev/null || true + set -o pipefail + fallback="TEMPLATIZE_USED" +fi + +if ! grep -q '^{' "$logtypes_file" 2>/dev/null; then + echo "error: could not build a logtype baseline (shapes dump empty and the" \ + "templatize fallback produced nothing; see $err_file)" >&2 + exit 1 +fi + +# --- 3. Logtype count + classification-cache probe --------------------------- +logtype_count="$("$CACHE_BIN" count --logtypes-file "$logtypes_file")" +"$CACHE_BIN" diff "${cache_args[@]}" --logtypes-file "$logtypes_file" > "$diff_file" +header="$(head -1 "$diff_file")" +mode="$(printf '%s' "$header" | cut -f1)" +app_key="$(printf '%s' "$header" | cut -f2)" +base_key="" +[[ "$mode" == "GROWTH" ]] && base_key="$(printf '%s' "$header" | cut -f3)" +# grep exits 1 when there is nothing to classify — the normal UPTODATE case. +grep '^{' "$diff_file" > "$to_classify_file" || true +to_classify="$(grep -c '^{' "$to_classify_file" || true)" + +echo "LOGTYPE_COUNT=$logtype_count" +echo "FALLBACK=$fallback" +echo "CACHE_MODE=$mode" +echo "APP_KEY=$app_key" +echo "BASE_KEY=$base_key" +echo "TO_CLASSIFY=$to_classify" +echo "LOGTYPES_FILE=$logtypes_file" +echo "TO_CLASSIFY_FILE=$to_classify_file" +[[ "$fallback" == "TEMPLATIZE_USED" ]] && echo "FREQS_FILE=$freqs_file" + +# Fetch the cache entries the next steps need, so the caller doesn't have to. +if [[ "$mode" == "UPTODATE" ]]; then + "$CACHE_BIN" get "${cache_args[@]}" "$app_key" > "${out_dir}/logtype-classification.json" + echo "CLASSIFICATION_FILE=${out_dir}/logtype-classification.json" +elif [[ "$mode" == "GROWTH" && -n "$base_key" ]]; then + "$CACHE_BIN" get "${cache_args[@]}" "$base_key" > "${out_dir}/logtype-base-classification.json" + echo "BASE_CLASSIFICATION_FILE=${out_dir}/logtype-base-classification.json" +fi diff --git a/plugins/clp/skills-claude/dev/SKILL.md b/plugins/clp/skills-claude/dev/SKILL.md index 418a59e..66ae524 100644 --- a/plugins/clp/skills-claude/dev/SKILL.md +++ b/plugins/clp/skills-claude/dev/SKILL.md @@ -132,7 +132,10 @@ claude plugin validate . claude plugin validate ./plugins/clp # Wrapper syntax -for f in plugins/clp/bin/clp-s-*; do bash -n "$f"; done +for f in plugins/clp/bin/clp-s-* plugins/clp/bin/logtype-cache \ + plugins/clp/bin/logtype-insights-bootstrap \ + plugins/clp/bin/logtype-cluster; do bash -n "$f"; done +python3 -m py_compile plugins/clp/bin/logtype-cluster.py # Static analysis (if shellcheck is installed) shellcheck \ @@ -140,6 +143,8 @@ shellcheck \ plugins/clp/bin/clp-s-compress-session \ plugins/clp/bin/clp-s-search-kql \ plugins/clp/bin/clp-s-decompress \ + plugins/clp/bin/logtype-insights-bootstrap \ + plugins/clp/bin/logtype-cluster \ plugins/clp/bin/lib/clp-common.sh ``` diff --git a/plugins/clp/skills-claude/logtype-insights/SKILL.md b/plugins/clp/skills-claude/logtype-insights/SKILL.md index a603207..d93e37d 100644 --- a/plugins/clp/skills-claude/logtype-insights/SKILL.md +++ b/plugins/clp/skills-claude/logtype-insights/SKILL.md @@ -1,12 +1,14 @@ --- name: logtype-insights -description: App-agnostic logtype-baseline log analysis with CLP. Dump the archive's logtype dictionary first, classify the real templates into (generic + app-discovered) categories, and drive targeted KQL from them — no blind queries. Caches the classification and dynamically updates it when the archive grows (classifies only new templates, merges into the existing entry), and reports the archive's logtype count. Works on any structurized or native-JSON CLP archive (vLLM, MongoDB, nginx, …). +description: App-agnostic logtype-baseline log analysis with CLP. Dump the archive's logtype dictionary first, classify the real templates into (generic + app-discovered) categories, and drive targeted KQL from them — no blind queries. Caches the classification and updates it incrementally when the archive grows; reports the archive's logtype count. Works on any structurized or native-JSON CLP archive (vLLM, MongoDB, nginx, …). allowed-tools: - "Agent" - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/clp-s-compress-folder:*)" - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql:*)" - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/clp-s-decompress:*)" - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/logtype-cache:*)" + - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/logtype-insights-bootstrap:*)" + - "Bash(${CLAUDE_PLUGIN_ROOT}/bin/logtype-cluster:*)" - "Bash(jq:*)" - "Bash(grep:*)" - "Bash(sort:*)" @@ -23,614 +25,147 @@ allowed-tools: # Logtype Insights (App-Agnostic, Logtype-Baseline) -End-to-end analysis of **any** CLP archive — structurized text logs (vLLM -wrapper logs → `timestamp/logger/level/message`), native JSON logs (MongoDB → -`t.$date/s/c/msg/attr`), or other JSON — using the **logtype baseline** method: -dump the archive's logtype dictionary first, classify those *real* message +End-to-end analysis of **any** CLP archive using the **logtype baseline** +method: dump the archive's logtype dictionary (the complete vocabulary of +distinct message templates, `<*>` marking variables — tens to a few hundred +templates no matter how many millions of records), classify those *real* templates into categories, and derive every later query from a template that is -guaranteed to exist. No blind keyword batteries, no queries wasted on keywords -that aren't there. +guaranteed to exist. No blind keyword batteries. -The logtype method is **not application-specific**: the dictionary dump, the -generic category taxonomy, the classification cache, and the project+grep -retrieval pattern all work on any archive. The only thing that changes between -applications is the set of templates — which the skill reads from the archive -itself rather than guessing. +The classification is a property of the **application**, not the capture, so it +is cached (keyed by a fingerprint of the template set) and updated +incrementally when the archive grows — re-analyzing the same app skips +classification entirely. For a single ad-hoc KQL query, use the `search` skill. To compress raw logs first, use `compress-folder`. -## Why a logtype baseline beats blind search +## References — read on demand, not up front -A CLP logtype is a message template with variables replaced by `<*>`, e.g. -`Triton not installed or not compatible; certain GPU-related functions ...` or -MongoDB's `Slow query`, `attr.durationMillis=<*>`. The logtype dictionary is the -**complete vocabulary** of distinct message shapes in the archive — for a -typical run, tens to a few hundred templates, no matter how many millions of -records. Dumping it gives you, in one cheap pass that reads the dictionary -rather than every record: - -- Every kind of event the run actually produced (no guessing keywords). -- The static tokens of each template, which you turn into queries that always - match — so counts are exact and zero queries return zero by surprise. -- A natural unit for "top repeated messages": frequency per template. - -The blind variants run a fixed battery of hardcoded -queries; on an unfamiliar archive many return nothing. This skill runs **1 dump -+ schema discovery + a handful of targeted queries**, each grounded in a real -template. - -## Why the classification is cached - -Classifying the templates into categories and deriving a query plan is the one -expensive step, and it is a property of the **application**, not the individual -capture: the same app build emits the same message templates on every run, so -the same classification applies. This skill caches the classification keyed by a -fingerprint of the template set (`sha256` of the sorted logtypes). On a cache -hit (same app), classification is skipped entirely and the skill goes straight -to the insight pass with the pre-made plan — so re-analyzing the same -application costs only the cheap Haiku insight pass, not the classification. -When the archive **grows** (new logs add new message templates), the cache is -updated **dynamically**: only the newly-appearing templates are classified and -merged into the existing entry, instead of reclassifying the whole dictionary -(see "Classification cache details"). The skill also reports the number of -logtypes in the current archive. +- `${CLAUDE_PLUGIN_ROOT}/skills-claude/references/logtype-classify.md` — read + at step 6 (classification subagent prompt, cluster/expand contract, cache + store commands). +- `${CLAUDE_PLUGIN_ROOT}/skills-claude/references/logtype-insight.md` — read at + step 7 (insight subagent prompt, report format). +- `${CLAUDE_PLUGIN_ROOT}/skills-claude/references/logtype-baseline.md` — read + only if the dump/fallback misbehaves or when drilling into individual + templates (stats.log_shapes encodings, CLP-string limitation, + retrieve/count/analysis patterns, semantic-search flags). ## Supported inputs - A CLP archive directory (any kind). Primary input. -- A folder of raw logs — compress first with the app-appropriate settings, since - compression is the one app-specific step: - - vLLM wrapper text logs: `--structurize` (produces `timestamp/logger/level/message`). +- A folder of raw logs — compress first (compression is the one app-specific + step), then point this skill at the resulting archive: + - vLLM wrapper text logs: `--structurize`. - MongoDB JSON: `--extensions '*' --timestamp-key t.$date` (native). - Generic JSON with a known timestamp field: `--timestamp-key `. - - Then point this skill at the resulting archive. - If nothing was provided, ask for an archive or folder path. ## Workflow Each Bash call runs in its own shell, so shell variables do not persist between -steps. Re-declare `ARCHIVE`, `SEARCH`, and `CACHE` (and, on the GROWTH path, -`MODE`/`APP_KEY`/`BASE_KEY`) in any command that uses them, or run the dependent -commands together in one call. - -1. Determine the input: - - If the user provided an archive path, use it. - - If the user provided a folder, compress it with the app-appropriate - settings (above) and use the resulting archive. If the app is unknown, ask - the user how the logs should be compressed (structurize vs native - `--timestamp-key`), or have them compress first and pass the archive. - - If nothing was provided, ask for an archive or folder path. - -2. Report compression stats when you compressed the folder: - - `Raw input bytes`, `Archive bytes`, `Compression ratio`, - `File size reduction`, `Input files`, `Archives dir`, `Archive metadata`. - -3. **Discover the schema** (cheap; do this in the parent). A no-projection - search returns the full original record, so one sample line reveals the - field names: - - ```bash - ARCHIVE= - SEARCH="${CLAUDE_PLUGIN_ROOT}/bin/clp-s-search-kql" - - # One full record (reveals the JSON keys / structurized fields): - "$SEARCH" "$ARCHIVE" '*' 2>/dev/null | grep '^{' | head -1 - ``` - - Identify and record, as `schema`, the field names for: - - **timestamp** — e.g. `timestamp` (vLLM structurized) or `t.$date` (Mongo). - If it is a real epoch (native JSON), `--tge`/`--tle` work; if it is a - structurized string, they do not. - - **severity** — e.g. `level` (vLLM) or `s` (Mongo). - - **logger/component** — e.g. `logger` (vLLM) or `c` (Mongo). - - **message** — the clp-string field whose logtypes appear in - `stats.log_shapes` — e.g. `message` (vLLM) or `msg` (Mongo). - - **payload** (optional) — e.g. `attr` (Mongo); note the useful leaf paths - (e.g. `attr.durationMillis`, `attr.host`). - Also note the distinct values of the severity and logger fields (one count - query each) so the classifier and insight pass can use the real vocabularies: - ```bash - "$SEARCH" --projection "$ARCHIVE" '*' | grep '^{' | jq -r '.' | sort | uniq -c | sort -rn - "$SEARCH" --projection "$ARCHIVE" '*' | grep '^{' | jq -r '.' | sort | uniq -c | sort -rn - ``` - -4. **Dump the logtype baseline** (cheap; reads the dictionary, not every record): - - ```bash - # Dictionary dump (shapes API, clp-core >= 0.13). stats.log_shapes emits one - # raw line per template — {"archive_id":"...","count":...,"id":N,"shape":"..."} — - # where the shape string encodes variables in an archive-dependent form - # (raw placeholder bytes on regular archives, %rule.name% TextShape - # placeholders on clpp/--experimental archives). `logtype-cache normalize` - # detects the encoding per line and renders both to `<*>`, emitting the - # canonical, deduplicated {"logtype":"...<*>..."} NDJSON used everywhere - # below. The wrapper adds --experimental for stats queries automatically and - # rejects the legacy `stats.logtypes` spelling. - # It also prints archive-metadata header lines to stdout, so filter to JSON - # records with grep '^{' first (the repo-wide idiom; required by the - # jq-based pipelines). - "$SEARCH" "$ARCHIVE" 'stats.log_shapes' 2>/tmp/logtypes.err \ - | grep '^{' \ - | "${CLAUDE_PLUGIN_ROOT}/bin/logtype-cache" normalize > /tmp/logtypes.ndjson +steps — re-declare them or run dependent commands together in one call. - # Summary: how many distinct templates, and the templates themselves. - jq -s 'length' /tmp/logtypes.ndjson - jq -r '.logtype' /tmp/logtypes.ndjson - ``` +**Keep the user posted at every step.** Before each command or subagent, say in +one short line what you are about to do; after it, report the key numbers it +produced. Never chain steps silently — steps 5–7 run long, and without your +narration the user sees only a spinner. - **Fallback if `stats.log_shapes` emits no NDJSON** (e.g. the binary predates - the shapes API and fails to open the archive with `--experimental`). If - `/tmp/logtypes.ndjson` has - zero JSON lines, build an approximate baseline by projecting the message - field for all records and templatizing the variable runs (O(records), but - produces templates AND counts in one pass): +1. **Determine the input.** Archive path → use it. Folder → compress with the + app-appropriate settings above (ask the user if the app is unknown). Nothing + → ask. - ```bash - MSG= # e.g. message (vLLM) or msg (Mongo) - "$SEARCH" --projection "$MSG" "$ARCHIVE" '*' \ - | grep '^{' | jq -r --arg f "$MSG" '.[$f]' \ - | sed -E 's/\{[^}]+\}/<*>/g; s/0x[0-9a-fA-F]+/<*>/g; s/\b[0-9]+\b/<*>/g' \ - | sort | uniq -c | sort -rn > /tmp/logtype-freqs.txt - ``` +2. **Report compression stats** when you compressed the folder: `Raw input + bytes`, `Archive bytes`, `Compression ratio`, `File size reduction`, + `Input files`, `Archives dir`, `Archive metadata`. - (The wrapper prints archive-metadata header lines to stdout, so `grep '^{'` - filters to JSON records before `jq` — same idiom as `grep -c '^{'` for - counting.) - -5. **Report the logtype count and probe the cache for dynamic update.** The - archive's logtype count and the cache state are determined in one step: +3. **Bootstrap.** Tell the user you are sampling the schema, dumping the + logtype dictionary, and probing the classification cache — then run the one + command that does all of it: ```bash - CACHE="${CLAUDE_PLUGIN_ROOT}/bin/logtype-cache" - - # Number of distinct logtypes in the current archive (report this to the user): - LOGTYPE_COUNT=$("$CACHE" count --logtypes-file /tmp/logtypes.ndjson) - echo "Logtypes in this archive: $LOGTYPE_COUNT" - - # Dynamic-update probe. Emits a header line then NDJSON {"logtype":"..."} for - # the templates that need classifying (empty on UPTODATE): - # UPTODATE\t\t -> reuse cache, nothing to classify - # GROWTH \t\t\t\t -> archive grew from ; - # classify the new templates - # NEW \t\t -> no base; classify all - "$CACHE" diff --logtypes-file /tmp/logtypes.ndjson > /tmp/lt-diff.out - HEADER="$(head -1 /tmp/lt-diff.out)" - MODE="$(printf '%s' "$HEADER" | cut -f1)" - APP_KEY="$(printf '%s' "$HEADER" | cut -f2)" - BASE_KEY="$(printf '%s' "$HEADER" | cut -f3)" # only set when MODE=GROWTH - # `|| true` because grep exits 1 when there is nothing to classify, which is - # the normal UPTODATE (cache-hit) case — not an error. - grep '^{' /tmp/lt-diff.out > /tmp/logtypes-to-classify.ndjson || true + "${CLAUDE_PLUGIN_ROOT}/bin/logtype-insights-bootstrap" ``` - - **UPTODATE:** the archive's template set is unchanged since the cached - classification — reuse it. Verify the cached `schema` matches the schema you - discovered in step 3; if it matches, load it and skip to step 7: - `"$CACHE" get "$APP_KEY" > /tmp/logtype-classification.json`. If the schema - differs, treat as NEW (reclassify all). - - **GROWTH:** the archive grew from a previous capture (`base_key`). Only the - `` new logtypes need classifying — the existing templates keep their - cached categories. Save the base classification for the classifier to reuse: - `"$CACHE" get "$BASE_KEY" > /tmp/logtype-base-classification.json` where - `BASE_KEY="$(printf '%s' "$HEADER" | cut -f3)"`. Then classify the new - templates (step 6) and merge. - - **NEW:** first capture of this application (no compatible base). Classify - all templates (step 6) and store. - -6. **(GROWTH / NEW only) Classify the templates diff emitted.** Spawn a - **classification subagent** with the Agent tool, model `sonnet` (fall back to - `haiku`). It classifies ONLY the templates in - `/tmp/logtypes-to-classify.ndjson` (the new ones for GROWTH, all of them for - NEW), in the discovered field names, and writes the result as structured JSON - to `/tmp/logtype-new-class.json`. (On UPTODATE this step is skipped.) + From its `KEY=VALUE` output record: + - `SAMPLE=` + `DIST field=... distinct=N values=...` → pick the **schema**: + timestamp, severity, logger, **message** (the clp-string field — high + distinct-count prose), payload leaves if any. Low-distinct fields are + severity/logger-like; note their value vocabularies from the DIST lines. + - `LOGTYPE_COUNT=` → report to the user. + - `FALLBACK=TEMPLATIZE_NEEDS_MESSAGE` → the clp-s binary predates the + shapes API; tell the user ("old clp-s binary — rebuilding the baseline + via message templatization"), then re-run ONCE adding + `--message `. No other re-runs are needed. + - `CACHE_MODE=` / `APP_KEY=` / `BASE_KEY=` / `TO_CLASSIFY=` → step 4. + + Then tell the user what the bootstrap found, in 2–3 lines: the logtype + count, the schema you picked, whether the templatize fallback was used, and + the cache mode. + +4. **Branch on `CACHE_MODE`** — and announce the branch to the user: + UPTODATE → "cached classification found; skipping straight to the insight + pass"; GROWTH → "N of M templates are new; classifying only those"; NEW → + "first capture of this app; classifying all N templates". + - **UPTODATE** — the cached plan was already fetched to + `/tmp/logtype-classification.json` (`CLASSIFICATION_FILE=`). Verify its + `.schema` matches step 3's schema; if it does, skip to step 7. If it + differs, treat as NEW (continue, clustering `/tmp/logtypes.ndjson`). + - **GROWTH** — only the new templates in `/tmp/logtypes-to-classify.ndjson` + need classifying; the base plan was fetched to + `/tmp/logtype-base-classification.json`. Continue to step 5. + - **NEW** — first capture of this app; classify all of + `/tmp/logtypes-to-classify.ndjson`. Continue to step 5. + +5. **Cluster the templates to classify** — merges semantically similar + templates so the classification subagent sees one representative per + cluster instead of every template (in step 4's schema-mismatch case, pass + `--input /tmp/logtypes.ndjson` instead): - Classification subagent prompt template (fill in `ARCHIVE`, the `schema`, and - paste the templates to classify from step 5; for GROWTH also paste the base - taxonomy/plan so existing categories are reused): - - ``` - You are classifying the logtype templates listed below for a CLP archive, so a - later insight pass can run targeted queries. Do NOT write the final report — - only the classification JSON. Classify ONLY the templates listed below (not any - others) — for an incremental update these are the NEW templates; for a first - run they are all of them. - - Archive: ARCHIVE - Discovered schema (field names in this archive): - timestamp: - severity: - logger: - message: (the clp-string field whose logtypes these are) - payload: (leaf paths if any, e.g. attr.durationMillis) - Severity values seen: - Logger values seen: - - TEMPLATES TO CLASSIFY (one {"logtype":"..."} per line; classify each): - - - [Only for GROWTH] Existing categories from the previous classification — REUSE - these where a template fits; add a new category only if none fits. Existing - query-plan labels (do not duplicate): - - Method: - 1. Classify EACH template above into the best-fitting category. Use this GENERIC - default taxonomy, AND any APP-SPECIFIC categories already in use (GROWTH) or - that you discover from the templates (NEW), e.g. for MongoDB: - workload/operations (slow query, write-concern waits), replication/election, - sharding, indexing, WiredTiger/storage; for vLLM: worker-health, kv-cache, - model-loading. Generic defaults: - - errors / exceptions / failures - - warnings - - performance (latency / throughput / timing / "took <*> ms") - - config / startup / initialization - - network / connectivity / timeout - - resource (memory / disk / file-descriptors / storage pressure) - - lifecycle / state-transitions (start/stop/election/stepdown/restart) - - security / auth / access - - other (note but don't deep-search) - 2. Build a QUERY PLAN: a list of targeted queries, each derived from one or - more of the templates above, expressed in the discovered field names. For - each plan entry give: label, the KQL filter (using the searchable scalar - fields — severity/logger/payload leaves; NOT message:term which is a - clp-string and returns 0), the columns to --projection, and the method: - - "count" -> count matches via `... | grep -c '^{'` - - "project+grep" -> project the message field and grep its static text - - "project+jq" -> project message/payload and jq-filter (e.g. a - numeric threshold on a payload leaf) - - "semantic" -> semantic("...") AND , ONLY for an - ambiguous template or to group similar ones - Example plan entry (Mongo schema): - {"label":"Slow queries","kql":"attr.durationMillis:*", - "project":"t.$date,attr.durationMillis,msg", - "jq":"select((.attr.durationMillis//0)>100)","method":"project+jq"} - Example plan entry (vLLM schema): - {"label":"Memory warnings","kql":"level:WARNING", - "project":"timestamp,level,message","grep":"memory|OOM|KV", - "method":"project+grep"} - For GROWTH, reuse existing plan labels where the new templates fit an - existing category; add new plan entries only for genuinely new signals. - 3. Remember: the message field is a clp-string. KQL `message:term` / - `msg:term` and `message:*term*` / `msg:*term*` return 0. Only the scalar - fields (severity, logger, payload leaves) are KQL-searchable. Retrieve - message content by projecting the message field and grepping. - - Write the result as valid JSON to /tmp/logtype-new-class.json with EXACTLY - this shape, then print "DONE" and nothing else: - { - "schema": {"timestamp":"","severity":"","logger":"","message":"","payload":["",...]}, - "taxonomy": [{"category":"","description":""}], - "templates": [{"logtype":"