tmaudit — Deploy & Quick Start

This directory is a deployable Python package named tmaudit. It ships three things in one place:

1. The templates for the three operational scripts that every PAPER<N>_CONSOLIDATED/ directory needs (verify_p<N>.py, _compile_check.py, _fix_abstract_unicode.py).
2. The per-paper configurations (C1..C10 audit rules: C1–C7 in v0.1.0–v0.2.0, plus C8 statistical power, C9 figure-caption, and C10 reproducibility in v0.3.0; abstract Unicode replacements) for Paper 1, 2, 3, 4, 5.
3. A CLI (tmaudit) that forks the templates and (optionally) runs them.

Scope, stated plainly: the built-in rulepack covers this project's five papers. verify / compile / fix-unicode / audit-all take --paper 1..5, and --paper-dir changes where the forked script is written — not which rules it contains. For a manuscript those configs were not written for, tmaudit verify --paper-dir DIR (no --paper) forks the factory template into DIR: the per-paper categories report not applicable and the generic categories run. To add checks the generic categories do not have, write your own plugin: see docs/PLUGINS.md.

Four ways to use it

(A) Run the single-file tmaudit.pyz (zero install)

# on any machine with Python 3.9+
python tmaudit.pyz list
python tmaudit.pyz verify --paper 1
python tmaudit.pyz compile --paper 5
python tmaudit.pyz fix-unicode --paper 1
python tmaudit.pyz audit-all

The .pyz is a self-extracting Python zip archive; the user does not need to pip install anything. The file is ~64 KB and contains all five PAPER_CONFIGS and all three templates.

It is a convenient way to hand the tool to a collaborator working in the same research tree: the built-in configs point at PAPER1..PAPER5_CONSOLIDATED, so they will need TMAUDIT_RESEARCH_DIR set to a tree with that layout. For a manuscript outside those five papers the built-in commands will not audit it — write a plugin instead (docs/PLUGINS.md).

(B) pip install --editable . (development)

cd <your tmaudit checkout>
python -m pip install --editable .
python -m pip install --editable ./tmaudit_example_plugin
tmaudit list
tmaudit verify --paper 1

The second install is not optional for tests: tests/test_example_plugin.py asserts that the in-repo demo plugin is discoverable via registered entry points. Without that editable install, 11 plugin tests fail with KeyError: 'flag-long-abstract'. The CI workflow (ci.yml) runs the same two installs.

Edits to src/tmaudit/templates/*.py or src/tmaudit/configs/*.py are picked up immediately on the next tmaudit invocation. This is the recommended form for ongoing development of the templates themselves.

(C) pip install . (deployment to a venv)

python -m venv .venv
source .venv/bin/activate           # or .venv\Scripts\activate on Windows
pip install .
tmaudit list

This produces a tmaudit console-script entry in the venv. The template and config files are inside the wheel.

(D) pip install tmaudit (from PyPI)

pip install tmaudit          # or: pipx install tmaudit
tmaudit list

Published on PyPI via trusted publishing (OIDC) from .github/workflows/publish-pypi.yml; new versions are cut from GitHub Releases. This is the quickest form for someone who just wants the CLI.

Extend it: your own checks

The built-in categories are this project's rulepack for five papers. To add checks of your own — including for a manuscript those configs do not cover — write a plugin: a function decorated with @check, registered under the tmaudit.plugins entry-point group, receiving your paper's root .tex and returning Finding objects. (The generic categories run without a plugin via tmaudit verify --paper-dir /path/to/your/paper.)

tmaudit verify --paper-dir /path/to/your/paper   # generic checks, no config
pip install -e ./my-paper-checks      # plugins are found via installed entry points
tmaudit plugins list                  # confirm it registered
tmaudit plugins run my-check --paper-dir /path/to/your/paper

The full contract, a copy-paste package layout, and what a plugin does and does not get are in docs/PLUGINS.md. A working three-check example ships in tmaudit_example_plugin/.

Self-audit: the audit core on 25 real papers

The rulepack is per-paper by design, so the sharpest question a reader can ask is: what does this tool do on a paper its config was not written for? We ran all ten categories over 25 real arXiv cs.CL LaTeX sources, exactly as main() runs them.

It found four defects in the tool itself:

• the factory CHECKS_CONFIG was one specific paper's config, so a generic run fired that paper's checks against any document;
• "clean" could mean "did not run" — a category with no configuration reported zero findings, indistinguishable from a pass;
• one missing config key aborted the whole audit with a KeyError;
• the BibTeX key pattern fabricated "missing citation" findings for keys that were present in refs.bib, whenever a key sat on its own line.

All four are fixed in v0.5.1, with regression tests; the two smaller findings (a SyntaxWarning on import, and an observation failure filed as a HIGH finding) are fixed as well. The v0.5.0 findings are kept verbatim as evidence, and the harness restores the v0.5.0 config so the config-driven defects replay: case-studies/arxiv-corpus/ has the coverage matrix, the method, and what the numbers do not establish.

Auditing a paper the rulepack was not written for

case-studies/arxiv-2502.14074/ runs the two public extension paths — verify --paper-dir and two plugin checks — against one real paper (Investigating Non-Transitivity in LLM-as-a-Judge, arXiv:2502.14074, ICML 2025). It reports two reporting-standards observations, records three candidate weaknesses that were checked and withdrawn, and surfaced a caption-parsing defect in the tool (nested braces), fixed in this commit. Every input is pinned by SHA-256; the results, and what the audit does not establish, are in its README.

v0.3.0 highlights

Released: 2026-07-10. This release adds 3 new audit categories (C8, C9, C10) and brings the project to 10 categories, 176 unit tests, 12/12 meta-test bugs caught, 5/5 papers auditable.

Category	What it checks	Severity	Always runs?
C1 (v0.1.0)	Symbol inline definition	HIGH	Yes
C2 (v0.1.0)	Bonferroni consistency	HIGH	Yes
C3 (v0.1.0)	Formalization of key concept	MEDIUM	Yes
C4 (v0.1.0)	Self-citation threshold	MEDIUM	Yes
C5 (v0.1.0)	Test-name d	MEDIUM	Yes
C6 (v0.1.1)	Blacklist word frequency	LOW	Yes
C7 (v0.1.2)	Ceremonial citation detection	MEDIUM	Yes
C8 (v0.3.0)	Statistical power (d-mismatch, post-hoc power)	HIGH/MED	No (opt-in)
C9 (v0.3.0)	Figure-caption consistency (4 sub-checks)	HIGH/MED	Yes (sub-check 3 opt-in)
C10 (v0.3.0)	Reproducibility (availability, metadata, consistency)	HIGH/MED/LOW	Yes (sub-check 2 opt-in)
C7-LLM (v0.2.0)	LLM fallback for C7 borderline cases (opt-in via TMAUDIT_LLM_* env)	MEDIUM	No

v0.2.0 added (still shipping): audit-all --output report.md generates a single Markdown report covering all papers; the report is uploaded as a CI artefact.

v0.2.0 added (still shipping): per-paper config for Paper 2, Paper 3, Paper 4 (was placeholder in v0.1.1). All 5 papers (P1..P5) are now auditable.

Per-paper config for v0.3.0: c8_claimed_effects (statistical), c9_figure_keywords (figure-caption), c10_reproducibility_claims (reproducibility). Empty list = opt-out (with graceful degradation in C9 and C10).

See RELEASE_NOTES_v0.3.0.md for the full release notes and engineering_notes_verify_template.md §12 (C8+C10) and §13 (C9) for the design rationale.

CLI reference

tmaudit list                              # show all known paper configs
tmaudit verify --paper N [--dry-run]      # fork verify_pN.py + run audit
tmaudit compile --paper N [--dry-run]     # fork _compile_check.py + 4-pass build
tmaudit fix-unicode --paper N [--dry-run] # fork _fix_abstract_unicode.py
tmaudit audit-all [--start 1] [--end 5]   # verify all papers in a range

Every subcommand also accepts --paper-dir PATH to override the default paper directory.

The standard 3-step protocol for any new paper

cd F:\Research\PAPER<N>_CONSOLIDATED

# 1. audit
tmaudit verify --paper N

# 2. fix abstract Unicode (if any finding)
tmaudit fix-unicode --paper N
tmaudit verify --paper N

# 3. compile
tmaudit compile --paper N

Package layout

TEMPLATE/
├── pyproject.toml                            # build config
├── tmaudit.pyz                               # single-file distribution (64 KB)
├── README.md                                 # this file
├── engineering_notes_verify_template.md      # 6-bug engineering diary
├── _setup_pkg.py                             # one-time: copy templates into src/
├── _build_pyz.py                             # rebuild tmaudit.pyz after any src/ change
├── _compile_appendix.py                      # LaTeX compile for the appendix
├── _regen_appendix.py                        # regenerate appendix_engineering.tex
├── _status.py                                # TEMPLATE/ state summary
└── src/
    └── tmaudit/                              # the actual package
        ├── __init__.py
        ├── __main__.py                       # `python -m tmaudit`
        ├── cli.py                            # argparse CLI
        ├── forge.py                          # template-substitution engine
        ├── templates/
        │   ├── __init__.py
        │   ├── verify_TEMPLATE.py            # ≈ 22 KB, ~400 lines
        │   ├── compile_check_TEMPLATE.py     # ≈ 4.5 KB
        │   └── fix_abstract_unicode_TEMPLATE.py  # ≈ 5 KB
        └── configs/
            ├── __init__.py                   # re-exports
            ├── paper_configs.py              # VERIFY_CONFIGS for paper 1..5
            └── compile_configs.py            # COMPILE_CONFIGS for paper 1..5

Adding a new paper (e.g., Paper 6)

Edit two files:

1. src/tmaudit/configs/paper_configs.py — add a new entry to PAPER_CONFIGS (or VERIFY_CONFIGS after re-export). The fields are documented in the docstring at the top of that file.
2. src/tmaudit/configs/compile_configs.py — add a matching entry to COMPILE_CONFIGS. The fields are documented in the docstring there.

Then:

python -m pip install --editable . --break-system-packages
python _build_pyz.py
tmaudit list               # confirm Paper 6 shows up
tmaudit verify --paper 6   # fork + run; will report any HIGH-severity
                           # issues caused by your config choices

Verifying the .pyz is portable

# On the source machine:
python _build_pyz.py
# -> tmaudit.pyz (64 KB)

# Copy tmaudit.pyz to a fresh machine with no Python packages installed
# (only a base Python 3.9+ interpreter):
scp tmaudit.pyz other-host:/tmp/
ssh other-host 'python /tmp/tmaudit.pyz list'
# -> should print the same 5-paper table

The .pyz has no external dependencies (only the Python standard library). It works on Linux, macOS, and Windows alike.

Bug history

The twelve historical bugs encountered while building this package are documented in engineering_notes_verify_template.md. Each bug has a dedicated regression test in tests/test_forge_happy.py, tests/test_bug5_unit.py, tests/test_c6_threshold.py, tests/test_c7_citation_context.py, tests/test_cache.py, tests/test_c10_reproducibility.py, tests/test_c8_statistical_power.py, and tests/test_c9_figure_caption.py. The meta-test (_check_all_regressions.py) injects each bug in turn, asserts the regression test catches it, and restores the source. 14/14 bugs are caught.

The fix patterns (raw-string regex literals, brace-counting state machines, non-letter-class regex, second/third-pass audit logic, sentinel-character sub-string extraction, etc.) are all implemented in src/tmaudit/forge.py and src/tmaudit/templates/verify_TEMPLATE.py.

Continuous integration

Workflow

.github/workflows/ci.yml runs on every push and pull request. It consists of seven jobs:

1. pytest (matrix: Python 3.9, 3.10, 3.11, 3.12, 3.13 on ubuntu-latest — every version requires-python >=3.9 claims) — runs the 246-test suite and uploads JUnit XML artifacts.
2. meta-test (Python 3.11) — runs _check_all_regressions.py to verify that each of the 14 regression tests actually catches its target bug, then runs _check_pyz_fresh.py to confirm the committed tmaudit.pyz still matches src/tmaudit/, and finally builds the archive and runs it as a smoke test.
3. validate-community-files (Python 3.11) — runs _check_github_templates.py to verify the .github/ directory structure (4 issue templates, PR template, CODEOWNERS, dependabot.yml, CODE_OF_CONDUCT, SUPPORT, workflows).
4. audit-all (Python 3.11) — runs tmaudit audit-all --start 1 --end 5 --output audit-report.md --no-cache and uploads the Markdown report as a CI artefact.
5. lint (Python 3.11) — runs pyflakes on src/tmaudit/ and tests/.
6. mutation-control (Python 3.11) — proves the pytest and lint gates can actually fail: injects a known mutation, asserts both gates go red, then asserts a deliberately neutralised gate (wrapped in || true) is rejected as a negative control.
7. secret-scan (Python 3.11) — scans the full history with gitleaks, so a committed credential fails the build.

Local validation (before pushing)

To run the same checks locally before pushing:

pip install -e . pytest pyflakes
pip install -e ./tmaudit_example_plugin
pytest
python _check_all_regressions.py
python _check_pyz_fresh.py
python _build_pyz.py
python tmaudit.pyz list
python -m pyflakes src/tmaudit/ tests/

pytest runs the full suite: 246 passed, 3 skipped on a machine without the research tree — the two @pytest.mark.slow end-to-end audit tests require the real PAPER<N>_CONSOLIDATED papers and auto-skip when the directory is absent (point TMAUDIT_RESEARCH_DIR at the tree to run them). Scope note: CI runs pytest -m "not slow", which deselects those two and reports 246 passed, 1 skipped, 2 deselected — both lines above are the same 249 collected tests, differing only in the two deselected slow tests.

Local CI simulation (act / _act_*)

The preferred tool for local CI simulation is nektos/act:

act -l                    # list jobs
act -j pytest             # run the pytest job
act -j meta-test          # run the meta-test job (after pytest)
act -j lint               # run the lint job

act requires act installed and a working Docker daemon. If either is unavailable (e.g., on a restricted sandbox machine), this repo ships two fallback simulators:

Script	act equivalent	What it does
_act_dryrun.py	act -n (dry-run)	Parses ci.yml, expands the matrix, prints what each step would do. No commands are executed.
_act_simulate.py	act (real run)	Same as dryrun, but actually runs the run: steps via subprocess. Implements stubs for actions/checkout@v4, actions/setup-python@v5, actions/upload-artifact@v4.

python _act_dryrun.py            # show what would be executed
python _act_dryrun.py --list     # show jobs only
python _act_simulate.py --list   # list jobs (act -l equivalent)
python _act_simulate.py --job pytest    # run pytest job

CI simulation results

_act_dryrun.py was used to validate the workflow before pushing. Results:

• ✅ ci.yml is valid YAML (parsed by PyYAML)
• ✅ 7 jobs (a 5-version pytest matrix plus 6 single-run jobs)
• ✅ meta-test correctly depends on pytest (Kahn's algorithm topo sort)
• ✅ matrix.python-version substituted correctly into ${{ ... }} expressions
• ✅ if: always() preserved on the upload-artifact step
• ✅ with: arguments (e.g., cache: pip, cache-dependency-path: pyproject.toml) preserved

The CI workflow's run: blocks are identical to commands we have already executed successfully on the host:

CI step	Local result
python -m pip install -e .	tmaudit installed ✅
python -m pip install pytest scipy	deps installed ✅
python -m pytest -m "not slow"	246 passed, 1 skipped in ~5s ✅
python _check_all_regressions.py	14/14 bugs caught ✅
python _check_github_templates.py	44/44 OK ✅
python -m src.tmaudit audit-all ...	audit-report.md generated ✅
python -m src.tmaudit audit-all ...	report uploaded as artefact ✅
python _build_pyz.py	tmaudit.pyz built ✅
python tmaudit.pyz list	5 papers listed ✅
python -m pyflakes src/tmaudit/ tests/	0 errors ✅

What _act_* cannot simulate

• actions/setup-python@v5 cannot actually install a different Python version (the host has only one Python; the matrix includes 3.9–3.13). The simulator prints a warning but does not fail.
• actions/upload-artifact@v4 is simulated by copying files to _act_run/artifacts/<name>/.
• The Docker isolation that real act provides is not present; steps run on the host shell.

Pre-push hook recommendation

To enforce CI validation before every push:

# .git/hooks/pre-push
python _check_yaml.py && python _act_dryrun.py

The _check_yaml.py script validates ci.yml is parseable and has the expected structure (catches YAML syntax errors before pushing).

Full evidence

For the full simulation transcript (including the 5 sandbox issues encountered and how they were worked around), see act_summary.md.