Self-Correction Can Amplify Hallucinations: Fact-Level Repair with Graph-Based Evidence Routing in Multimodal Generation

EMNLP 2026 (Main Conference)

Overview of TIGER (Figure 1): generate an initial output, extract fact graphs from the input and the output independently, rank claims by risk, and locally repair the selected high-risk facts for T rounds with a frozen backbone.

🌌 Introduction

TIGER is an inference-time framework that repairs faithfulness hallucinations in multimodal generation — fluent outputs that contain specific facts the input does not support — while keeping the backbone frozen.

Existing self-correction methods (Volcano, Woodpecker, DeGF) generate feedback by jointly conditioning on the input and the current output. We show that this design lets hallucinated claims bias the model's reading of the input, so naive self-correction can make the output worse (CHAIR_s .070 → .080 on COCO). TIGER redesigns the feedback loop in two steps:

1. Atomic projection. The input and the current output are extracted independently into structured fact graphs, an observation graph $G_{\mathbf{X}}$ and a claim graph $G_{\mathbf{Y}_t}$, so unsupported claims in the output cannot influence how the input is read.
2. Deterministic fact-level risk. Every claim receives a risk score $r(f) = (1 - s(f)) + \lambda, c(f)$ from its coreference-propagated support and its conflict with the observation graph. Risk is rankable, so TIGER repairs only the top-$\lceil \alpha N \rceil$ facts per round under a fixed budget, and the expected total risk decreases geometrically to an explicit floor (Theorem 3.1).

  

Left (Figure 2): free-form generation re-introduces co-occurring but absent objects (a boat behind the wakeboarder); atomic enumeration does not, and lowers the co-occurrence hallucination rate on every cue pair. Right (Figure 6a): on 1,000 COCO images with an absent co-occurring object b, joint feedback (L1) and text feedback (L2) mention b in 5.0% and 4.0% of samples, TIGER's atomic feedback (L3) in 2.0% (95% Wilson CIs).

TIGER supports four text-output cross-modal paths — image→text, image+text→text, audio→text, and video→text — with a single extraction template, and works with open-source and proprietary backbones.

📰 News

• [2026-08] Camera-ready code release: the tiger package with the full repair loop, the python -m tiger.eval entry point, the experiment configs, SLURM launchers, scoring and plotting scripts used in the paper (see docs/REPRODUCE.md).
• [2026-08] TIGER is accepted to the EMNLP 2026 main conference.

📊 Results

TIGER reduces unsupported content while preserving task quality across five benchmarks and five backbones (Qwen2.5-Omni-7B, LLaVA-1.5-7B, GPT-5.5, Gemini 3.5 Flash, Claude Haiku 4.5).

Benchmark (path)	Metric	Frozen	Best baseline	TIGER
COCO (image→text)	CHAIRs ↓ / BERTScore ↑	.070 / .588	.060 / .637	.050 / .643
AMBER (image→text)	Disc. Acc ↑ / CHAIRg ↓	77.0 / 5.0	79.7 / 4.8	82.0 / 4.3
MMHal-Bench (image+text→text)	BERTScore ↑	.703	.742	.756
Clotho (audio→text)	AEHR ↓ / CLAP ↑	.803 / .332	.777 / .364	.757 / .376
VideoHallucer (video→text)	HallucRate ↓ / Paired ↑	.015 / .170	.015 / .189	.010 / .198

  

Left (Figure 3): CHAIR_s reduction on three proprietary API backbones. Right (Figure 4): feedback-paradigm ablation on COCO — naive joint feedback (L1) amplifies hallucination; atomic projection (L2) and deterministic risk ranking (L3, TIGER) are both needed.

Hyperparameter sensitivity and inference cost (Figures 5 and 8)

  

Left: CHAIR_s on COCO when varying T, α, λ one at a time (default T=5, α=0.2, λ=0.5). Right: wall-clock per COCO sample; TIGER at T=5 is faster than the other iterative methods, and T=3 or risk-based early stopping recovers most of the gain at about two thirds of the cost.

⚙️ Installation

git clone https://github.com/kzhao5/tiger.git
cd tiger
pip install -r requirements.txt
pip install -e .
python -m spacy download en_core_web_sm          # used by the text-side extractor
# optional extras
pip install -e ".[api]"       # GPT-5.5 / Gemini / Claude backbones
pip install -e ".[metrics]"   # text and audio scoring (jiwer, rouge-score, sacrebleu, librosa, panns-inference)
pip install "clip @ git+https://github.com/openai/CLIP.git"   # CLIP verifier for the BoN+CLIP baseline

Requirements: Python ≥ 3.10, PyTorch ≥ 2.1, transformers ≥ 4.46 (with Qwen2_5OmniForConditionalGeneration). A single GPU with ≥ 24 GB VRAM runs Qwen2.5-Omni-7B (~16 GB of weights, downloaded automatically on first use).

Smoke test without a GPU or model download (uses a dummy backbone and checks the run-directory layout):

python -m tiger.eval --config configs/experiment/smoke_dummy.yaml --mode tiger --phase smoke_test

🚀 Quick Start

python examples/demo.py

The demo loads examples/sample_image.jpg, runs the full pipeline — extract $G_{\mathbf{X}}$ → generate $\mathbf{Y}0$ → extract $G{\mathbf{Y}_t}$ → score risk → repair for $T$ rounds — and prints the repaired description together with the final risk and graph sizes.

Programmatic use

from tiger import (
    GenerationSettings, PlannerConfig, QwenOmniBackbone,
    TigerInferenceEngine, UnifiedSample,
)
from tiger.schemas import TaskType

backbone = QwenOmniBackbone(model_name="Qwen/Qwen2.5-Omni-7B", enable_audio_output=False)
engine = TigerInferenceEngine(
    backbone=backbone,
    planner_config=PlannerConfig(max_steps=5),                 # T repair rounds
    generation_settings=GenerationSettings(max_new_tokens=128),  # temperature 0.7, top-p 0.9 (Table 11)
    alpha_batch=0.2,        # repair the top ceil(alpha * N) facts per round, N = |G_Y_t|
    lambda_conflict=0.5,    # conflict weight in r(f) = (1 - s) + lambda * c
)

sample = UnifiedSample(
    sample_id="my_sample",
    task_type=TaskType.IMAGE_TEXT_TO_TEXT_FREEFORM,
    prompt="Please describe this image in detail.",
    image_paths=["path/to/image.jpg"],          # or audio_paths=[...] / video_paths=[...]
)

out = engine.run(sample, mode="tiger")
print(out.prediction.text)                       # repaired output Y_T
print(out.risk, len(out.observation_graph.facts), len(out.claim_graph.facts))

out also carries the observation graph, the final claim graph, the per-fact assessments (support, conflict, risk), and the per-round history in out.planner_state, so every edit is traceable to the evidence that triggered it.

What the code implements

The engine follows Algorithm 1 of the paper line by line, with the prompts of Appendix B used verbatim:

Paper	Code
$G_{\mathbf{X}} = \Phi(\mathcal{P}{\text{ext}}, \mathbf{X})$ once; $G{\mathbf{Y}t} = \Phi(\mathcal{P}{\text{ext}}, \mathbf{Y}_t)$ per round (Eq. 4)	FactExtractor.extract_observation_graph / extract_claim_graph in tiger/evidence.py
$\mathrm{sim}(f,g)$ = mean of per-field clipped cosine (Eq. 11–12); $s_0(f)$ (Eq. 13); coreference propagation with decay $\gamma$ over $K$ hops (Eq. 14)	fact_similarity, local_support, propagated_support
$\mathrm{conflict}(f,g) = \tfrac12(\sigma_s+\sigma_p)(1-\sigma_o)$, $c(f)$ (Eq. 15–16)	fact_conflict, conflict_score
$r(f) = (1 - s(f)) + \lambda,c(f)$ (Eq. 8 / 17)	assess_candidate
$\Psi_\alpha$: top-$\lceil \alpha N \rceil$ facts, $N = \lvert G_{\mathbf{Y}_t}\rvert$ (Eq. 9)	repair loop in tiger/pipeline.py
$\mathbf{Y}{t+1} = \Phi(\mathcal{P}{\text{refine}}, \mathbf{X}, \mathbf{Y}_t, \mathcal{F}_t)$ (Eq. 2)	TigerInferenceEngine._repair_batch
Shared sampling decoder for $\mathcal{P}{\text{gen}}$, $\mathcal{P}{\text{ext}}$, $\mathcal{P}_{\text{refine}}$: temperature 0.7, top-p 0.9 (Table 11)	GenerationSettings defaults
Loop exit when no remaining fact has risk ≥ 0.3; optional early stopping on the normalized risk (Appendix E.3)	planner/early_stop options

Hyperparameters

Argument	Default	Paper symbol	Meaning
planner_config.max_steps	5	$T$	Number of repair rounds.
alpha_batch	0.2	$\alpha$	Fraction of claim facts repaired per round (0 = single highest-risk fact).
lambda_conflict	0.5	$\lambda$	Weight of the conflict term in the risk.
gamma	0.7	$\gamma$	Decay of support propagated along coreference edges.
max_hops	3	$K$	Coreference neighborhood size for support propagation.
k_repair	1	–	Candidate rewrites per round; the lowest-risk one is kept (the paper uses 1).

The defaults are the free-form operating point of the paper (COCO, AMBER, Clotho). For short-answer QA (MMHal-Bench, VideoHallucer) the paper uses max_steps=3, alpha_batch=0 (single-fact repair), and lambda_conflict=1.5; both operating points are in configs/experiment/.

Modes

engine.run(sample, mode=...) and python -m tiger.eval --mode ... default to "tiger". The other modes implement the baselines and ablations of the paper on the same backbone: frozen, volcano, woodpecker, degf, baseline_scalar / visualprm / cycle_reward (best-of-N), and the ablations tiger_feedback_raw (L1), tiger_feedback_text (L2), flat_baseline, tiger_no_graph, tiger_no_gy, tiger_no_lambda, tiger_random_batch.

🔬 Reproducing the Paper

python -m tiger.eval --config configs/experiment/coco_i2t.yaml --mode tiger --seed 42
python scripts/eval/score_predictions.py outputs/seed42/image_text_to_text_freeform/Qwen2.5-Omni-7B/*coco*

• configs/experiment/ — one YAML per benchmark and backbone (Tables 2–4, 14), sweeps/ for Figure 5 and Table 17, analysis/ for the unified configuration, extraction perturbation, and early stopping (Appendices E.3, F.2, F.4).
• scripts/slurm/ — single-GPU launcher and the method × seed matrix.
• scripts/eval/ — CHAIR / BERTScore, CLAP / AEHR, VideoHallucer, AMBER scoring and table builders (Table 12).
• scripts/data/ — SCS-1000 probe-set and VideoHallucer manifest builders.
• scripts/plots/ — figure scripts.

Step-by-step instructions, data layout, and the mapping from every table and figure to its config are in docs/REPRODUCE.md.

🧩 Repository Layout

tiger/
  pipeline.py      TigerInferenceEngine: Algorithm 1 (extract, score, select, repair) + baseline modes
  evidence.py      P_ext prompts, G_X / G_Y extraction, sim / support / conflict / risk (Appendix C)
  eval.py          python -m tiger.eval entry point (run directories, metrics, compute cost)
  datasets.py      jsonl / Hugging Face loaders -> UnifiedSample
  models.py        GenerationSettings, BaseBackbone, QwenOmniBackbone, build_backbone
  backbones/       LLaVA-1.5, Azure OpenAI, Gemini, Claude backbones
  schemas.py       UnifiedSample, Fact, ObservationGraph, ClaimGraph, InferenceOutput
  planner.py       PlannerConfig (number of repair rounds)
  tools.py         Deterministic modality-specific extractors (Grounding DINO, PANNs, Stanza; Appendix F.5)
  verifiers.py     CLIP / cycle-consistency verifiers used by the best-of-N baselines
  metrics.py       Per-sample metric helpers
configs/           base.yaml + experiment configs (Table 11 operating points)
scripts/           slurm/, eval/, data/, plots/
data/              released manifests (media paths relative to dataset.media_root)
examples/          demo.py, sample_image.jpg
docs/REPRODUCE.md  experiment-by-experiment reproduction guide
assets/            figures used in this README

🔌 Adding a Backbone

Subclass tiger.models.BaseBackbone, implement generate(...) (and extract_facts_prompt(...) if the backbone should also serve as the fact extractor), then register it in build_backbone():

from tiger.models import BaseBackbone
from tiger.schemas import CandidateOutput

class MyBackbone(BaseBackbone):
    def generate(self, sample, prompt, settings):
        ...
        return CandidateOutput(text=generated_text)

Every prompt used by TIGER ($\mathcal{P}{\text{gen}}$, $\mathcal{P}{\text{ext}}$, $\mathcal{P}_{\text{refine}}$) is model-agnostic and listed verbatim in Appendix B of the paper; the same templates are used for the open-source and the API backbones.

📖 Citation

@inproceedings{zhao2026tiger,
  title     = {Self-Correction Can Amplify Hallucinations: Fact-Level Repair with Graph-Based Evidence Routing in Multimodal Generation},
  author    = {Zhao, Kaixiang and Yu, Tianrun and Huang, Shawn and Jenkins, Porter and Dong, Yushun and Hughes, Amanda},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026}
}

🤝 Acknowledgments

TIGER builds on Qwen2.5-Omni, Sentence-BERT, Grounding DINO, PANNs, and Stanza. We thank the authors of Volcano, Woodpecker, and DeGF for releasing their prompts and code, which we re-implemented on a shared frozen backbone for the comparisons in the paper.

📄 License

This project is released under the MIT License.