Give text-only LLMs eyes. A Pi Agent skill + zero-dependency Python CLI that adds image understanding and document parsing (OCR, tables, formulas, PDF → Markdown) to any text-only model such as DeepSeek, using free-tier third-party multimodal APIs.
《慧眼》，灵感源自 1993 年的流行歌曲《雾里看花》。 作者在使用 DeepSeek 的过程中，仿佛听到它发出这样的呼唤：“借我借我一双慧眼吧，让我把这纷扰看得清清楚楚……”。 如今有了“慧眼-Skill”，我终于可以对 DeepSeek 说：“/multimodal-skill——给你戴上‘慧眼’，开工！”
DeepSeek (and many coding models) are text-only — they cannot see. When you paste a screenshot, a PDF, or a chart into your agent, the model has no way to understand it. This skill is that missing bridge:
screenshot / PDF / image
│
▼
mm_cli.py (Python stdlib only — no pip install, no venv)
│
├─ document → Markdown (PaddleOCR-VL / MinerU / DeepSeek-OCR)
└─ image → text (GLM-4V-Flash / Qwen3-VL / qwen-vl-max)
│
▼
Markdown/text → back into the LLM's context
The model reads the extracted Markdown as its "retina": faithful OCR and layout parsing are delegated to specialist APIs (a 0.9B document VLM beats frontier models on faithful parsing — OmniDocBench), while the text model handles semantics.
- Zero dependencies — one
mm_cli.py, pure Python standard library (urllib/json/base64). Nopip install, norequirements.txt, no venv. - Free-first — every default provider has a generous free tier: PaddleOCR official API (3,000 pages/day/model), MinerU (free, no key), Zhipu GLM-4V-Flash (fully free), plus SiliconFlow & DashScope fallbacks.
- China-network friendly — all default providers are mainland-reachable; no proxy required. (Gemini/Mistral/HuggingFace are blocked from CN — dropped.)
- Self-healing — model deprecation, rate limits (429), queue-full, and bad
keys are handled automatically: candidate-model rotation, provider fallback
chains, backoff, and a short-lived failure memory. See
references/troubleshooting.md. - Format-aware routing — txt/md/csv/tsv and local HTML are parsed locally
with zero model cost (stdlib encoding detection,
csv→ Markdown tables,html.parser→ Markdown); docx/xlsx/pptx auto-route to MinerU as chain head; PDF/images keep the PaddleOCR-first chain. Seereferences/formats.md. - Content-addressed caching —
sha256(file + prompt + model)keys, TTL (30d documents / 24h images), LRU eviction (2,000 entries / 2 GB). Saves quota, makes retries free. - Clean output discipline — results to stdout, diagnostics to stderr, exit codes 0–6 for agent automation.
- LLM-facing extraction metadata — every
doc parseresult carries a factual<!-- mm-meta: {...} -->header (format, mode, provider, model, pages, stats, over) so the consuming LLM knows exactly who parsed the file and can decide trust/verification itself. Never fabricated confidence scores. - Output guardrails — every string handed to the LLM is checked against configurable limits (bytes / lines / single-line / estimated tokens, default 64K tokens). On over-limit the full text is not emitted: it is written to a UTF-8 file and the path is returned, so the LLM reads/greps snippets with its own harness tools. Never silently truncated.
doc parse auto-detects the file type (magic bytes first, extension
fallback) and routes accordingly. Manual --provider bypasses routing
(debug only).
| Format | Detection | Route / chain head | Local zero-model parse | Notes |
|---|---|---|---|---|
PDF (.pdf) |
%PDF magic |
paddleocr → mineru → siliconflow | No | PaddleOCR excels at scans/layout; ≤100 pages/file (PaddleOCR), ≤20 pages/10MB (MinerU flash) |
| Images PNG/JPEG/WebP/GIF/BMP | magic bytes | paddleocr → mineru → siliconflow | No | Parsable as single-page docs; Q&A via image ask |
TIFF (.tif/.tiff) |
II*\0 / MM\0* magic |
paddleocr → mineru → siliconflow | No | Reported upstream as image/tiff |
Word/Excel/PPT (.docx/.xlsx/.pptx) |
PK zip + [Content_Types].xml |
mineru (other providers lack this capability and are filtered out) | No | MinerU officially supports Office; flash ≤10MB/20 pages; convert large files to PDF |
Legacy Office (.doc/.xls/.ppt) |
OLE2 D0CF11E0 magic |
No provider support → clear error with convert-to-docx/pdf hint | No | Avoid wasting quota on unsupported formats |
Plain text .txt/.md/.tsv/.log/.json/.yaml etc. |
no magic + non-binary heuristic | local read (zero quota, zero latency, data never leaves the machine) | ✅ | Encoding auto-detection: UTF-16 (BOM/heuristic) → UTF-8 → GB18030 → Latin-1 |
| CSV/TSV | text + extension | local csv → Markdown table |
✅ | csv.Sniffer auto-detects delimiter; handles quoted commas/newlines |
Local HTML (.html/.htm) |
text + extension | local html.parser → Markdown |
✅ | Headings/lists/tables/links/images/code blocks; falls back to mineru when extraction is too short (JS-rendered pages) |
| Unknown | no magic, no extension match | text probe first (printable ratio over whole file) → error only if that fails | probe | Error message lists all supported formats |
Every result carries a <!-- mm-meta: {...} --> header (meta field with
--json): format (group), mode (local = deterministic local parse /
model = vendor model), provider/model (actual parser), pages (when
the vendor reports it), stats (bytes/lines/tokens measurements) and over
(limit flags). When over-limit, paths.result points to the exported UTF-8
file (and paths.source to the original local file). The skill reports
facts only — confidence judgment and whether to suggest manual verification
are left to the consuming LLM.
| Requirement | Check |
|---|---|
| Python 3.8+ | python3 --version |
| Pi Agent | pi --version |
| At least one API key | see Configuration |
| Network | mainland China OK (defaults); no proxy needed |
Add to ~/.pi/agent/settings.json:
{
"packages": ["git:github.com/v587d/multimodal-skill"]
}Then run pi update --extensions (or restart Pi). The skill registers as
/skill:multimodal-skill.
git clone https://github.com/v587d/multimodal-skill.git
# Pi: mv multimodal-skill ~/.pi/agent/skills/multimodal-skill
# Generic: mv multimodal-skill <your_agent_skill_dir>/multimodal-skillCopy .env.example to .env in the skill directory (or export the variables).
At least one document provider + one image provider is recommended. The CLI
reads keys with priority: system environment > .env.
| Variable | Service | Used for | Where to get it |
|---|---|---|---|
PADDLEOCR_ACCESS_TOKEN |
PaddleOCR official API | document parsing (default) | https://aistudio.baidu.com/index/accessToken |
ZHIPUAI_API_KEY |
Zhipu GLM | image understanding (default) | https://open.bigmodel.cn/usercenter/proj-mgmt/apikeys |
MINERU_API_TOKEN |
MinerU precision API | document parsing (optional; flash mode needs no key) | https://mineru.net/apiManage |
SILICONFLOW_API_KEY |
SiliconFlow | image + document fallback | https://cloud.siliconflow.cn/account/ak |
DASHSCOPE_API_KEY |
Alibaba DashScope | image + document (optional) | https://bailian.console.aliyun.com/ |
python3 <skill_dir>/scripts/mm_cli.py config open # creates & opens ~/.config/multimodal-skill/config.jsonBeyond provider chains and model candidates, the limits section guards every
string handed to the LLM (defaults; all configurable):
| Check | Key | Default | Purpose |
|---|---|---|---|
| Total bytes | max_bytes |
5 MB | overall size cap |
| Total lines | max_lines |
20000 | guards huge line counts |
| Single-line bytes | max_line_bytes |
4096 | guards minified/base64 blobs |
| Estimated tokens | max_tokens |
64K | context budget (CJK-weighted estimator, exact CJK count via small translate table — benchmarked ~96ms @ 5MB) |
| Input hard cap | hard_max_bytes |
20 MB | stat pre-check before reading the file |
On over-limit the full text is not emitted: the CLI writes it to a UTF-8
file and returns the path (meta.paths.result; plus meta.paths.source for
local text inputs), so the consuming LLM can read/grep snippets with its own
tools. cache clear also cleans the export directory.
Put paid/any model IDs first in a pipeline's model list to prefer them
(e.g. "image_models": ["glm-4.5v", "glm-4v-flash"] — paid first, free
fallback). See config.example.json for the full
template (JSONC comments allowed). Quick look: mm_cli.py providers.
Note: the provider list is fixed in code (each provider speaks a different protocol); configuration can only override model candidates and default chains — it cannot add providers.
CLI=python3 <skill_dir>/scripts/mm_cli.py
# 0. Health check — connectivity, keys, models
$CLI doctor
# 1. Paste a screenshot into Pi TUI → you get a local path → ask about it
$CLI image ask /path/to/screenshot.png "What error is shown here?"
# 2. Parse a document (PDF/scanned/image) into Markdown
$CLI doc parse report.pdf --pages 1-20 --out report.md
# 3. Local text/CSV/HTML is parsed locally — zero quota, zero latency
$CLI doc parse notes.csv # → Markdown table
$CLI doc parse page.html # → Markdown via html.parser
# 4. Zero-config fallback (MinerU flash, no key needed)
$CLI doc parse scan.pdf --provider mineru
# 5. Over-limit output is exported to disk; the CLI returns the path
$CLI doc parse huge.log # → 输出超限，返回落盘路径，由 LLM 自行 grepFull command reference: references/api.md.
| Provider | Pipeline | Free tier | Reachable from CN |
|---|---|---|---|
| PaddleOCR official API | document | 3,000 pages/day/model (async job API) | ✅ |
| MinerU | document | flash: free, no key, ≤20 pages/10MB; precision: 1,000 pages/day high-priority | ✅ |
| SiliconFlow DeepSeek-OCR | document | measured zero-cost on free credit | ✅ |
| Zhipu GLM-4V-Flash | image | fully free (GLM-4.6V-Flash free, busy at peak) | ✅ |
| SiliconFlow Qwen3-VL | image | pay-per-use (cheap) | ✅ |
| DashScope qwen-vl-max/ocr | image + document | ~1M tokens/model free (90 days) | ✅ |
- Keys live in
.env(permission 600) or environment variables — never hardcoded;.envandconfig.jsonare gitignored. - Free tiers may train on submitted data (a common policy across all
providers) — do not send confidential documents/screenshots to free tiers.
Use paid models (via
config.json) for sensitive material. - The CLI never auto-executes anything from API responses; outputs are plain text/Markdown on stdout.
- Error messages may echo provider responses — don't paste secrets into files you send to parsing APIs.
Cache is disk-only, file-backed — no memory state, no daemon, no load-on-start / flush-on-exit cycle. Every CLI invocation is a fresh process that reads and writes entries directly on disk (write-through):
- Key = content-addressed
sha256(file bytes + pipeline + provider + model + prompt + params); file name is the key. - TTL checked lazily on read: 30 days for documents, 24 h for images (
--ttl/--no-cacheto override). - LRU eviction by directory scan on write: 2,000 entries / 2 GB cap, oldest-accessed dropped first.
In short: a cache-flavored file operator — crash-safe, survives restarts, and shared across sessions (a parse cached in one session is a quota saved in the next).
Two pipelines (industry best practice: keep faithful parsing separate from open-ended vision):
doc parse— specialist document parsers → Markdown (tables, LaTeX formulas, reading order). PaddleOCR-VL-1.6 is OmniDocBench SOTA class.image ask— OpenAI-compatible VLM chat for screenshots/photos/charts.
Every call runs through an adaptive chain: try model candidates in order → on model-deprecation/rate-limit/auth failure, rotate or switch provider → aggregate error with a meaningful exit code (2 usage / 3 auth / 4 rate / 5 model / 6 network).
Common issues (PaddleOCR slow queue, GLM 429s, MinerU upload signature,
Chinese output garbling, cache quirks): references/troubleshooting.md.