diff --git a/AGENTS.md b/AGENTS.md index 6a1f10d6..45ed82a9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,7 +1,7 @@ # Repository Guidelines ## Project Structure & Module Organization -`src/Undefined/` contains the main runtime package. Core areas include `ai/`, `services/`, `skills/`, `cognitive/`, `memes/`, `knowledge/`, `api/`, `webui/`, `config/`, `mcp/`, and `automations/` (`AutomationService` runtime plus JSON storage); media-facing integrations live in `arxiv/`, `bilibili/`, `github/`, and `attachments/`. `src/Undefined/deploy/` is the containerized one-shot deployment path (`uv run deploy`, templates under `deploy/templates/`) and does not take part in runtime. `tests/` holds the pytest suite. +`src/Undefined/` contains the main runtime package. Core areas include `ai/`, `services/`, `skills/`, `cognitive/`, `memes/`, `knowledge/`, `api/`, `webui/`, `config/`, `mcp/`, and `automations/` (`AutomationService` runtime plus JSON storage); media-facing integrations live in `arxiv/`, `bilibili/`, `github/`, `jm/`, and `attachments/`. `src/Undefined/deploy/` is the containerized one-shot deployment path (`uv run deploy`, templates under `deploy/templates/`) and does not take part in runtime. `tests/` holds the pytest suite. > Single source of truth for the module-level directory tree: [docs/development.md](docs/development.md). This file, `CLAUDE.md`, and `ARCHITECTURE.md` only keep overviews — update the tree first when the layout changes. `apps/undefined-console/` is the Tauri + Vite management client and `apps/undefined-chat/` is the native-first Tauri + React 19 chat client (both connect to the same Management/Runtime services), while `code/NagaAgent/` remains a git submodule and should be updated deliberately, with upstream syncs kept separate from repo-local changes. Runtime and generated state primarily lives under `data/`, `logs/`, and `dist/`; the root `knowledge/` directory stores knowledge-base data rather than application code. Prefer editing source files and docs over generated outputs unless the task is explicitly about runtime state. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index cd4fbfac..e0e86a58 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -40,6 +40,12 @@ graph TB GitHubSender["sender.py
图片卡片发送
• 头像 • 简介 • 统计"] end + subgraph JmModule["JM 模块 (jm/)"] + JmParser["parser.py
车号解析
• JM+数字 • 禁漫链接 • 分享卡片"] + JmDownloader["downloader.py
整本下载合成
• jmcpy 逐章下载
• 合并为加密 PDF"] + JmSender["sender.py
合并转发发送
• 信息/密码/PDF • 失败回退"] + end + subgraph SecurityLayer["安全防线 (services/)"] SecurityService["SecurityService
安全服务
• 注入攻击检测
• 速率限制
[security.py]"] InjectionAgent["InjectionResponseAgent
注入响应生成
[injection_response_agent.py]"] @@ -228,8 +234,13 @@ graph TB GitHubParser -->|"仓库ID"| GitHubClient GitHubClient -->|"public仓库信息"| GitHubSender GitHubSender -->|"发送图片卡片"| OneBotClient - - MessageHandler -->|"2.7 自动化"| AutomationService + + MessageHandler -->|"2.7 非命令自动管线"| JmParser + JmParser -->|"车号"| JmDownloader + JmDownloader -->|"加密 PDF"| JmSender + JmSender -->|"发送合并转发"| OneBotClient + + MessageHandler -->|"2.8 自动化"| AutomationService AutomationService -->|"读写"| AutomationStorage AutomationStorage -->|"异步读写"| IOUtils @@ -850,6 +861,7 @@ description: 从 PDF 文件中提取文本和表格,填写表单。当用户 | **Bilibili** | `bilibili.auto_extract_enabled`, `bilibili.cookie`, `bilibili.prefer_quality` | B站视频自动提取与下载 | | **arXiv** | `arxiv.auto_extract_enabled`, `arxiv.max_file_size`, `arxiv.auto_extract_max_items` | arXiv 论文自动提取、搜索与 PDF 发送 | | **GitHub** | `github.auto_extract_enabled`, `github.request_timeout_seconds`, `github.request_retries`, `github.auto_extract_max_items` | GitHub public 仓库自动提取与图片卡片发送 | +| **JM(禁漫)** | `jm.auto_extract_enabled`, `jm.max_file_size`, `jm.max_chapters`, `jm.auto_extract_max_items` | JM 车号自动提取、整本加密 PDF 与合并转发 | | **思考链** | `*.thinking_enabled` | 思维链支持 | | **思维链兼容** | `*.thinking_tool_call_compat` | 思维链 + 工具调用兼容 | | **WebUI** | `webui.url`, `webui.port`, `webui.password` | 配置控制台 | diff --git a/CHANGELOG.md b/CHANGELOG.md index 30aa2a4d..9b71e641 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,21 @@ +## v3.18.0 JM 漫画自动提取 + +本版本新增禁漫自动提取管线:消息里出现 `JM` 车号或禁漫链接时,自动获取本子详情、把全部章节合成为一个带密码的 PDF,以「本子信息 / 解密密码 / PDF 文件」三节点合并转发发送(文件只放在转发里,群聊下同时进入群文件列表),并新增与 `arxiv_paper` 同构的 `jm_book` 工具供 AI 按需取详情或附件。功能默认关闭,密码随机生成且不写进历史。 + +- 新增 `jm` 自动处理管线(`src/Undefined/skills/pipelines/jm/`,`order = 12`)与 `src/Undefined/jm/`(车号解析、jmcpy 客户端设置、整本下载合成、合并转发发送),依赖新增并固定到 `jmcpy>=0.1.3`。 +- 触发规则:`JM` 前缀加 5–8 位车号(`JM1114751` / `jm 1114751` / `jm:1114751` / `jm-1114751`,大小写不敏感),以及主机名含 `18comic` 或 `jmcomic` 的链接(`/album/`、`/photo/`、`?id=`),QQ 分享卡片里的链接同样解析,车号与链接混排时按原文出现顺序取(默认只取第一个);分享卡片结构异常(`data` 不是对象、JSON 损坏)只跳过该段,不影响同一条消息里其它段落的检测。前缀必须落在词边界上(`xxjm1234567`、`t.me/jm1234567`、`video_jm1234567.mp4` 都不触发),数字后不跟数字(9 位以上不会被截成 8 位);裸数字不触发,避免群号、时间戳误报。 +- 下载与合成:按章节 `order` 顺序逐章下载(`[jm].max_chapters` 可限制),`decode=False` 取服务端原始字节无损落盘,自己解扰后每页只编码一次 JPEG(`[jm].image_quality`,色度 4:4:4)并直接作为 PDF 图像数据;页尺寸统一按 `[jm].pdf_dpi` 换算(页宽 = 像素宽 ÷ DPI × 72),页像素始终不缩放,画质上限由站点源图分辨率决定。不再复用 `jmcpy.imaging.write_pdf`:jmcpy 的下载 API 一章只能产出一个 PDF,而这里要把多个章节合进同一个文件;`jmcpy>=0.1.3` 修掉了追加页退回默认 72 DPI、同份 PDF 内页面尺寸不一致的问题,并把页内 JPEG 色度采样默认值改成 4:4:4。 +- 逐页容错:单页解码失败只跳过该页并计入「下载失败 N 页」,其余页照常出 PDF,全部页失败才算空结果;动图(`.gif`)不做分块还原,与 jmcpy 的 `UNDECODED_SUFFIXES` 语义一致。 +- 体积语义:`[jm].max_file_size` 先按各章节原始图片累计预判并停止下载,组装时再按**编码后字节**累计复核,超过即提前中止、PDF 不落盘;因此它只是近似内存上限(JPEG 重编码可能比源图更大)。超过上限或没有下到任何页面时只发信息与状态两个节点,不发密码与文件,且信息里不展示 PDF 大小。 +- 发送结构固定为三个节点:本子信息(车号、标题、作者、章节数、标签、观看/点赞、页数、大小、简介预览,末行是可直接复制的 `JM<车号>`,不再输出站点链接)、PDF 解密密码(每次随机生成 8 位,字母表去掉 `0/O/1/l/I` 等形近字符)、PDF 文件(本地合成后随转发上传)。密码只出现在转发节点中,写入历史的摘要只说「密码见转发节点」,后续 AI 轮次不会复述密码。 +- 文件只放在转发里,不额外发独立文件消息:群聊下 NapCat 会把它作为群文件上传(`isGroupFile`、`busid=102`,元素里带 `fileId` / `fileMd5` / `fileSha1`),节点里的文件可以直接下载,同一个 PDF 因此也会出现在群文件列表里。 +- 投递语义:「投递结果未确认」(`delivery_uncertain`)与「文件传输错误」(`file_transfer_error`)不降级重发,直接上抛调用方——这两种情况下转发可能已经送达,换个 action 再发会造成真实重复投递;只有协议端明确拒绝才退化为「信息 + 密码两条普通消息 + 独立文件消息」(此时文件没有别的入口),该路径的密码消息写历史时只记「PDF 解密密码已单独发送」,上传也失败时再补一条不写历史的提示。 +- 失败与资源:车号不存在或接口不可用时发送一行 `JM 提取失败:<原因>` 并记录异常类型与堆栈;整本下载跑在模块专用线程池(同时最多 2 本)而不是事件循环默认执行器;任务目录在成功、超限、空结果与下载抛异常四条路径上都会清理;被取消时线程仍在写盘,改为等它跑完再删,不留下残留原图或未加密 PDF。 +- 新增 `jm_book` 工具(`src/Undefined/skills/tools/jm_book/`,与 `arxiv_paper` 同构):`output_mode=send`(默认,等价自动提取)、`uid`(只下载并注册**未加密** PDF 附件 UID,不发送消息,供 `file_analysis_agent` 用 `extract_pdf` / `describe_pdf_page` 解析)、`info`(只返回标题/作者/章节/标签/观看点赞/简介与链接,不下载);`callable.json` 把它共享给 `file_analysis_agent`,后者提示词里加了「车号/链接 → 先 `jm_book(output_mode="uid")`」的规则。 +- 新增 `[jm]` 配置段:`auto_extract_enabled`(默认 `false`)、`use_proxy`、`auto_extract_group_ids` / `auto_extract_private_ids`(空时跟随全局 access)、`auto_extract_max_items`(默认 1,上限 5)、`max_file_size`、`max_chapters`、`pdf_dpi`、`image_quality`、`download_concurrency`、`request_timeout`、`image_timeout`、`session_dir`(jmcpy 配置目录,可复用已保存的登录态);`[jm].use_proxy` 支持环境变量 `JM_USE_PROXY`。 + +--- + ## v3.17.2 侧写有序合并 本版本修复群聊与用户侧写并发更新时的乱序覆盖:同一实体按史官任务入队顺序逐次合并,每次都基于前一次提交后的完整侧写;不同实体继续并发处理,QQ 与绑定的 iLink 私聊也遵循同一用户顺序。 diff --git a/CLAUDE.md b/CLAUDE.md index 57f7373d..83a5bc14 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -70,6 +70,7 @@ bash scripts/install_git_hooks.sh | `arxiv/` | arXiv 论文解析、元信息获取、PDF 下载与发送 | | `bilibili/` | B 站链接/BV 解析、视频下载与发送 | | `github/` | GitHub public 仓库解析、API 获取与图片卡片发送 | +| `jm/` | 禁漫(JM / 18comic)车号解析、整本 PDF 下载合成与加密合并转发 | | `api/` | Runtime API / Management API 相关服务;路由拆分在 `api/routes/`,包含 `chat`、`cognitive`、`health`、`memes`、`memory`、`naga`、`system`、`tools` | | `webui/` | aiohttp 管理控制台;路由拆分在 `webui/routes/`,覆盖配置、日志、运行态、表情包与系统管理 | | `mcp/` | MCP 工具注册、连接与转换 | @@ -86,7 +87,7 @@ OneBot WebSocket → onebot/ → handlers/ → 附件登记 / 访问控制 / 表情包入库 → SecurityService(注入检测) → CommandDispatcher(斜杠指令,命中即结束后续处理) - → skills/pipelines(Bilibili / arXiv / GitHub 并行自动提取) + → skills/pipelines(Bilibili / Douyin / arXiv / GitHub / JM 并行自动提取) → Automations(pipeline 后接入;consume_ai_loop 时 await 并拦截对应 AI,否则后台执行并立刻放行;发生在 MessageBatcher 之前) → MessageBatcher(同 sender 短时合并;拍一拍/buffer 内 @bot 旁路) → AICoordinator → QueueManager(按模型隔离, 6 条车道) diff --git a/README.md b/README.md index d00f26d1..3545f4b0 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,8 @@ - **可选音乐工具集**:配置独立部署的 [lxmusic2api](https://github.com/69gg/lxmusic2api) 后,主 AI 可搜索歌曲/歌单、浏览热搜与排行榜、读取歌词/封面/评论、跨平台匹配并交付普通音频附件;完整 Track 通过任务内短引用流转,只暴露面向用户的高层 `music.*` 能力,不暴露下载作业生命周期接口。详见 [使用指南](docs/usage.md#音乐-music) 与 [配置说明](docs/configuration.md#4201-lxmusic2api-音乐服务)。 - **arXiv 论文提取、搜索与分析**:自动检测消息中的 arXiv 链接/标识并发送论文信息与 PDF;`arxiv_paper` 也可只返回 PDF 附件 UID,供 `file_analysis_agent` 做文本提取或指定页视觉分析;`arxiv_search` 负责论文检索。 - **GitHub 仓库卡片**:自动检测 GitHub 仓库链接或 `owner/repo` 仓库 ID,获取 public 仓库信息并发送简洁图片卡片,展示头像、简介、stars、forks、issues、contributors 等概览;API 与头像下载统一遵循 `[github].use_proxy`,头像以内联资源进入离线渲染器。 -- **自动处理管线**:Bilibili、arXiv、GitHub 等自动提取统一运行在 `skills/pipelines` 中,斜杠命令优先级更高;命令输入/输出会写入历史,非命令消息会并行检测和处理命中管线,结果通过统一发送层写入历史并登记附件 UID 后再进入 AI 回复。远程大附件超过 `[attachments].remote_download_max_size_mb` 时只登记 URL 引用,避免无界下载和缓存膨胀。 +- **禁漫(JM)本子提取**:自动检测 `JM`+车号或禁漫链接,下载全部章节并合成为一个 AES-256 加密 PDF,以「本子信息 / 解密密码 / PDF 文件」三节点合并转发发送(文件只在转发里,群聊下同时进入群文件列表),密码每次随机生成且不写入历史;`jm_book` 工具也可只返回本子信息或未加密 PDF 附件 UID,供 `file_analysis_agent` 解析。默认关闭,`[jm].auto_extract_enabled` 打开。详见 [配置说明](docs/configuration.md#4203-jm-禁漫自动提取)。 +- **自动处理管线**:Bilibili、arXiv、GitHub、JM 等自动提取统一运行在 `skills/pipelines` 中,斜杠命令优先级更高;命令输入/输出会写入历史,非命令消息会并行检测和处理命中管线,结果通过统一发送层写入历史并登记附件 UID 后再进入 AI 回复。远程大附件超过 `[attachments].remote_download_max_size_mb` 时只登记 URL 引用,避免无界下载和缓存膨胀。 - **同 sender 短时消息合并**:默认开启。连续发的多条消息会合并到同一轮 AI 调用,AI 一次看到全部意图自行识别"独立请求/修正/打断";告别"画猫→改成狗"的重复触发与回复打架。主提示词按 batcher 的"当前输入批次"语义适配,关闭该功能可能导致连续补充/修正消息与提示词不匹配,需要单独适配。可选投机预发送让用户停顿时 LLM 提前开跑、新消息可在未发出回复前取消,进一步压低响应延迟。详见 [docs/message-batching.md](docs/message-batching.md)。 - **思维链支持**:支持开启思维链,提升复杂逻辑推理能力。 - **高并发架构**:基于 `asyncio` 全异步设计,支持多队列消息处理与工具并发执行,轻松应对高并发场景。 diff --git a/apps/undefined-chat/package-lock.json b/apps/undefined-chat/package-lock.json index 56a283d7..836ece28 100644 --- a/apps/undefined-chat/package-lock.json +++ b/apps/undefined-chat/package-lock.json @@ -1,12 +1,12 @@ { "name": "undefined-chat", - "version": "3.17.2", + "version": "3.18.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "undefined-chat", - "version": "3.17.2", + "version": "3.18.0", "dependencies": { "@tauri-apps/api": "^2.3.0", "@tauri-apps/plugin-dialog": "^2.7.1", diff --git a/apps/undefined-chat/package.json b/apps/undefined-chat/package.json index 7e72f3fa..249ab812 100644 --- a/apps/undefined-chat/package.json +++ b/apps/undefined-chat/package.json @@ -1,7 +1,7 @@ { "name": "undefined-chat", "private": true, - "version": "3.17.2", + "version": "3.18.0", "type": "module", "scripts": { "tauri": "tauri", diff --git a/apps/undefined-chat/src-tauri/Cargo.lock b/apps/undefined-chat/src-tauri/Cargo.lock index 2714b18b..7fd69e38 100644 --- a/apps/undefined-chat/src-tauri/Cargo.lock +++ b/apps/undefined-chat/src-tauri/Cargo.lock @@ -5431,7 +5431,7 @@ dependencies = [ [[package]] name = "undefined_chat" -version = "3.17.2" +version = "3.18.0" dependencies = [ "futures-util", "keyring", diff --git a/apps/undefined-chat/src-tauri/Cargo.toml b/apps/undefined-chat/src-tauri/Cargo.toml index d1605b67..aa85a510 100644 --- a/apps/undefined-chat/src-tauri/Cargo.toml +++ b/apps/undefined-chat/src-tauri/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "undefined_chat" -version = "3.17.2" +version = "3.18.0" description = "Undefined native chat client" authors = ["Undefined contributors"] license = "MIT" diff --git a/apps/undefined-chat/src-tauri/tauri.conf.json b/apps/undefined-chat/src-tauri/tauri.conf.json index feb20b27..0cf1a429 100644 --- a/apps/undefined-chat/src-tauri/tauri.conf.json +++ b/apps/undefined-chat/src-tauri/tauri.conf.json @@ -1,7 +1,7 @@ { "$schema": "https://schema.tauri.app/config/2", "productName": "Undefined Chat", - "version": "3.17.2", + "version": "3.18.0", "identifier": "com.undefined.chat", "build": { "beforeDevCommand": "npm run dev", diff --git a/apps/undefined-console/package-lock.json b/apps/undefined-console/package-lock.json index edc76de9..c6c248a0 100644 --- a/apps/undefined-console/package-lock.json +++ b/apps/undefined-console/package-lock.json @@ -1,12 +1,12 @@ { "name": "undefined-console", - "version": "3.17.2", + "version": "3.18.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "undefined-console", - "version": "3.17.2", + "version": "3.18.0", "dependencies": { "@tauri-apps/api": "^2.3.0", "@tauri-apps/plugin-http": "^2.3.0" diff --git a/apps/undefined-console/package.json b/apps/undefined-console/package.json index ae67eb1d..9897006c 100644 --- a/apps/undefined-console/package.json +++ b/apps/undefined-console/package.json @@ -1,7 +1,7 @@ { "name": "undefined-console", "private": true, - "version": "3.17.2", + "version": "3.18.0", "type": "module", "scripts": { "tauri": "tauri", diff --git a/apps/undefined-console/src-tauri/Cargo.lock b/apps/undefined-console/src-tauri/Cargo.lock index 27357f05..c9d1ccb5 100644 --- a/apps/undefined-console/src-tauri/Cargo.lock +++ b/apps/undefined-console/src-tauri/Cargo.lock @@ -4063,7 +4063,7 @@ checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb" [[package]] name = "undefined_console" -version = "3.17.2" +version = "3.18.0" dependencies = [ "serde", "serde_json", diff --git a/apps/undefined-console/src-tauri/Cargo.toml b/apps/undefined-console/src-tauri/Cargo.toml index 29d3aea7..bba400fd 100644 --- a/apps/undefined-console/src-tauri/Cargo.toml +++ b/apps/undefined-console/src-tauri/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "undefined_console" -version = "3.17.2" +version = "3.18.0" description = "Undefined cross-platform management console" authors = ["Undefined contributors"] license = "MIT" diff --git a/apps/undefined-console/src-tauri/tauri.conf.json b/apps/undefined-console/src-tauri/tauri.conf.json index 07abfec4..464fef3d 100644 --- a/apps/undefined-console/src-tauri/tauri.conf.json +++ b/apps/undefined-console/src-tauri/tauri.conf.json @@ -1,7 +1,7 @@ { "$schema": "https://schema.tauri.app/config/2", "productName": "Undefined Console", - "version": "3.17.2", + "version": "3.18.0", "identifier": "com.undefined.console", "build": { "beforeDevCommand": "npm run dev", diff --git a/config.toml.example b/config.toml.example index f4f09a8b..1d0d7b6f 100644 --- a/config.toml.example +++ b/config.toml.example @@ -1572,6 +1572,49 @@ auto_extract_private_ids = [] # en: Max number of GitHub repositories to auto-process from one message. <=0 falls back to 3, >10 is clamped to 10. auto_extract_max_items = 3 +# zh: 禁漫(JM / 18comic)自动提取配置。命中后下载整本并合成为一个加密 PDF,以合并转发发送(信息 / 解密密码 / PDF 文件)。 +# en: JM (18comic) auto-extraction settings. On match, the whole book is downloaded, merged into one encrypted PDF, and sent as a merged forward (info / password / PDF file). +[jm] +# zh: 是否让禁漫接口与图片下载使用 [proxy] 代理地址。默认关闭。 +# en: Whether JM API and image downloads use proxy addresses from [proxy]. Disabled by default. +use_proxy = false +# zh: 是否启用自动提取(检测到 JM+数字车号或禁漫链接时自动下载并发送)。默认关闭。 +# en: Enable auto-extraction (auto-download and send when JM book IDs or JM links are detected). Disabled by default. +auto_extract_enabled = false +# zh: 自动提取功能的群聊白名单(空=跟随全局 access.allowed_group_ids)。 +# en: Group allowlist for auto-extraction (empty = follow global access.allowed_group_ids). +auto_extract_group_ids = [] +# zh: 自动提取功能的私聊白名单(空=跟随全局 access.allowed_private_ids)。 +# en: Private chat allowlist for auto-extraction (empty = follow global access.allowed_private_ids). +auto_extract_private_ids = [] +# zh: 单条消息最多自动处理几个车号。<=0 回退 1,>5 截断到 5;每项是一整本 PDF,所以默认只处理 1 个。 +# en: Max number of book IDs to auto-process from one message. <=0 falls back to 1, >5 is clamped to 5; each item is a whole-book PDF, so only one is processed by default. +auto_extract_max_items = 1 +# zh: PDF 体积上限(MB),超过则只发送信息节点(按编码后字节判定,合成中即会中止)。0=不限,<0 回退 100。 +# en: PDF size limit in MB; exceeding it sends the info node only (checked on encoded bytes, aborts mid-build). 0=unlimited, <0 falls back to 100. +max_file_size = 100 +# zh: 单本最多下载并合并几章(取前 N 章)。0=不限(全部章节),<0 回退 0。 +# en: Max chapters downloaded and merged per book (first N). 0=unlimited, <0 falls back to 0. +max_chapters = 0 +# zh: PDF 页面分辨率(DPI),所有页统一按它换算物理尺寸(页宽 = 像素宽 ÷ DPI × 72)。<=0 回退 150。页像素不做缩放。 +# en: PDF page resolution (DPI); every page uses it for its physical size (page width = pixels / DPI * 72). <=0 falls back to 150. Page pixels are never rescaled. +pdf_dpi = 150.0 +# zh: PDF 内页 JPEG 质量(1-100,色度 4:4:4,每页只编码一次)。<1 回退 95,>100 截断到 100;提高到 98/100 约增加 30%/50% 体积。 +# en: JPEG quality for PDF pages (1-100, chroma 4:4:4, exactly one encode per page). <1 falls back to 95, >100 is clamped to 100; 98/100 costs roughly 30%/50% more size. +image_quality = 95 +# zh: 图片下载并发数。<1 回退 8,>32 截断到 32。 +# en: Image download concurrency. <1 falls back to 8, >32 is clamped to 32. +download_concurrency = 8 +# zh: 接口请求超时(秒)。<=0 回退 20,>120 截断到 120。 +# en: API request timeout (seconds). <=0 falls back to 20, >120 is clamped to 120. +request_timeout = 20.0 +# zh: 图片下载超时(秒)。<=0 回退 60,>600 截断到 600。 +# en: Image download timeout (seconds). <=0 falls back to 60, >600 is clamped to 600. +image_timeout = 60.0 +# zh: jmcpy 的配置目录(登录态与会话文件所在目录,可指向已登录的 JMCPY_HOME)。留空使用平台默认目录。 +# en: jmcpy config directory (login session files; point it at an existing JMCPY_HOME). Empty uses the platform default directory. +session_dir = "" + # zh: Code Delivery Agent 配置(代码交付 Agent,在 Docker 容器中编写代码并打包上传)。 # en: Code Delivery Agent settings (writes code in Docker containers and delivers packaged results). [code_delivery] diff --git a/docs/configuration.md b/docs/configuration.md index 8f46a020..c1750999 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1113,9 +1113,54 @@ api_key = "replace-with-your-key" - GitHub API 与仓库所有者头像下载默认不使用代理;需要时设置 `[github].use_proxy = true`,两类请求都会使用 `[proxy]` 中的代理地址。头像下载后以内联资源交给离线卡片渲染器,浏览器不会额外联网。 - 自动提取失败日志会记录异常类型、`repr(exc)` 和堆栈,便于定位代理连接失败等 `str(exc)` 为空的异常。 +--- + +### 4.20.3 `[jm]` 禁漫自动提取 + +| 字段 | 默认值 | 说明 | 约束/回退 | +|---|---:|---|---| +| `use_proxy` | `false` | 禁漫接口与图片下载是否使用 `[proxy]` 中的代理地址 | | +| `auto_extract_enabled` | `false` | 是否自动提取 JM 车号或禁漫链接 | | +| `auto_extract_group_ids` | `[]` | 功能级群白名单 | 空时跟随全局 access | +| `auto_extract_private_ids` | `[]` | 功能级私聊白名单 | 空时跟随全局 access | +| `auto_extract_max_items` | `1` | 单条消息最多自动处理几个车号 | `<=0` 回退 `1`,`>5` 截断到 `5` | +| `max_file_size` | `100` | PDF 体积上限(MB),`0` 不限 | `<0` 回退 `100` | +| `max_chapters` | `0` | 单本最多下载并合并几章,`0` 不限 | `<0` 回退 `0` | +| `pdf_dpi` | `150.0` | PDF 页面分辨率(DPI),所有页统一按它换算物理尺寸 | `<=0` 回退 `150` | +| `image_quality` | `95` | PDF 内页 JPEG 质量(1-100,色度 4:4:4) | `<1` 回退 `95`,`>100` 截断到 `100` | +| `download_concurrency` | `8` | 图片下载并发数 | `<1` 回退 `8`,`>32` 截断到 `32` | +| `request_timeout` | `20.0` | 接口请求超时(秒) | `<=0` 回退 `20`,`>120` 截断到 `120` | +| `image_timeout` | `60.0` | 图片下载超时(秒) | `<=0` 回退 `60`,`>600` 截断到 `600` | +| `session_dir` | `""` | jmcpy 配置目录(登录态会话文件所在目录) | 空时用 jmcpy 的平台默认目录 | + +触发规则: +- 匹配 `JM` 前缀加 5–8 位车号:`JM1114751`、`jm 1114751`、`jm:1114751`、`jm-1114751`;前缀前必须是边界(`xxjm1234567` 不触发),数字后不跟数字(9 位以上不会被截成 8 位)。 +- 匹配主机名含 `18comic` 或 `jmcomic` 的链接,车号取自 `/album/`、`/photo/` 或 `?id=`。 +- 裸数字不触发,避免群号、时间戳等误报;QQ 分享卡片(`json` 消息段)里的链接同样会解析。 + +自动提取行为: +- 命中后发送一次外层合并转发,固定三个节点:**本子信息**(车号、标题、作者、章节数、标签、观看/点赞、页数、大小、简介预览,末尾是可直接复制的 `JM<车号>`)、**PDF 解密密码**(每次随机生成 8 位,字母表去掉 `0/O/1/l/I` 等形近字符)、**PDF 文件**(本地合成后随转发上传,群聊下会成为真正的群文件)。文件只走转发,不额外发独立文件消息。 +- 多章节本子会把全部章节按顺序下载后合成为一个 PDF(`max_chapters` 可限制章数),PDF 用 AES-256 加密,打开时需要密码。 +- 每页按 `pdf_dpi` 统一换算物理尺寸(`页宽 = 像素宽 ÷ DPI × 72`),下载与合成都不缩放像素:PDF 里的页就是站点源图的原始分辨率,画质上限由站点自己提供的图决定。 +- 每页只做一次有损编码:取服务端原始字节(无损落盘)→ 解扰 → JPEG(`image_quality`,4:4:4)→ 直接作为 PDF 的图像数据,中间不再多一代重编码。需要更高画质时可把 `image_quality` 提到 `98`/`100`(约 `+30%`/`+50%` 体积),源图本身的分辨率无法通过配置提升。 +- 单次合并转发里的 PDF 在内存中合成后写盘,所以 `max_file_size` 只是**近似**内存上限:先按各章节原始图片累计预判(超限即停止下载),组装时再按**编码后字节**累计复核,超过上限就提前中止、不发文件;PDF 是重编码 JPEG(4:4:4 / q95 可能比源 WebP 大),峰值可以略高于该值。 +- 密码只出现在转发节点里;写入历史的摘要只说明「密码见转发节点」,因此后续 AI 轮次能看到「发过什么」,但不会把密码复述出来。 +- 转发节点里的 PDF 随转发一起上传:群聊下 NapCat 会把它作为群文件上传(`busid=102`),所以同一个 PDF 也会出现在群文件列表里,节点里的文件可以直接下载(已用 14MB 的 PDF 实测)。文件只发一次,不额外发独立文件消息。 +- 转发本身被拒时退化为两条普通消息(信息、密码)+ 一条独立文件消息(此时文件没有别的入口,必须发出来);密码消息写进历史时只记「PDF 解密密码已单独发送」,不重复写摘要,上传彻底失败时再补一条不写历史的提示。 +- 「投递结果未确认」(`delivery_uncertain`)与「文件传输错误」(`file_transfer_error`)**不降级重发**,直接上抛给调用方:这两种情况下转发可能已经送达,换个 action 再发会造成真实重复投递(判据与 B 站发送器一致)。 +- 降级路径会把密码作为普通消息发出(转发发不出去时这是唯一能拿到密码的方式),因此这条消息的正文也会进入本地 INFO 日志的「内容摘要」,转发节点路径则不会。密码始终不进历史摘要与转发卡片之外的用户可见正文。 +- 下载量超过 `max_file_size` 或没有下到任何页面时,只发送信息与状态说明两个节点(此时信息里不展示 PDF 大小,因为还没有 PDF),不发密码与文件。 +- 车号不存在、接口不可用时发送一行 `JM 提取失败:<原因>`,日志记录异常类型与堆栈;车号与链接混排时按原文顺序取第一个,分享卡片结构异常只跳过该段。 +- 除了自动提取,`jm_book` 工具让 AI 也能按需处理车号:`output_mode=send`(默认,等价自动提取)、`uid`(只下载并注册**未加密** PDF 附件 UID,交给 `file_analysis_agent` 用 `extract_pdf` / `describe_pdf_page` 解析,不发送消息)、`info`(只返回本子信息,不下载)。 +- jmcpy 的 HTTP 请求是异步的,但图片解码与 PDF 合成是同步 CPU 工作;这两步在线程中执行,不阻塞事件循环。副作用是单次自动提取不可取消,由 `request_timeout` / `image_timeout` 与 jmcpy 的多端点重试兜底。 +- 默认不登录即可使用;需要在 jmcpy 中登录(例如访问账号可见内容)时,用 jmcpy 自己的方式保存会话,再把 `session_dir` 指向同一配置目录(或设置 `JMCPY_HOME`)。 +- 请遵守所在地法律法规与目标站点服务条款,只处理你有权获取的内容。 + +--- + 自动提取调度说明: - 斜杠命令优先级高于自动处理管线;命中命令后直接分发并结束本轮后续处理,不会触发自动提取或 AI 自动回复。命令输入和命令输出会写入历史,供后续 AI 轮次读取。 -- 同一条消息内,自动处理管线会并行检测 Bilibili、Douyin、arXiv、GitHub 等已注册管线。 +- 同一条消息内,自动处理管线会并行检测 Bilibili、Douyin、arXiv、GitHub、JM 等已注册管线。 - 检测到多个管线时会并行处理全部命中结果;通常单条消息只会命中一个管线,因此不手动维护优先级。 - 自动提取发送出的信息消息、图片卡片、文件或视频摘要会通过统一发送层写入消息历史,本地媒体和文件会自动登记为会话附件 UID,随后才进入 AI 自动回复,因此 AI 可以读取刚刚的自动提取结果。 - 管线实现位于 `src/Undefined/skills/pipelines/`,跟随 `[skills]` 热重载配置自动重新加载。开发新管线请参考 [自动处理管线开发指南](pipelines.md)。 @@ -1549,6 +1594,12 @@ api_key = "replace-with-your-key" | `image_gen.provider` | `IMAGE_GEN_PROVIDER` | | `image_gen.use_proxy` | `IMAGE_GEN_USE_PROXY` | +#### `jm` + +| TOML 路径 | 环境变量 | +|-----------|----------| +| `jm.use_proxy` | `JM_USE_PROXY` | + #### `logging` | TOML 路径 | 环境变量 | diff --git a/docs/development.md b/docs/development.md index 0e8a696e..7fda37ae 100644 --- a/docs/development.md +++ b/docs/development.md @@ -24,6 +24,7 @@ src/Undefined/ │ └── job_queue.py # 有序文件队列、阶段回队、检查点与遗留任务恢复(梗库共用) ├── config/ # 配置系统(parsers/ 域解析 + load_sections/ 分段加载 + loader shim) ├── handlers/ # OneBot 消息分流(message_flow / poke / repeat / auto_extract) +├── jm/ # 禁漫(JM / 18comic)车号解析、整本 PDF 下载合成与合并转发 ├── onebot/ # OneBot WebSocket 客户端 ├── skills/ # 技能插件核心目录 (存放所有的工具与智能体) │ ├── tools/ # 基础原子的工具 (独立的功能单元,如读写文件、网络请求等) @@ -31,7 +32,7 @@ src/Undefined/ │ │ └── cognitive/ # 认知记忆主动暴露工具 (search_events, get_profile 等) │ ├── agents/ # 智能体 (含 runner/ 通用循环子包) │ ├── commands/ # 中心化斜杠指令系统 (实现如 /help, /stats, /admin 等平台功能) -│ ├── pipelines/ # 自动提取管线 (bilibili / bilibili_opus / arxiv / github 等) +│ ├── pipelines/ # 自动提取管线 (bilibili / bilibili_opus / arxiv / github / jm 等) │ └── anthropic_skills/# Anthropic 协议集成的外部 Skills (兼容 SKILL.md 格式) ├── api/ # Management API + Runtime API │ ├── routes/ # 路由子模块 (chat, tools, naga/, system, memes, memory, cognitive, health) diff --git a/docs/pipelines.md b/docs/pipelines.md index 7bb088eb..e6d56f48 100644 --- a/docs/pipelines.md +++ b/docs/pipelines.md @@ -1,6 +1,6 @@ # 自动处理管线开发指南 -自动处理管线位于 `src/Undefined/skills/pipelines/`,用于在普通消息进入 AI 自动回复前执行自动提取,例如 Bilibili 视频、Bilibili 图文(opus)、抖音视频、arXiv 论文和 GitHub 仓库卡片。斜杠命令优先级高于自动处理管线,命中命令后不会继续触发自动提取或 AI 回复。 +自动处理管线位于 `src/Undefined/skills/pipelines/`,用于在普通消息进入 AI 自动回复前执行自动提取,例如 Bilibili 视频、Bilibili 图文(opus)、抖音视频、arXiv 论文、GitHub 仓库卡片和禁漫(JM)本子。斜杠命令优先级高于自动处理管线,命中命令后不会继续触发自动提取或 AI 回复。 `MessageHandler` 启动时会通过异步初始化在线程中加载管线配置和 handler 模块,避免目录扫描、`config.json` 读取和模块导入阻塞事件循环;注册 OneBot 消息回调前会等待首次加载完成,后续热重载也在线程中执行。 @@ -48,6 +48,29 @@ Bilibili 图文管线命中 `bilibili.com/opus/`、`t.bilibili.com/`、` 管线只在 `auto_extract_enabled` 与 `opus_enabled` 同时为真、且会话命中白名单时生效;与视频管线相互独立,同一条消息同时包含 BV 号与图文链接时两条管线各自发送。 +## 内置 JM(禁漫)管线 + +JM 管线命中 `JM` 前缀加 5–8 位车号(`JM1114751`、`jm 1114751`、`jm:1114751`)或主机名含 `18comic` / `jmcomic` 的链接(`/album/`、`/photo/`、`?id=`)后,发送一次外层合并转发,节点顺序固定: + +1. `本子信息`:车号与标题、作者、章节数、标签、观看/点赞、页数、PDF 大小、简介预览,末行是可直接复制的 `JM<车号>`(不带站点链接); +2. `解密密码`:每次随机生成的 8 位 PDF 打开密码; +3. `PDF`:本地合成好的加密 PDF(群聊下会作为真正的群文件上传)。 + +PDF 随转发一起上传,节点里的文件可以直接下载(已用 14MB 的 PDF 实测);群聊下 NapCat 会把它作为群文件上传(`isGroupFile`、`busid=102`,元素里带 `fileId` / `fileMd5` / `fileSha1`),所以同一个 PDF 也会出现在群文件列表里。文件只发一次,不额外发独立文件消息;只有转发本身发送失败时才退化为「信息 + 密码 + 独立文件消息」。 + +多章节本子按章节顺序全部下载(`[jm].max_chapters` 可限制),再合成为一个 AES-256 加密 PDF。裸数字不触发,避免群号、时间戳等误报;`xxjm1234567` 这类前缀也不触发。 + +失败语义: + +- 转发被拒时退化为两条普通消息(信息、密码)+ 独立文件消息(这种兜底情况下文件没有别的入口)。密码始终不会写进历史摘要。 +- 下载量或 PDF 体积超过 `[jm].max_file_size`(原始图片累计预判 + 组装时按编码后字节复核)、以及没有下到任何页面时,只发信息与状态两个节点,不发密码与文件。 + +PDF 合成没有走 `jmcpy.imaging.write_pdf`,而是自己用 PyMuPDF 逐页写入:jmcpy 的下载 API 一章只能产出一个 PDF,而这里要把多个章节合进同一个文件;同时 PATH 输出会先把解扰后的图重新编码一次、合成时再编码第二次,这里改为 `decode=False` 取服务端原始字节(无损落盘)、自己解扰、每页只编码一次 JPEG(`[jm].image_quality`,色度 4:4:4)后直接作为 PDF 图像数据,并按 `[jm].pdf_dpi` 统一页尺寸、做 AES-256 加密。页像素始终不做缩放,画质上限由站点源图自身分辨率决定。(jmcpy ≤0.1.1 的 `write_pdf` 另有「追加页退回 72 DPI、同文档页尺寸不一致」的问题,已在 0.1.2 修复;页内色度采样从 0.1.3 起默认也是 4:4:4。本管线依赖 `jmcpy>=0.1.3`。) + +AI 侧另有 `jm_book` 工具(`src/Undefined/skills/tools/jm_book/`),与 `arxiv_paper` 同构:`send` 等价自动提取,`uid` 只注册未加密 PDF 附件 UID(共享给 `file_analysis_agent`),`info` 只取详情。 + +实现说明:图片解码与 PDF 合成是同步 CPU 工作,下载整体在 `asyncio.to_thread` 中通过 jmcpy 的同步客户端执行,不阻塞事件循环;单次任务不可取消,由 `[jm].request_timeout` / `image_timeout` 与 jmcpy 的多端点重试兜底。PDF 在内存中合成后写盘,因此 `[jm].max_file_size` 只是近似内存上限(预判按源图字节、复核按编码后字节,JPEG 重编码可能更大)。下载跑在模块专用的线程池(上限 2 个并发本子)里,不占用事件循环默认执行器;单页解码失败只跳过该页并计入「下载失败 N 页」,不会让整本失败;任务目录在成功、超限、空结果与异常四条路径上都会清理;协程被取消时线程仍在写盘,改为等它跑完再删,避免残留原图或未加密 PDF。 + ## 目录结构 ```text @@ -68,7 +91,10 @@ src/Undefined/skills/pipelines/ ├── arxiv/ │ ├── config.json │ └── handler.py -└── github/ +├── github/ +│ ├── config.json +│ └── handler.py +└── jm/ ├── config.json └── handler.py ``` diff --git a/docs/usage.md b/docs/usage.md index 7ee819b1..8daf3364 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -76,7 +76,7 @@ Undefined 搭载了基于 ChromaDB 向量数据库的后台认知系统,无需 ### `file_analysis_agent` — 文件分析助手 -支持对代码、PDF、Word、Excel 等多种格式文件进行解析与分析。用户可以直接发送文件,也可以提供附件 UID、URL、arXiv ID/URL、Bilibili BV/AV/URL 或抖音链接/aweme_id。 +支持对代码、PDF、Word、Excel 等多种格式文件进行解析与分析。用户可以直接发送文件,也可以提供附件 UID、URL、arXiv ID/URL、Bilibili BV/AV/URL、抖音链接/aweme_id 或禁漫车号/链接。 **子工具**:`download_file`、`extract_pdf`、`describe_pdf_page`、`extract_docx`、`extract_xlsx`、`analyze_code`、`analyze_multimodal` @@ -372,6 +372,7 @@ QQ/NapCat 在 `sendMsg` 阶段返回超时并不等于消息未送达:服务 | `bilibili_opus` | 发送哔哩哔哩图文(opus)合并转发、返回正文图片附件 UID、获取图文信息,或按字数区间 / 关键词读取正文文字(支持动态 ID、图文链接、b23.tv 短链) | | `douyin_video` | 发送抖音视频、返回视频附件 UID,或只获取视频信息(支持短链、长链、aweme_id) | | `arxiv_paper` | 发送 arXiv 论文 PDF、返回 PDF 附件 UID,或只获取论文信息(支持 arXiv ID、链接) | +| `jm_book` | 发送禁漫(JM)本子合并转发、返回未加密 PDF 附件 UID,或只获取本子信息(支持 `JM350234`、裸车号、禁漫链接) | | `fetch_image_uid` | 将指定 URL 的图片下载并转换为系统内部 uid | | `task_progress` | 向用户发送长任务的阶段性进度通知 | | `changelog_query` | 查询系统内置版本更新日志 | diff --git a/pyproject.toml b/pyproject.toml index 76123935..3b1c5064 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "Undefined-bot" -version = "3.17.2" +version = "3.18.0" description = "QQ bot platform with cognitive memory architecture and multi-agent Skills, via OneBot V11." readme = "README.md" authors = [ @@ -48,6 +48,7 @@ dependencies = [ "silk-python>=0.2.8,<0.3.0", "qrcode>=8.2,<9.0", "regex>=2026.2.28", + "jmcpy>=0.1.3", ] [project.urls] diff --git a/src/Undefined/__init__.py b/src/Undefined/__init__.py index 3f1bde21..9a7b1e24 100644 --- a/src/Undefined/__init__.py +++ b/src/Undefined/__init__.py @@ -24,7 +24,7 @@ from .skills.registry import BaseRegistry as BaseRegistry from .skills.tools import ToolRegistry as ToolRegistry -__version__: str = "3.17.2" +__version__: str = "3.18.0" # symbol -> (module_path, attribute_name);首次访问时才 importlib 加载 _LAZY_IMPORTS: dict[str, tuple[str, str]] = { diff --git a/src/Undefined/config/config_class.py b/src/Undefined/config/config_class.py index 10be7021..db7f710e 100644 --- a/src/Undefined/config/config_class.py +++ b/src/Undefined/config/config_class.py @@ -241,6 +241,20 @@ class Config: github_auto_extract_group_ids: list[int] github_auto_extract_private_ids: list[int] github_auto_extract_max_items: int + # JM(禁漫)自动提取 + jm_use_proxy: bool + jm_auto_extract_enabled: bool + jm_auto_extract_group_ids: list[int] + jm_auto_extract_private_ids: list[int] + jm_auto_extract_max_items: int + jm_max_file_size: int + jm_max_chapters: int + jm_pdf_dpi: float + jm_image_quality: int + jm_download_concurrency: int + jm_request_timeout: float + jm_image_timeout: float + jm_session_dir: str # 认知记忆 cognitive: CognitiveConfig # 表情包库 @@ -321,6 +335,16 @@ class Config: init=False, repr=False, ) + _jm_group_ids_set: set[int] = dataclass_field( + default_factory=set, + init=False, + repr=False, + ) + _jm_private_ids_set: set[int] = dataclass_field( + default_factory=set, + init=False, + repr=False, + ) def __post_init__(self) -> None: self._refresh_runtime_sets() @@ -359,6 +383,10 @@ def _refresh_runtime_sets(self) -> None: self._github_private_ids_set = { int(item) for item in self.github_auto_extract_private_ids } + self._jm_group_ids_set = {int(item) for item in self.jm_auto_extract_group_ids} + self._jm_private_ids_set = { + int(item) for item in self.jm_auto_extract_private_ids + } @classmethod def load(cls, config_path: Optional[Path] = None, strict: bool = True) -> "Config": @@ -553,6 +581,18 @@ def is_github_auto_extract_allowed_private(self, user_id: int) -> bool: return int(user_id) in self._github_private_ids_set return self.is_private_allowed(user_id) + def is_jm_auto_extract_allowed_group(self, group_id: int) -> bool: + """群聊是否允许 JM 自动提取。""" + if self._jm_group_ids_set: + return int(group_id) in self._jm_group_ids_set + return self.is_group_allowed(group_id) + + def is_jm_auto_extract_allowed_private(self, user_id: int) -> bool: + """私聊是否允许 JM 自动提取。""" + if self._jm_private_ids_set: + return int(user_id) in self._jm_private_ids_set + return self.is_private_allowed(user_id) + def should_process_group_message(self, is_at_bot: bool) -> bool: """是否处理该条群消息。""" diff --git a/src/Undefined/config/env_registry.py b/src/Undefined/config/env_registry.py index 5029cfc7..d017a969 100644 --- a/src/Undefined/config/env_registry.py +++ b/src/Undefined/config/env_registry.py @@ -72,6 +72,7 @@ ("history", "max_records"): "HISTORY_MAX_RECORDS", ("image_gen", "use_proxy"): "IMAGE_GEN_USE_PROXY", ("image_gen", "provider"): "IMAGE_GEN_PROVIDER", + ("jm", "use_proxy"): "JM_USE_PROXY", ("logging", "backup_count"): "LOG_BACKUP_COUNT", ("logging", "file_path"): "LOG_FILE_PATH", ("logging", "level"): "LOG_LEVEL", diff --git a/src/Undefined/config/load_sections/integrations.py b/src/Undefined/config/load_sections/integrations.py index f275d36f..804fa346 100644 --- a/src/Undefined/config/load_sections/integrations.py +++ b/src/Undefined/config/load_sections/integrations.py @@ -27,6 +27,17 @@ _MAX_GITHUB_REQUEST_RETRIES: int = 5 _DEFAULT_GITHUB_AUTO_EXTRACT_MAX_ITEMS: int = 3 _MAX_GITHUB_AUTO_EXTRACT_MAX_ITEMS: int = 10 +_DEFAULT_JM_REQUEST_TIMEOUT_SECONDS: float = 20.0 +_MAX_JM_REQUEST_TIMEOUT_SECONDS: float = 120.0 +_DEFAULT_JM_IMAGE_TIMEOUT_SECONDS: float = 60.0 +_MAX_JM_IMAGE_TIMEOUT_SECONDS: float = 600.0 +_DEFAULT_JM_AUTO_EXTRACT_MAX_ITEMS: int = 1 +_MAX_JM_AUTO_EXTRACT_MAX_ITEMS: int = 5 +_DEFAULT_JM_MAX_FILE_SIZE_MB: int = 100 +_DEFAULT_JM_PDF_DPI: float = 150.0 +_DEFAULT_JM_IMAGE_QUALITY: int = 95 +_DEFAULT_JM_DOWNLOAD_CONCURRENCY: int = 8 +_MAX_JM_DOWNLOAD_CONCURRENCY: int = 32 _DEFAULT_LXMUSIC2API_BASE_URL: str = "http://127.0.0.1:3000" @@ -221,6 +232,73 @@ def load_integrations( if github_auto_extract_max_items > _MAX_GITHUB_AUTO_EXTRACT_MAX_ITEMS: github_auto_extract_max_items = _MAX_GITHUB_AUTO_EXTRACT_MAX_ITEMS + # JM(禁漫)配置 + jm_use_proxy = _coerce_bool( + _get_value(data, ("jm", "use_proxy"), "JM_USE_PROXY"), False + ) + jm_auto_extract_enabled = _coerce_bool( + _get_value(data, ("jm", "auto_extract_enabled"), None), False + ) + jm_auto_extract_group_ids = _coerce_int_list( + _get_value(data, ("jm", "auto_extract_group_ids"), None) + ) + jm_auto_extract_private_ids = _coerce_int_list( + _get_value(data, ("jm", "auto_extract_private_ids"), None) + ) + jm_auto_extract_max_items = _coerce_int( + _get_value(data, ("jm", "auto_extract_max_items"), None), + _DEFAULT_JM_AUTO_EXTRACT_MAX_ITEMS, + ) + if jm_auto_extract_max_items <= 0: + jm_auto_extract_max_items = _DEFAULT_JM_AUTO_EXTRACT_MAX_ITEMS + if jm_auto_extract_max_items > _MAX_JM_AUTO_EXTRACT_MAX_ITEMS: + jm_auto_extract_max_items = _MAX_JM_AUTO_EXTRACT_MAX_ITEMS + jm_max_file_size = _coerce_int( + _get_value(data, ("jm", "max_file_size"), None), _DEFAULT_JM_MAX_FILE_SIZE_MB + ) + if jm_max_file_size < 0: + jm_max_file_size = _DEFAULT_JM_MAX_FILE_SIZE_MB + jm_max_chapters = _coerce_int(_get_value(data, ("jm", "max_chapters"), None), 0) + if jm_max_chapters < 0: + jm_max_chapters = 0 + jm_pdf_dpi = _coerce_float( + _get_value(data, ("jm", "pdf_dpi"), None), _DEFAULT_JM_PDF_DPI + ) + if jm_pdf_dpi <= 0: + jm_pdf_dpi = _DEFAULT_JM_PDF_DPI + jm_image_quality = _coerce_int( + _get_value(data, ("jm", "image_quality"), None), _DEFAULT_JM_IMAGE_QUALITY + ) + if jm_image_quality < 1: + jm_image_quality = _DEFAULT_JM_IMAGE_QUALITY + if jm_image_quality > 100: + jm_image_quality = 100 + jm_download_concurrency = _coerce_int( + _get_value(data, ("jm", "download_concurrency"), None), + _DEFAULT_JM_DOWNLOAD_CONCURRENCY, + ) + if jm_download_concurrency < 1: + jm_download_concurrency = _DEFAULT_JM_DOWNLOAD_CONCURRENCY + if jm_download_concurrency > _MAX_JM_DOWNLOAD_CONCURRENCY: + jm_download_concurrency = _MAX_JM_DOWNLOAD_CONCURRENCY + jm_request_timeout = _coerce_float( + _get_value(data, ("jm", "request_timeout"), None), + _DEFAULT_JM_REQUEST_TIMEOUT_SECONDS, + ) + if jm_request_timeout <= 0: + jm_request_timeout = _DEFAULT_JM_REQUEST_TIMEOUT_SECONDS + if jm_request_timeout > _MAX_JM_REQUEST_TIMEOUT_SECONDS: + jm_request_timeout = _MAX_JM_REQUEST_TIMEOUT_SECONDS + jm_image_timeout = _coerce_float( + _get_value(data, ("jm", "image_timeout"), None), + _DEFAULT_JM_IMAGE_TIMEOUT_SECONDS, + ) + if jm_image_timeout <= 0: + jm_image_timeout = _DEFAULT_JM_IMAGE_TIMEOUT_SECONDS + if jm_image_timeout > _MAX_JM_IMAGE_TIMEOUT_SECONDS: + jm_image_timeout = _MAX_JM_IMAGE_TIMEOUT_SECONDS + jm_session_dir = _coerce_str(_get_value(data, ("jm", "session_dir"), None), "") + # Code Delivery Agent 配置 code_delivery_enabled = _coerce_bool( _get_value(data, ("code_delivery", "enabled"), None), True @@ -386,6 +464,19 @@ def load_integrations( "github_auto_extract_group_ids": github_auto_extract_group_ids, "github_auto_extract_private_ids": github_auto_extract_private_ids, "github_auto_extract_max_items": github_auto_extract_max_items, + "jm_use_proxy": jm_use_proxy, + "jm_auto_extract_enabled": jm_auto_extract_enabled, + "jm_auto_extract_group_ids": jm_auto_extract_group_ids, + "jm_auto_extract_private_ids": jm_auto_extract_private_ids, + "jm_auto_extract_max_items": jm_auto_extract_max_items, + "jm_max_file_size": jm_max_file_size, + "jm_max_chapters": jm_max_chapters, + "jm_pdf_dpi": jm_pdf_dpi, + "jm_image_quality": jm_image_quality, + "jm_download_concurrency": jm_download_concurrency, + "jm_request_timeout": jm_request_timeout, + "jm_image_timeout": jm_image_timeout, + "jm_session_dir": jm_session_dir, "code_delivery_enabled": code_delivery_enabled, "code_delivery_task_root": code_delivery_task_root, "code_delivery_docker_image": code_delivery_docker_image, diff --git a/src/Undefined/handlers/auto_extract.py b/src/Undefined/handlers/auto_extract.py index a5ad0dd4..64beb4bb 100644 --- a/src/Undefined/handlers/auto_extract.py +++ b/src/Undefined/handlers/auto_extract.py @@ -148,6 +148,29 @@ def _extract_github_repo_ids( return repo_ids + def _extract_jm_ids( + self, text: str, message_content: list[dict[str, Any]] + ) -> list[str]: + """从文本和消息段中提取禁漫(JM)车号。""" + from Undefined.jm.parser import extract_from_json_message, extract_jm_ids + + book_ids: list[str] = [] + seen: set[str] = set() + + for book_id in extract_jm_ids(text): + if book_id in seen: + continue + seen.add(book_id) + book_ids.append(book_id) + + for book_id in extract_from_json_message(message_content): + if book_id in seen: + continue + seen.add(book_id) + book_ids.append(book_id) + + return book_ids + async def _handle_bilibili_extract( self, target_id: int, @@ -399,3 +422,49 @@ async def _handle_github_extract( type(exc).__name__, exc, ) + + async def _handle_jm_extract( + self, + target_id: int, + book_ids: list[str], + target_type: str, + sender: Any | None = None, + ) -> None: + """处理禁漫(JM)自动提取和发送。""" + from Undefined.jm.sender import send_jm_book + + max_items = max(1, int(getattr(self.config, "jm_auto_extract_max_items", 1))) + resolved_sender = sender or self.sender + for book_id in book_ids[:max_items]: + try: + result = await send_jm_book( + book_id, + sender=resolved_sender, + target_type=target_type, # type: ignore[arg-type] + target_id=target_id, + config=self.config, + ) + logger.info( + "[JM] 自动提取完成 %s → %s:%s: %s", + book_id, + target_type, + target_id, + result, + ) + except Exception as exc: + logger.exception( + "[JM] 自动提取失败 %s → %s:%s: exc_type=%s exc=%r", + book_id, + target_type, + target_id, + type(exc).__name__, + exc, + ) + try: + error_msg = f"JM 提取失败:{exc}" + if target_type == "group": + await resolved_sender.send_group_message(target_id, error_msg) + else: + await resolved_sender.send_private_message(target_id, error_msg) + except Exception: + pass diff --git a/src/Undefined/jm/__init__.py b/src/Undefined/jm/__init__.py new file mode 100644 index 00000000..52fa8110 --- /dev/null +++ b/src/Undefined/jm/__init__.py @@ -0,0 +1 @@ +"""禁漫(JM / 18comic)自动提取:标识解析、PDF 下载合成与合并转发。""" diff --git a/src/Undefined/jm/client.py b/src/Undefined/jm/client.py new file mode 100644 index 00000000..4d73d5e9 --- /dev/null +++ b/src/Undefined/jm/client.py @@ -0,0 +1,29 @@ +"""jmcpy 客户端设置。 + +jmcpy 的接口与图片 CDN 都走 HTTPS,因此这里只按 scheme 取 ``[proxy]`` 里的代理, +再叠加 ``[jm]`` 的超时、并发与会话目录。 +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +from jmcpy import Settings + +from Undefined.skills.http_config import get_request_proxy + +#: 仅用于按 scheme 选择代理地址(jmcpy 的请求全部是 HTTPS) +_PROXY_PROBE_URL = "https://18comic.vip" + + +def build_settings(config: Any) -> Settings: + """把 ``[jm]`` 配置映射成 jmcpy 的 :class:`~jmcpy.Settings`。""" + home = str(getattr(config, "jm_session_dir", "") or "").strip() + return Settings( + proxy=get_request_proxy(_PROXY_PROBE_URL, proxy_scope="jm", config=config), + timeout=float(getattr(config, "jm_request_timeout", 20.0)), + image_timeout=float(getattr(config, "jm_image_timeout", 60.0)), + concurrency=int(getattr(config, "jm_download_concurrency", 8)), + home=Path(home) if home else None, + ) diff --git a/src/Undefined/jm/downloader.py b/src/Undefined/jm/downloader.py new file mode 100644 index 00000000..a15782ee --- /dev/null +++ b/src/Undefined/jm/downloader.py @@ -0,0 +1,395 @@ +"""禁漫本子下载:逐章取图,再合成一个加密 PDF。 + +jmcpy 的异步实现内部仍会同步执行图片解码与 PDF 合成(``_to_artifact`` / +``_finalize`` 直接在协程里跑),放在事件循环里会阻塞其它消息处理,因此这里用 +同步 ``Client`` 配合 :func:`asyncio.to_thread`,线程里可以放心做 CPU 与磁盘工作。 +代价是单次任务不可取消,由 jmcpy 自己的请求超时与多端点重试兜底。 + +合成不走 ``jmcpy.imaging.write_pdf``,而是自己用 PyMuPDF 逐页写入,原因是: + +* 要把**多个章节**合进同一个 PDF——jmcpy 的下载 API 只能一章一个 PDF; +* 每页只编码一次:``download(output=PATH)`` 会先把解扰后的图重新编码一次 + (WebP/JPEG),合成时再编码第二次;这里用 ``decode=False`` 取服务端原始字节 + (无损落盘),自己解扰后只编码一次 JPEG(4:4:4,漫画的彩色描边在 4:2:0 下 + 会发虚); +* 页尺寸由我们统一控制(页宽 = 像素宽 ÷ dpi × 72)。 + +jmcpy ≤0.1.1 的 ``write_pdf`` 还会让追加页退回默认 72 DPI(同一份 PDF 里第一页 +5.6in 宽、其余页 11.7in 宽),该问题已在 0.1.2 修复;这里保留自建组装是为了上面 +三点,与那个 bug 无关。 +""" + +from __future__ import annotations + +import asyncio +from concurrent.futures import ThreadPoolExecutor +from dataclasses import dataclass +import functools +import io +import logging +import shutil +from pathlib import Path +from typing import Any, Literal +import uuid + +import fitz +from jmcpy import Book, Client, ExportFormat, Settings +from jmcpy.imaging import block_count, descramble, load_image +from jmcpy.models import Picture +from jmcpy.texts import sanitize_filename +from PIL import Image + +from Undefined.jm.client import build_settings +from Undefined.utils.http_download import cleanup_download_dir +from Undefined.utils.paths import DOWNLOAD_CACHE_DIR, ensure_dir + +logger = logging.getLogger(__name__) + +_JM_DOWNLOAD_DIR = DOWNLOAD_CACHE_DIR / "jm" +_BOOK_ID_PREFIX = "JM" +#: 整本下载是分钟级、不可取消的任务:用专用线程池,避免占满事件循环默认执行器 +#: (默认池同时承担 utils/io 的磁盘写入),同时把并发本子数限制在这个数量。 +_MAX_CONCURRENT_DOWNLOADS = 2 +_DOWNLOAD_POOL = ThreadPoolExecutor( + max_workers=_MAX_CONCURRENT_DOWNLOADS, thread_name_prefix="jm-download" +) +#: JPEG 色度不做下采样:漫画的彩色描边与文字在 4:2:0 下会发虚 +_JPEG_SUBSAMPLING = 0 +#: PDF 页物理尺寸 = 像素 ÷ DPI,所有页统一用这个 DPI +_POINTS_PER_INCH = 72.0 + + +@dataclass(frozen=True, slots=True) +class JmDownload: + """一次本子下载的结果。""" + + #: ok=PDF 可用;oversize=超过体积上限;empty=没有下到任何页面 + status: Literal["ok", "oversize", "empty"] + book: Book + pdf_path: Path | None + page_count: int + size_bytes: int | None + #: 本子声明的章节总数(单章本子为 1) + chapter_count: int + #: 本次实际下载并合并的章节数 + downloaded_chapters: int + #: 下载失败的页面数(jmcpy 逐页收集,不打断整章) + failed_pages: int + + @property + def ok(self) -> bool: + return self.status == "ok" and self.pdf_path is not None + + +def _chapter_ids(book: Book, *, max_chapters: int) -> list[int]: + # 按本子内的章节序号排序,合出来的 PDF 与阅读顺序一致 + ordered = sorted(book.chapters, key=lambda chapter: int(chapter.order)) + chapter_ids = [int(chapter.chapter_id) for chapter in ordered] + if not chapter_ids: + # 单章本子的章节号就是车号 + chapter_ids = [int(book.book_id)] + if max_chapters > 0: + chapter_ids = chapter_ids[:max_chapters] + return chapter_ids + + +def _pdf_file_name(book: Book) -> str: + title = sanitize_filename(book.title) or "漫画" + return f"{_BOOK_ID_PREFIX}{book.book_id} {title}.pdf" + + +def _limit_bytes(max_file_size_mb: int) -> int | None: + return max_file_size_mb * 1024 * 1024 if max_file_size_mb > 0 else None + + +def _load_page(page_path: Path, picture: Picture) -> Image.Image: + """读取一页原图并按需解扰(服务端把竖直分块打乱,块数由章节号与文件名决定)。""" + image = load_image(page_path.read_bytes()) + # 动图不做分块还原(与 jmcpy 的 UNDECODED_SUFFIXES 语义一致) + blocks = block_count(picture.scramble_id, picture.chapter_id, picture.filename) + if blocks > 1 and not picture.is_animated: + image = descramble(image, blocks) + # PDF 只稳定支持这几种颜色模式,其余统一转 RGB + return image if image.mode == "RGB" else image.convert("RGB") + + +@dataclass(frozen=True, slots=True) +class _PdfBuild: + """PDF 组装结果。""" + + status: Literal["ok", "oversize", "empty"] + #: ``oversize`` / ``empty`` 时为 ``None`` + path: Path | None + page_count: int + #: 成功写入的页在 PDF 里的编码字节数(``max_file_size`` 按它判定) + encoded_bytes: int + #: 组装阶段因解码失败被跳过的页数 + failed_pages: int + + +def _write_pdf( + pages: list[tuple[Path, Picture]], + output: Path, + *, + dpi: float, + quality: int, + password: str | None, + limit_bytes: int | None, +) -> _PdfBuild: + """把页面按顺序合成一个 PDF(页尺寸统一、每页只编码一次)。 + + ``password`` 为 ``None`` 时不加密——附件 UID 模式要交给 PDF 解析,加密会让 + ``extract_pdf`` / ``describe_pdf_page`` 打不开。单页解码失败只跳过该页并计数, + 不整本放弃;累计编码字节超过 ``limit_bytes`` 时提前中止(此时 PDF 未落盘)。 + """ + document = fitz.open() + page_count = 0 + encoded_bytes = 0 + failed_pages = 0 + try: + for page_path, picture in pages: + try: + image = _load_page(page_path, picture) + except Exception: + failed_pages += 1 + logger.warning( + "[JM] 跳过无法解码的页面: %s", page_path.name, exc_info=True + ) + continue + width, height = image.size + page = document.new_page( + width=width * _POINTS_PER_INCH / dpi, + height=height * _POINTS_PER_INCH / dpi, + ) + buffer = io.BytesIO() + image.save( + buffer, + format="JPEG", + quality=quality, + subsampling=_JPEG_SUBSAMPLING, + ) + data = buffer.getvalue() + encoded_bytes += len(data) + if limit_bytes is not None and encoded_bytes > limit_bytes: + logger.info( + "[JM] PDF 编码后体积超过上限,提前中止: bytes=%s limit=%s", + encoded_bytes, + limit_bytes, + ) + return _PdfBuild( + "oversize", None, page_count, encoded_bytes, failed_pages + ) + # MuPDF 对 JPEG 流做 DCT 直通:上面这一次编码就是 PDF 里的最终数据 + page.insert_image(page.rect, stream=data) + page_count += 1 + + if page_count == 0: + return _PdfBuild("empty", None, 0, encoded_bytes, failed_pages) + + if password is None: + document.save(str(output), garbage=3, deflate=True) + else: + document.save( + str(output), + encryption=fitz.PDF_ENCRYPT_AES_256, + user_pw=password, + owner_pw=password, + garbage=3, + deflate=True, + ) + finally: + document.close() + return _PdfBuild("ok", output, page_count, encoded_bytes, failed_pages) + + +def _download_sync( + book_id: str, + task_dir: Path, + *, + password: str | None, + settings: Settings, + max_chapters: int, + max_file_size_mb: int, + pdf_dpi: float, + image_quality: int, +) -> JmDownload: + limit_bytes = _limit_bytes(max_file_size_mb) + with Client(settings) as client: + book = client.get_book(book_id) + # 声明总数取本子自身的章节列表,不受 max_chapters 截断影响 + declared_chapters = len(book.chapters) or 1 + chapter_ids = _chapter_ids(book, max_chapters=max_chapters) + pages: list[tuple[Path, Picture]] = [] + failed_pages = 0 + downloaded_chapters = 0 + total_bytes = 0 + + for index, chapter_id in enumerate(chapter_ids, start=1): + chapter = client.get_chapter(chapter_id) + if not len(chapter): + logger.info( + "[JM] 章节没有图片,跳过: book=%s chapter=%s", book_id, chapter_id + ) + continue + # 每章独立的目录:同名章节复用同名目录时,overwrite=False 会拿旧图。 + # decode=False 取服务端原始字节(无损),解扰与编码在写 PDF 时做一次。 + result = client.download( + chapter, + output=ExportFormat.PATH, + dest=task_dir / f"c{index:03d}", + decode=False, + concurrency=settings.concurrency, + ) + for item in result: + if item.path is not None: + pages.append((item.path, item.picture)) + failed_pages += len(result.failures) + downloaded_chapters += 1 + total_bytes += sum(item.size or 0 for item in result) + if limit_bytes is not None and total_bytes > limit_bytes: + logger.info( + "[JM] 图片总量超过体积上限,提前结束: book=%s bytes=%s limit=%sMB", + book_id, + total_bytes, + max_file_size_mb, + ) + return JmDownload( + status="oversize", + book=book, + pdf_path=None, + page_count=len(pages), + size_bytes=total_bytes, + chapter_count=declared_chapters, + downloaded_chapters=downloaded_chapters, + failed_pages=failed_pages, + ) + + if not pages: + return JmDownload( + status="empty", + book=book, + pdf_path=None, + page_count=0, + size_bytes=None, + chapter_count=declared_chapters, + downloaded_chapters=downloaded_chapters, + failed_pages=failed_pages, + ) + + build = _write_pdf( + pages, + task_dir / _pdf_file_name(book), + dpi=pdf_dpi, + quality=image_quality, + password=password, + limit_bytes=limit_bytes, + ) + failed_pages += build.failed_pages + if build.status != "ok" or build.path is None: + logger.info( + "[JM] PDF 未生成: book=%s status=%s pages=%s bytes=%s", + book_id, + build.status, + build.page_count, + build.encoded_bytes, + ) + return JmDownload( + status=build.status, + book=book, + pdf_path=None, + page_count=build.page_count, + size_bytes=build.encoded_bytes or None, + chapter_count=declared_chapters, + downloaded_chapters=downloaded_chapters, + failed_pages=failed_pages, + ) + + return JmDownload( + status="ok", + book=book, + pdf_path=build.path, + page_count=build.page_count, + size_bytes=build.path.stat().st_size, + chapter_count=declared_chapters, + downloaded_chapters=downloaded_chapters, + failed_pages=failed_pages, + ) + + +def _fetch_book_sync(book_id: str, settings: Settings) -> Book: + with Client(settings) as client: + return client.get_book(book_id) + + +async def fetch_book(book_id: str, *, config: Any) -> Book: + """只取本子详情,不下载任何图片(线程里跑,不阻塞事件循环)。""" + return await asyncio.to_thread(_fetch_book_sync, book_id, build_settings(config)) + + +async def download_book_pdf( + book_id: str, + *, + config: Any, + password: str | None = None, +) -> tuple[JmDownload, Path]: + """下载整本并合成 PDF,返回 ``(结果, 任务目录)``。 + + ``password`` 给出时用 AES-256 加密(自动提取用),``None`` 时输出未加密 PDF + (附件 UID 模式用,便于 PDF 解析)。调用方负责在发送或登记完成后清理任务目录 + (``cleanup_download_path``);本函数抛异常或被取消时自己清理,不留残留目录。 + """ + task_dir = ensure_dir(_JM_DOWNLOAD_DIR / uuid.uuid4().hex) + loop = asyncio.get_running_loop() + future = loop.run_in_executor( + _DOWNLOAD_POOL, + functools.partial( + _download_sync, + book_id, + task_dir, + password=password, + settings=build_settings(config), + max_chapters=int(getattr(config, "jm_max_chapters", 0)), + max_file_size_mb=int(getattr(config, "jm_max_file_size", 100)), + pdf_dpi=float(getattr(config, "jm_pdf_dpi", 150.0)), + image_quality=int(getattr(config, "jm_image_quality", 95)), + ), + ) + try: + # shield:协程被取消时不要把线程一起取消——同步下载停不下来,强行删目录 + # 只会被后续写入重新创建,所以等它跑完再清理 + result = await asyncio.shield(future) + except BaseException: + if future.done(): + # 异常路径下调用方拿不到 task_dir,必须在这里清掉已下载的原图 + await cleanup_download_dir(task_dir) + else: + _cleanup_when_finished(future, task_dir, loop=loop) + raise + return result, task_dir + + +def _remove_dir_now(path: Path) -> None: + shutil.rmtree(path, ignore_errors=True) + + +def _cleanup_when_finished( + future: "asyncio.Future[JmDownload]", + task_dir: Path, + *, + loop: asyncio.AbstractEventLoop, +) -> None: + """线程仍在写盘时,等它结束后再删任务目录(取消/超时路径)。""" + + def _on_done(_future: "asyncio.Future[JmDownload]") -> None: + try: + loop.run_in_executor(None, _remove_dir_now, task_dir) + except RuntimeError: + # 事件循环已关闭:退化为同步删除,避免残留原图或未加密 PDF + _remove_dir_now(task_dir) + + future.add_done_callback(_on_done) + + +async def cleanup_download_path(task_dir: Path) -> None: + """清理本次任务的临时目录。""" + await cleanup_download_dir(task_dir) diff --git a/src/Undefined/jm/parser.py b/src/Undefined/jm/parser.py new file mode 100644 index 00000000..8ad2334b --- /dev/null +++ b/src/Undefined/jm/parser.py @@ -0,0 +1,155 @@ +"""禁漫(JM / 18comic)车号解析。 + +触发形态有两种:``JM`` 前缀加数字(``JM1114751`` / ``jm 1114751``), +以及禁漫链接(``https://18comic.vip/album/1114751``、``?id=1114751``)。 +两种形态最终都归一化成纯数字车号,交给 :func:`jmcpy.texts.normalize_book_id` +解析,避免自己维护一份路径规则。 +""" + +from __future__ import annotations + +import html +import json +import logging +import re +from collections.abc import Iterator +from typing import Any +from urllib.parse import urlsplit + +from jmcpy.errors import JmcpyError +from jmcpy.texts import normalize_book_id + +logger = logging.getLogger(__name__) + +#: 车号位数范围:5 位(早期短号)到 8 位(当前号段) +_MIN_BOOK_ID = 10_000 +_MAX_BOOK_ID = 99_999_999 + +#: ``jm`` + 数字;前缀前不能是字母数字,也不能是 URL/路径/文件名的分隔符 +#: (``xxjm1234567``、``t.me/jm1234567``、``video_jm1234567.mp4`` 都不算车号), +#: 数字后不能跟数字(避免截断长数字) +_JM_TOKEN_REGEX = re.compile( + r"(?()\"']+", re.I) +#: 链接归属:主机名包含这些关键词时按禁漫链接处理 +_JM_HOST_KEYWORDS = ("18comic", "jmcomic") +#: 链接末尾常被中文标点或引号包裹 +_URL_WRAPPERS = ".,;:!?)]}>\"'”’" + + +def _strip_wrapper_chars(value: str) -> str: + return html.unescape(value).strip().rstrip(_URL_WRAPPERS) + + +def _is_jm_host(host: str) -> bool: + return any(keyword in host for keyword in _JM_HOST_KEYWORDS) + + +def normalize_jm_id(value: int | str) -> str | None: + """把车号、``JM<数字>`` 或禁漫链接归一化成纯数字字符串。 + + 位数不在 5–8 位范围内、或链接不在禁漫域名下时返回 ``None`` + (普通数字串与非禁漫链接都不会被误判成车号)。 + """ + if isinstance(value, str) and "://" in value: + if not _is_jm_host(urlsplit(value).hostname or ""): + return None + try: + digits = normalize_book_id(value) + except (JmcpyError, ValueError): + return None + if not digits.isdigit(): + return None + if not _MIN_BOOK_ID <= int(digits) <= _MAX_BOOK_ID: + return None + return str(int(digits)) + + +def _append_candidate( + candidate: str, + *, + results: list[str], + seen: set[str], +) -> None: + normalized = normalize_jm_id(candidate) + if normalized is None or normalized in seen: + return + seen.add(normalized) + results.append(normalized) + + +def _iter_candidates(text: str) -> Iterator[tuple[int, str]]: + """产出 ``(位置, 候选串)``:链接与 ``JM`` 车号混排时也按原文顺序排列。""" + for match in _URL_REGEX.finditer(text): + candidate = _strip_wrapper_chars(match.group(0)) + if _is_jm_host(urlsplit(candidate).hostname or ""): + yield match.start(), candidate + for match in _JM_TOKEN_REGEX.finditer(text): + yield match.start(), match.group(1) + + +def extract_jm_ids(text: str) -> list[str]: + """从纯文本中提取禁漫车号(按出现顺序去重)。""" + results: list[str] = [] + seen: set[str] = set() + if not text: + return results + + # 链接与车号统一按原文位置排序:混排时取到的第一个必须是用户先写的那个 + for _position, candidate in sorted( + _iter_candidates(text), key=lambda item: item[0] + ): + _append_candidate(candidate, results=results, seen=seen) + + return results + + +def _collect_json_strings(value: Any) -> list[str]: + if isinstance(value, str): + return [value] + if isinstance(value, list): + strings: list[str] = [] + for item in value: + strings.extend(_collect_json_strings(item)) + return strings + if isinstance(value, dict): + strings = [] + for item in value.values(): + strings.extend(_collect_json_strings(item)) + return strings + return [] + + +def extract_from_json_message(segments: list[dict[str, Any]]) -> list[str]: + """从 QQ JSON 分享卡片中提取禁漫车号。""" + results: list[str] = [] + seen: set[str] = set() + + for segment in segments: + if segment.get("type") != "json": + continue + + # data 可能是字符串/列表等畸形结构:不能让它抛异常,否则同一条消息里 + # 后面所有段落(含合法卡片)的检测都会被丢掉 + segment_data = segment.get("data") + raw_data = ( + segment_data.get("data", "") if isinstance(segment_data, dict) else "" + ) + if not raw_data: + continue + + try: + payload = json.loads(html.unescape(str(raw_data))) + except (TypeError, json.JSONDecodeError): + logger.debug("[JM] JSON 消息解析失败,跳过", exc_info=True) + continue + + for item in _collect_json_strings(payload): + for book_id in extract_jm_ids(item): + if book_id in seen: + continue + seen.add(book_id) + results.append(book_id) + + return results diff --git a/src/Undefined/jm/sender.py b/src/Undefined/jm/sender.py new file mode 100644 index 00000000..f7d74bc4 --- /dev/null +++ b/src/Undefined/jm/sender.py @@ -0,0 +1,463 @@ +"""禁漫的合并转发发送与附件交付。 + +自动提取一次发送三个节点:本子信息、PDF 解密密码、PDF 文件(本地合成后随转发上传)。 +工具侧另有两条路:``format_jm_book_info`` 只输出详情(不下载), +``fetch_jm_book_attachment`` 下载整本并把**未加密** PDF 注册为附件 UID(不发送), +供 ``file_analysis_agent`` 继续解析。密码始终不会写进历史。 +""" + +from __future__ import annotations + +import logging +from pathlib import Path +import secrets +from typing import TYPE_CHECKING, Any, Literal + +from jmcpy import Book + +from Undefined.jm.downloader import ( + JmDownload, + cleanup_download_path, + download_book_pdf, +) + +if TYPE_CHECKING: + from Undefined.utils.sender import MessageSender + +logger = logging.getLogger(__name__) + +_BOT_NAME = "Undefined" +_DEFAULT_BOT_UIN = "10000" +#: 密码位数固定 8 位 +PASSWORD_LENGTH = 8 +#: 去掉形近字符(0/O、1/l/I),方便在手机上输入 +_PASSWORD_ALPHABET = "ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz23456789" +#: 公开站点车号链接(仅用于历史摘要,信息节点展示的是 ``JM<车号>``) +_ALBUM_URL_TEMPLATE = "https://18comic.vip/album/{book_id}" +_DESCRIPTION_PREVIEW_CHARS = 200 +_MAX_TAGS = 10 + + +def generate_pdf_password(length: int = PASSWORD_LENGTH) -> str: + """生成 PDF 打开密码。""" + return "".join(secrets.choice(_PASSWORD_ALPHABET) for _ in range(length)) + + +def build_album_url(book_id: int | str) -> str: + return _ALBUM_URL_TEMPLATE.format(book_id=book_id) + + +def _node(content: str | list[dict[str, Any]], *, name: str) -> dict[str, Any]: + return { + "type": "node", + "data": { + "name": name, + "uin": _DEFAULT_BOT_UIN, + "content": content, + }, + } + + +def _format_size(size_bytes: int | None) -> str: + if size_bytes is None: + return "未知" + return f"{size_bytes / 1024 / 1024:.1f}MB" + + +def _preview(text: str | None, limit: int = _DESCRIPTION_PREVIEW_CHARS) -> str: + normalized = " ".join(str(text or "").split()) + if len(normalized) <= limit: + return normalized + return normalized[:limit].rstrip() + "..." + + +def build_info_text(result: JmDownload, *, note: str = "") -> str: + """信息节点(也是历史摘要的来源)的正文。""" + book: Book = result.book + lines = [f"「JM{book.book_id} {book.title or '未知标题'}」"] + + if book.authors: + lines.append(f"作者: {', '.join(book.authors)}") + lines.append( + f"章节: {result.chapter_count} 章(本次合并 {result.downloaded_chapters} 章)" + ) + if book.tags: + lines.append(f"标签: {'、'.join(book.tags[:_MAX_TAGS])}") + + stats: list[str] = [] + if book.views is not None: + stats.append(f"观看 {book.views}") + if book.likes is not None: + stats.append(f"点赞 {book.likes}") + if stats: + lines.append(" | ".join(stats)) + + summary = f"页数: {result.page_count}" + if result.pdf_path is not None and result.size_bytes is not None: + summary += f" | PDF: {_format_size(result.size_bytes)}" + lines.append(summary) + if result.failed_pages: + lines.append(f"下载失败 {result.failed_pages} 页") + if note: + lines.append(note) + + description = _preview(book.description) + if description and description != book.title: + lines.extend(["---", description]) + + # 末尾给可复制的车号而不是站点链接:QQ 里点不开,车号还能直接再触发一次提取 + lines.extend(["---", f"JM{book.book_id}"]) + return "\n".join(lines) + + +def build_password_text(password: str) -> str: + return f"PDF 解密密码({len(password)} 位):{password}" + + +def build_forward_nodes( + info_text: str, + *, + pdf_path: Path | None = None, + password: str = "", + status_text: str = "", +) -> list[dict[str, Any]]: + """构建合并转发节点:信息 / 密码或状态 / PDF 文件。 + + PDF 是本地合成好的真实文件,随转发一起上传:群聊下 NapCat 会把它作为群文件上传 + (``isGroupFile``、``busid=102``,元素里带 ``fileId`` / ``fileMd5`` / ``fileSha1``), + 因此同一个 PDF 也会出现在群文件列表里,转发节点里的文件可以直接下载 + (2026-10-01 用 14MB PDF 实测通过)。文件只发一次,不另外发独立文件消息。 + """ + if pdf_path is None: + # 没有可发送的 PDF 时用状态说明替代密码与文件节点 + return [ + _node(info_text, name="本子信息"), + _node(status_text or "PDF 未发送", name="状态"), + ] + return [ + _node(info_text, name="本子信息"), + _node(build_password_text(password), name="解密密码"), + _node( + [ + { + "type": "file", + "data": { + "file": f"file://{pdf_path.resolve()}", + "name": pdf_path.name, + }, + } + ], + name="PDF", + ), + ] + + +def build_history_message(result: JmDownload) -> str: + """历史摘要:只写可复述的信息,不含 PDF 密码。""" + book = result.book + lines = [ + f"[JM] 「{book.title or '未知标题'}」", + f"车号: JM{book.book_id}", + f"章节: {result.chapter_count} 章 | 页数: {result.page_count}", + ] + if result.ok: + lines.append( + f"PDF: 已发送加密文件({_format_size(result.size_bytes)},密码见转发节点)" + ) + else: + lines.append("PDF: 未发送") + lines.append(build_album_url(book.book_id)) + return "\n".join(lines) + + +def format_jm_book_info(book: Book) -> str: + """本子详情的文本形式(工具 ``output_mode=info`` 用,不下载任何图片)。""" + lines = [f"「JM{book.book_id} {book.title or '未知标题'}」"] + if book.authors: + lines.append(f"作者: {', '.join(book.authors)}") + lines.append(f"章节: {len(book.chapters) or 1} 章") + if book.tags: + lines.append(f"标签: {'、'.join(book.tags[:_MAX_TAGS])}") + stats: list[str] = [] + if book.views is not None: + stats.append(f"观看 {book.views}") + if book.likes is not None: + stats.append(f"点赞 {book.likes}") + if book.comment_count is not None: + stats.append(f"评论 {book.comment_count}") + if stats: + lines.append(" | ".join(stats)) + + description = _preview(book.description) + if description and description != book.title: + lines.extend(["---", description]) + + lines.extend(["---", f"车号: JM{book.book_id}", build_album_url(book.book_id)]) + return "\n".join(lines) + + +def _build_uid_message( + result: JmDownload, + *, + uid: str, + file_name: str, +) -> str: + book = result.book + lines = [ + f"已获取禁漫本子 PDF:「{book.title or '未知标题'}」", + f"车号: JM{book.book_id}", + f"章节: {result.chapter_count} 章 | 页数: {result.page_count}", + f'PDF: ', + ] + if file_name: + lines.append(f"文件名: {file_name}") + if result.size_bytes is not None: + lines.append(f"大小: {_format_size(result.size_bytes)}") + lines.append("说明: 该 PDF 未加密,可直接解析") + lines.append(build_album_url(book.book_id)) + return "\n".join(lines) + + +async def fetch_jm_book_attachment( + *, + book_id: str, + attachment_registry: Any, + scope_key: str, + config: Any, +) -> str: + """下载整本并注册为附件 UID(工具 ``output_mode=uid``,不发送消息)。 + + 输出**未加密** PDF:这条路的用途是把文件交给 ``file_analysis_agent`` 解析, + 加密会让 ``extract_pdf`` / ``describe_pdf_page`` 打不开。 + """ + if attachment_registry is None: + return "缺少必要的运行时组件(attachment_registry)" + if not str(scope_key or "").strip(): + return "无法确定附件作用域,不能注册禁漫本子 PDF" + + result, task_dir = await download_book_pdf(book_id, config=config, password=None) + try: + if not result.ok or result.pdf_path is None: + reason = ( + f"PDF 超过体积上限(已下载 {result.page_count} 页)" + if result.status == "oversize" + else "没有下载到任何页面" + ) + return f"未能获取禁漫本子 PDF:{reason}|JM{book_id}" + record = await attachment_registry.register_local_file( + scope_key, + result.pdf_path, + kind="file", + display_name=result.pdf_path.name, + source_kind="jm_book", + source_ref=build_album_url(result.book.book_id), + segment_data={ + "book_id": result.book.book_id, + "title": result.book.title, + "page_count": result.page_count, + "chapters": result.chapter_count, + }, + ) + return _build_uid_message( + result, uid=str(record.uid), file_name=result.pdf_path.name + ) + finally: + await cleanup_download_path(task_dir) + + +async def _send_forward( + sender: "MessageSender", + target_type: Literal["group", "private"], + target_id: int, + nodes: list[dict[str, Any]], + *, + history_message: str, +) -> None: + if target_type == "group": + await sender.send_group_forward_message( + target_id, nodes, history_message=history_message + ) + else: + await sender.send_private_forward_message( + target_id, nodes, history_message=history_message + ) + + +async def _send_text( + sender: "MessageSender", + target_type: Literal["group", "private"], + target_id: int, + message: str, + *, + history_message: str | None = None, + auto_history: bool = True, +) -> None: + if target_type == "group": + await sender.send_group_message( + target_id, + message, + auto_history, + history_message=history_message, + ) + else: + await sender.send_private_message( + target_id, + message, + auto_history, + history_message=history_message, + ) + + +async def _send_file( + sender: "MessageSender", + target_type: Literal["group", "private"], + target_id: int, + file_path: str, + file_name: str, +) -> None: + if target_type == "group": + await sender.send_group_file(target_id, file_path, file_name) + else: + await sender.send_private_file(target_id, file_path, file_name) + + +def _is_fatal_delivery_error(exc: BaseException) -> bool: + """投递结果未确认 / 文件传输错误:绝不能降级重发(仓库约定)。 + + 判据与 ``bilibili/opus_sender.py`` 的 ``_FATAL_ERROR_FLAGS`` 一致:这类异常 + 可能已经送达,换个 action 再发一次会造成真实重复投递。 + """ + return bool(getattr(exc, "delivery_uncertain", False)) or bool( + getattr(exc, "file_transfer_error", False) + ) + + +async def _send_result( + sender: "MessageSender", + target_type: Literal["group", "private"], + target_id: int, + *, + book_id: str, + info_text: str, + password: str, + pdf_path: Path, + history_message: str, +) -> str: + """发送「信息 + 密码 + PDF」三节点合并转发,返回可记录的真实投递状态。 + + 文件只在转发里,不再额外发独立文件消息。群聊下这次上传会让 PDF 同时出现在群的 + 文件列表里(``busid=102``),那是 QQ 自己的行为,不是我们单独发出去的。 + 投递未确认 / 文件传输错误一律上抛,只有协议端**明确拒绝**才降级重发;降级时 + 文件没有别的入口,必须补发一条独立文件消息,否则用户拿不到 PDF。 + """ + nodes = build_forward_nodes(info_text, pdf_path=pdf_path, password=password) + try: + await _send_forward( + sender, target_type, target_id, nodes, history_message=history_message + ) + except Exception as exc: + if _is_fatal_delivery_error(exc): + logger.error("[JM] 合并转发投递结果未确认,不重发: book=%s", book_id) + raise + logger.exception("[JM] 合并转发失败,回退为普通消息 + 文件: book=%s", book_id) + # 历史里不留密码:历史摘要与用户可见正文分开;密码消息只记「已单独发送」 + await _send_text( + sender, + target_type, + target_id, + info_text, + history_message=history_message, + ) + await _send_text( + sender, + target_type, + target_id, + build_password_text(password), + history_message="[JM] PDF 解密密码已单独发送", + ) + try: + await _send_file( + sender, target_type, target_id, str(pdf_path), str(pdf_path.name) + ) + except Exception as file_exc: + if _is_fatal_delivery_error(file_exc): + logger.error("[JM] 文件投递结果未确认,不重发: book=%s", book_id) + raise + logger.exception("[JM] 降级后的 PDF 文件发送失败: book=%s", book_id) + await _send_text( + sender, + target_type, + target_id, + "PDF 上传失败,本次没有发送文件(信息与密码如上)。", + auto_history=False, + ) + return "合并转发与 PDF 文件均发送失败,已改为普通消息发送信息与密码" + return "合并转发被拒,已改为普通消息发送信息与密码,并补发独立 PDF 文件" + return "已发送合并转发与 PDF 文件" + + +async def send_jm_book( + book_id: str, + *, + sender: "MessageSender", + target_type: Literal["group", "private"], + target_id: int, + config: Any, +) -> str: + """下载整本并发送合并转发,返回可记录的状态说明。""" + password = generate_pdf_password() + result, task_dir = await download_book_pdf( + book_id, config=config, password=password + ) + try: + history_message = build_history_message(result) + pdf_path = result.pdf_path + if not result.ok or pdf_path is None: + note = ( + "PDF 未发送:文件超过体积上限" + if result.status == "oversize" + else "PDF 未发送:没有下载到任何页面" + ) + info_text = build_info_text(result, note=note) + nodes = build_forward_nodes(info_text, status_text=note) + try: + await _send_forward( + sender, + target_type, + target_id, + nodes, + history_message=history_message, + ) + except Exception as exc: + if _is_fatal_delivery_error(exc): + logger.error( + "[JM] 状态转发投递结果未确认,不重发: book=%s", book_id + ) + raise + logger.exception("[JM] 状态合并转发失败: book=%s", book_id) + await _send_text( + sender, + target_type, + target_id, + f"{info_text}\n{note}", + history_message=history_message, + ) + return f"JM{book_id} 已发送信息({note})" + + status = await _send_result( + sender, + target_type, + target_id, + book_id=book_id, + info_text=build_info_text(result), + password=password, + pdf_path=pdf_path, + history_message=history_message, + ) + return ( + f"JM{book_id} {status}" + f"({result.downloaded_chapters} 章 / {result.page_count} 页 / " + f"{_format_size(result.size_bytes)})" + ) + finally: + await cleanup_download_path(task_dir) diff --git a/src/Undefined/skills/README.md b/src/Undefined/skills/README.md index dcc71ca5..ca78dd17 100644 --- a/src/Undefined/skills/README.md +++ b/src/Undefined/skills/README.md @@ -14,9 +14,11 @@ skills/ │ ├── models.py │ ├── context.py │ ├── bilibili/ +│ ├── bilibili_opus/ │ ├── douyin/ │ ├── arxiv/ -│ └── github/ +│ ├── github/ +│ └── jm/ │ ├── tools/ # 基础小工具,直接暴露给 AI 调用 │ ├── __init__.py @@ -77,7 +79,7 @@ skills/ - **目录结构**: `pipelines/{pipeline_name}/config.json + handler.py`。 - **执行方式**: 同一条非命令消息会并行检测全部管线,并行处理全部命中结果;处理产出的消息通过统一发送层写入历史并自动登记本地媒体/文件附件后,再进入 AI 自动回复。 - **热重载**: 跟随 `[skills]` 的 `hot_reload`、`hot_reload_interval`、`hot_reload_debounce` 配置。 -- **示例**: `bilibili`, `douyin`, `arxiv`, `github` +- **示例**: `bilibili`, `bilibili_opus`, `douyin`, `arxiv`, `github`, `jm` ### 基础工具 diff --git a/src/Undefined/skills/agents/README.md b/src/Undefined/skills/agents/README.md index 1690dde9..19d380cc 100644 --- a/src/Undefined/skills/agents/README.md +++ b/src/Undefined/skills/agents/README.md @@ -275,10 +275,10 @@ mv skills/tools/my_tool skills/agents/my_agent/tools/ - **grok_search 参数**:使用 `search_request`,用自然语言完整叙述搜索要求,不要只传关键词。 ### file_analysis_agent(文件分析助手) -- **功能**:分析用户提供的附件、内部 UID、URL、legacy file_id、arXiv 论文标识或 Bilibili 视频标识,提取文件内容。 +- **功能**:分析用户提供的附件、内部 UID、URL、legacy file_id、arXiv 论文标识、Bilibili 视频标识或禁漫车号,提取文件内容。 - **适用场景**:PDF/Word/Excel/PPT/文本/代码/压缩包解析,图片、音频、视频等多模态内容识别,arXiv 论文 PDF 分析,Bilibili 视频内容分析。 - **不适用**:没有文件来源的开放式搜索、需要联网查资料的问题、执行文件或安全鉴定。 -- **子工具**:`download_file`, `detect_file_type`, `read_text_file`, `extract_pdf`, `describe_pdf_page`, `extract_docx`, `extract_xlsx`, `extract_pptx`, `extract_archive`, `analyze_code`, `analyze_multimodal`, `cleanup_temp`;还可调用共享主工具 `arxiv_paper(output_mode=uid)`、`bilibili_video(output_mode=uid)`、`bilibili_opus(output_mode=uid)` 与 `douyin_video(output_mode=uid)` 获取待分析附件 UID(图文返回 `pic_xxx`)。 +- **子工具**:`download_file`, `detect_file_type`, `read_text_file`, `extract_pdf`, `describe_pdf_page`, `extract_docx`, `extract_xlsx`, `extract_pptx`, `extract_archive`, `analyze_code`, `analyze_multimodal`, `cleanup_temp`;还可调用共享主工具 `arxiv_paper(output_mode=uid)`、`bilibili_video(output_mode=uid)`、`bilibili_opus(output_mode=uid)`、`douyin_video(output_mode=uid)` 与 `jm_book(output_mode=uid)` 获取待分析附件 UID(图文返回 `pic_xxx`,禁漫输出未加密 PDF)。 ### naga_code_analysis_agent(NagaAgent 代码分析助手) - **功能**:只读分析 NagaAgent 项目的结构、源码、配置、构建、部署和实现细节。 diff --git a/src/Undefined/skills/agents/file_analysis_agent/README.md b/src/Undefined/skills/agents/file_analysis_agent/README.md index f64dff9e..92b94552 100644 --- a/src/Undefined/skills/agents/file_analysis_agent/README.md +++ b/src/Undefined/skills/agents/file_analysis_agent/README.md @@ -7,7 +7,7 @@ - `intro.md`:能力说明 - `prompt.md`:系统提示词 - `tools/`:文件解析与分析工具 -- 共享主工具:通过 callable 仅可调用 `arxiv_paper(output_mode=uid)`、`bilibili_video(output_mode=uid)`、`bilibili_opus(output_mode=uid)` 与 `douyin_video(output_mode=uid)`,用于把 arXiv / Bilibili 视频与图文 / 抖音标识转换为当前会话附件 UID 后再分析(图文会把正文图片逐张登记为 `pic_xxx`) +- 共享主工具:通过 callable 仅可调用 `arxiv_paper(output_mode=uid)`、`bilibili_video(output_mode=uid)`、`bilibili_opus(output_mode=uid)`、`douyin_video(output_mode=uid)` 与 `jm_book(output_mode=uid)`,用于把 arXiv / Bilibili 视频与图文 / 抖音 / 禁漫车号转换为当前会话附件 UID 后再分析(图文会把正文图片逐张登记为 `pic_xxx`,禁漫车号输出未加密整本 PDF) 运行机制: - 由 `AgentRegistry` 自动发现并注册 diff --git a/src/Undefined/skills/agents/file_analysis_agent/config.json b/src/Undefined/skills/agents/file_analysis_agent/config.json index 183636cb..ae030353 100644 --- a/src/Undefined/skills/agents/file_analysis_agent/config.json +++ b/src/Undefined/skills/agents/file_analysis_agent/config.json @@ -2,7 +2,7 @@ "type": "function", "function": { "name": "file_analysis_agent", - "description": "文件分析助手,支持解析各种文件格式:文档(PDF、Word、PPT、Excel)、代码、压缩包、图片、音频、视频等。参数优先使用内部附件 UID,也兼容 URL、legacy file_id、arXiv ID/URL 和 Bilibili BV/AV/URL。", + "description": "文件分析助手,支持解析各种文件格式:文档(PDF、Word、PPT、Excel)、代码、压缩包、图片、音频、视频等。参数优先使用内部附件 UID,也兼容 URL、legacy file_id、arXiv ID/URL、Bilibili BV/AV/URL 和禁漫车号/链接。", "parameters": { "type": "object", "properties": { diff --git a/src/Undefined/skills/agents/file_analysis_agent/intro.md b/src/Undefined/skills/agents/file_analysis_agent/intro.md index 014a9cbc..0a1d2af9 100644 --- a/src/Undefined/skills/agents/file_analysis_agent/intro.md +++ b/src/Undefined/skills/agents/file_analysis_agent/intro.md @@ -1,6 +1,6 @@ # 文件分析助手 -用于分析用户提供的附件、内部 UID、URL、legacy file_id、arXiv 论文标识或 Bilibili 视频标识,并从文件内容中识别、提取、摘要或统计信息。 +用于分析用户提供的附件、内部 UID、URL、legacy file_id、arXiv 论文标识、Bilibili 视频标识或禁漫(JM)车号,并从文件内容中识别、提取、摘要或统计信息。 可处理: - PDF、Word、Excel、PPT、文本、代码和压缩包 diff --git a/src/Undefined/skills/agents/file_analysis_agent/prompt.md b/src/Undefined/skills/agents/file_analysis_agent/prompt.md index 03183f1d..bae93718 100644 --- a/src/Undefined/skills/agents/file_analysis_agent/prompt.md +++ b/src/Undefined/skills/agents/file_analysis_agent/prompt.md @@ -14,6 +14,7 @@ - 如果文件源是 arXiv ID、`arXiv:...`、`arxiv.org/abs/...` 或 `arxiv.org/pdf/...`,先调用共享工具 `arxiv_paper`,设置 `output_mode="uid"`,拿到 `` 后再按普通文件 UID 下载和分析。 - 如果文件源是 Bilibili BV 号、AV 号、B 站视频链接或 b23.tv 短链,先调用共享工具 `bilibili_video`,设置 `output_mode="uid"`,拿到 `` 后再按普通视频 UID 下载和分析。 - 如果文件源是抖音 `v.douyin.com` 短链、`douyin.com/video/` 长链或裸 aweme_id,先调用共享工具 `douyin_video`,设置 `output_mode="uid"`,拿到 `` 后再按普通视频 UID 下载和分析。 +- 如果文件源是禁漫车号或链接(`JM350234`、`350234`、`18comic.vip/album/350234`、`?id=350234`),先调用共享工具 `jm_book`,设置 `output_mode="uid"`,拿到 `` 后再按普通 PDF UID 用 `extract_pdf` / `describe_pdf_page` 分析。 - 如果文件源是 Bilibili 图文(`bilibili.com/opus/`、`t.bilibili.com/`、b23.tv 短链或裸动态 ID),先调用共享工具 `bilibili_opus`,设置 `output_mode="uid"`,拿到 `` 后再用 `analyze_multimodal` 逐张分析其中的图片;正文文字可从同一返回结果里读取,不需要再抓页面。 - 已经给出内部附件 UID 时,不要再调用 arXiv/Bilibili/抖音获取工具。 - 根据用户目标选择合适工具:文本读取、文件类型检测、PDF/Office/表格/代码/压缩包/多模态分析都按内容类型处理。 diff --git a/src/Undefined/skills/http_config.py b/src/Undefined/skills/http_config.py index 5d3f1299..c13c149f 100644 --- a/src/Undefined/skills/http_config.py +++ b/src/Undefined/skills/http_config.py @@ -74,6 +74,7 @@ def _scope_use_proxy(config: Any, proxy_scope: str) -> bool: "douyin": "douyin_use_proxy", "arxiv": "arxiv_use_proxy", "github": "github_use_proxy", + "jm": "jm_use_proxy", "naga": "naga.use_proxy", "api_callback": "api.tool_invoke_callback_use_proxy", } diff --git a/src/Undefined/skills/pipelines/context.py b/src/Undefined/skills/pipelines/context.py index 791e5543..ce1703c1 100644 --- a/src/Undefined/skills/pipelines/context.py +++ b/src/Undefined/skills/pipelines/context.py @@ -34,9 +34,11 @@ def build_pipeline_context( "extract_douyin_ids": handler._extract_douyin_ids, "extract_arxiv_ids": handler._extract_arxiv_ids, "extract_github_repo_ids": handler._extract_github_repo_ids, + "extract_jm_ids": handler._extract_jm_ids, "handle_bilibili_extract": handler._handle_bilibili_extract, "handle_bilibili_opus_extract": handler._handle_bilibili_opus_extract, "handle_douyin_extract": handler._handle_douyin_extract, "handle_arxiv_extract": handler._handle_arxiv_extract, "handle_github_extract": handler._handle_github_extract, + "handle_jm_extract": handler._handle_jm_extract, } diff --git a/src/Undefined/skills/pipelines/jm/config.json b/src/Undefined/skills/pipelines/jm/config.json new file mode 100644 index 00000000..03901819 --- /dev/null +++ b/src/Undefined/skills/pipelines/jm/config.json @@ -0,0 +1,6 @@ +{ + "name": "jm", + "description": "检测并处理禁漫(JM)车号或链接自动提取。", + "order": 12, + "enabled": true +} diff --git a/src/Undefined/skills/pipelines/jm/handler.py b/src/Undefined/skills/pipelines/jm/handler.py new file mode 100644 index 00000000..92c93a8a --- /dev/null +++ b/src/Undefined/skills/pipelines/jm/handler.py @@ -0,0 +1,46 @@ +from __future__ import annotations + +from typing import Any + +from Undefined.skills.pipelines.models import ( + PipelineContext, + PipelineDetection, +) + + +def _is_allowed(config: Any, target_type: str, target_id: int) -> bool: + if not getattr(config, "jm_auto_extract_enabled", False): + return False + if target_type == "group": + return bool(config.is_jm_auto_extract_allowed_group(target_id)) + return bool(config.is_jm_auto_extract_allowed_private(target_id)) + + +async def detect(context: PipelineContext) -> PipelineDetection | None: + target_id = int(context["target_id"]) + target_type = str(context["target_type"]) + config = context["config"] + if not _is_allowed(config, target_type, target_id): + return None + + extractor = context["extract_jm_ids"] + book_ids = extractor(context["text"], context["message_content"]) + if not book_ids: + return None + return PipelineDetection(name="jm", items=tuple(str(item) for item in book_ids)) + + +async def process( + detection: PipelineDetection, + context: PipelineContext, +) -> None: + handler = context["handle_jm_extract"] + args = ( + int(context["target_id"]), + list(detection.items), + str(context["target_type"]), + ) + if context.get("address") is None: + await handler(*args) + else: + await handler(*args, context["sender"]) diff --git a/src/Undefined/skills/shared.py b/src/Undefined/skills/shared.py index 01e2eaad..ffabc5f3 100644 --- a/src/Undefined/skills/shared.py +++ b/src/Undefined/skills/shared.py @@ -58,4 +58,75 @@ def parse_positive_int( return parsed, None -__all__ = ["parse_positive_int", "private_access_error"] +def jm_normalize_book_id(value: str) -> str | None: + """把车号 / ``JM<数字>`` / 禁漫链接归一化成纯数字车号(失败返回 `None`)。""" + from Undefined.jm.parser import normalize_jm_id + + return normalize_jm_id(value) + + +def jm_scope_key(context: dict[str, Any]) -> str: + """当前会话的附件作用域:优先取上下文里已解析的 ``scope_key``。""" + scope_key = str(context.get("scope_key") or "").strip() + if scope_key: + return scope_key + from Undefined.attachments import scope_from_context + + return scope_from_context(context) or "" + + +async def jm_book_info_text(book_id: str, *, config: Any) -> str: + """只取本子详情(不下载),返回可直接给用户/模型的文本。""" + from Undefined.jm.downloader import fetch_book + from Undefined.jm.sender import format_jm_book_info + + return format_jm_book_info(await fetch_book(book_id, config=config)) + + +async def jm_book_attachment_text( + book_id: str, + *, + registry: Any, + scope_key: str, + config: Any, +) -> str: + """下载整本并注册为未加密 PDF 附件 UID,返回概要文本。""" + from Undefined.jm.sender import fetch_jm_book_attachment + + return await fetch_jm_book_attachment( + book_id=book_id, + attachment_registry=registry, + scope_key=scope_key, + config=config, + ) + + +async def jm_send_book( + book_id: str, + *, + sender: Any, + target_type: str, + target_id: int, + config: Any, +) -> str: + """下载整本并发送「信息 / 密码 / PDF」三节点合并转发。""" + from Undefined.jm.sender import send_jm_book as _send_jm_book + + return await _send_jm_book( + book_id, + sender=sender, + target_type=target_type, # type: ignore[arg-type] + target_id=target_id, + config=config, + ) + + +__all__ = [ + "jm_book_attachment_text", + "jm_book_info_text", + "jm_normalize_book_id", + "jm_scope_key", + "jm_send_book", + "parse_positive_int", + "private_access_error", +] diff --git a/src/Undefined/skills/tools/jm_book/README.md b/src/Undefined/skills/tools/jm_book/README.md new file mode 100644 index 00000000..08063b28 --- /dev/null +++ b/src/Undefined/skills/tools/jm_book/README.md @@ -0,0 +1,33 @@ +# jm_book 工具 + +处理禁漫(JM / 18comic)本子。默认下载整本并通过合并转发发送;也支持只注册为附件 UID 供文件分析使用,或只获取本子信息。支持 `JM350234`、`350234`、`https://18comic.vip/album/350234` 与 `?id=` 链接。 + +常用参数: +- `book_id`:本子车号(`JM350234` / `350234` / 禁漫链接,车号 5-8 位) +- `target_type`:可选,目标会话类型(`group`/`private`) +- `target_id`:可选,目标会话 ID +- `output_mode`:可选,`send`(默认,发送三节点合并转发)、`uid`(只返回 ``,不发送消息)或 `info`(只返回本子信息,不下载不发送) + +`send` 模式流程: +1. 解析车号 +2. 逐章下载并按章节顺序合成为一个 AES-256 加密 PDF(`[jm].max_chapters` / `max_file_size` 限制) +3. 发送「本子信息 / PDF 解密密码 / PDF 文件」三节点合并转发,PDF 同时进入群文件列表 +4. 密码随机生成 8 位,只出现在转发节点里,不写入历史 + +`uid` 模式流程: +1. 下载整本并合成为**未加密** PDF(加密会让 PDF 解析工具打不开) +2. 注册为当前会话附件 UID +3. 返回本子概要和 ``,供 `file_analysis_agent` 继续解析 + +`info` 模式流程: +1. 调用移动端接口获取本子详情 +2. 返回标题、作者、章节数、标签、观看/点赞/评论、简介预览、车号与站点链接 +3. 不下载、不发送、不注册附件 + +配置依赖: +- `config.toml` 中的 `[jm]` 段控制功能开关、白名单、体积上限、页数上限、DPI 与画质等 + +目录结构: +- `config.json`:工具定义 +- `callable.json`:共享授权(暴露给 `file_analysis_agent`) +- `handler.py`:执行逻辑 diff --git a/src/Undefined/skills/tools/jm_book/callable.json b/src/Undefined/skills/tools/jm_book/callable.json new file mode 100644 index 00000000..8781b6c6 --- /dev/null +++ b/src/Undefined/skills/tools/jm_book/callable.json @@ -0,0 +1,4 @@ +{ + "enabled": true, + "allowed_callers": ["file_analysis_agent"] +} diff --git a/src/Undefined/skills/tools/jm_book/config.json b/src/Undefined/skills/tools/jm_book/config.json new file mode 100644 index 00000000..212ae8d9 --- /dev/null +++ b/src/Undefined/skills/tools/jm_book/config.json @@ -0,0 +1,31 @@ +{ + "type": "function", + "function": { + "name": "jm_book", + "description": "处理禁漫(JM / 18comic)本子。默认下载整本并通过合并转发发送;可设置 output_mode=uid 只注册 PDF 附件 UID,或 output_mode=info 只获取本子信息。支持 JM350234、350234 或禁漫链接。", + "parameters": { + "type": "object", + "properties": { + "book_id": { + "type": "string", + "description": "本子车号:如 JM350234、350234 或 https://18comic.vip/album/350234" + }, + "target_type": { + "type": "string", + "enum": ["group", "private"], + "description": "可选。目标会话类型" + }, + "target_id": { + "type": "integer", + "description": "可选。目标会话 ID" + }, + "output_mode": { + "type": "string", + "enum": ["send", "uid", "info"], + "description": "输出模式:send(默认,下载整本并发送「信息 / 解密密码 / PDF」三节点合并转发)、uid(只下载并注册未加密 PDF 附件 UID,不发送消息)、info(只返回本子信息,不下载不发送)" + } + }, + "required": ["book_id"] + } + } +} diff --git a/src/Undefined/skills/tools/jm_book/handler.py b/src/Undefined/skills/tools/jm_book/handler.py new file mode 100644 index 00000000..62b79ab3 --- /dev/null +++ b/src/Undefined/skills/tools/jm_book/handler.py @@ -0,0 +1,95 @@ +from __future__ import annotations + +import logging +from typing import Any, Literal + +from Undefined.skills.shared import ( + jm_book_attachment_text, + jm_book_info_text, + jm_normalize_book_id, + jm_scope_key, + jm_send_book, +) + +logger = logging.getLogger(__name__) + + +def _resolve_target( + args: dict[str, Any], context: dict[str, Any] +) -> tuple[tuple[Literal["group", "private"], int] | None, str | None]: + target_type_raw = args.get("target_type") + target_id_raw = args.get("target_id") + + if target_type_raw is not None and target_id_raw is not None: + target_type = str(target_type_raw).strip().lower() + if target_type not in ("group", "private"): + return None, "target_type 只能是 group 或 private" + try: + target_id = int(target_id_raw) + except (TypeError, ValueError): + return None, "target_id 必须是整数" + return (target_type, target_id), None # type: ignore[return-value] + + request_type = context.get("request_type") + if request_type == "group" and context.get("group_id"): + return ("group", int(context["group_id"])), None + if request_type == "private" and context.get("user_id"): + return ("private", int(context["user_id"])), None + + if context.get("group_id"): + return ("group", int(context["group_id"])), None + if context.get("user_id"): + return ("private", int(context["user_id"])), None + return None, "无法确定目标会话,请提供 target_type 与 target_id" + + +async def execute(args: dict[str, Any], context: dict[str, Any]) -> str: + raw_book_id = str(args.get("book_id", "")).strip() + if not raw_book_id: + return "book_id 不能为空" + + book_id = jm_normalize_book_id(raw_book_id) + if book_id is None: + return f"无法解析禁漫车号: {raw_book_id}(支持 JM350234、350234 或禁漫链接,车号 5-8 位)" + + output_mode = str(args.get("output_mode", "send") or "send").strip().lower() + if output_mode not in {"send", "uid", "info"}: + return "output_mode 只能是 send、uid 或 info" + + runtime_config = context.get("runtime_config") + + try: + if output_mode == "info": + return await jm_book_info_text(book_id, config=runtime_config) + + if output_mode == "uid": + if runtime_config is None: + return "缺少必要的运行时组件(runtime_config)" + return await jm_book_attachment_text( + book_id, + registry=context.get("attachment_registry"), + scope_key=jm_scope_key(context), + config=runtime_config, + ) + + target, error = _resolve_target(args, context) + if error or target is None: + return f"目标解析失败: {error or '参数错误'}" + target_type, target_id = target + + sender = context.get("sender") + if sender is None: + return "缺少必要的运行时组件(sender)" + if runtime_config is None: + return "缺少必要的运行时组件(runtime_config)" + + return await jm_send_book( + book_id, + sender=sender, + target_type=target_type, + target_id=target_id, + config=runtime_config, + ) + except Exception as exc: + logger.exception("[jm_book] 执行失败: %s", exc) + return f"禁漫本子处理失败: {exc}" diff --git a/tests/test_ai_client_setup_paths.py b/tests/test_ai_client_setup_paths.py index 2932a78f..51caef9c 100644 --- a/tests/test_ai_client_setup_paths.py +++ b/tests/test_ai_client_setup_paths.py @@ -118,4 +118,5 @@ async def _load() -> PipelineRegistry: "bilibili_opus", "douyin", "github", + "jm", } diff --git a/tests/test_handlers_jm_auto_extract.py b/tests/test_handlers_jm_auto_extract.py new file mode 100644 index 00000000..88cd7028 --- /dev/null +++ b/tests/test_handlers_jm_auto_extract.py @@ -0,0 +1,154 @@ +from __future__ import annotations + +from types import SimpleNamespace +from typing import Any +from unittest.mock import AsyncMock, MagicMock + +import pytest + +import Undefined.handlers as handlers_module +import Undefined.jm.sender as jm_sender_module +from Undefined.handlers import MessageHandler +from Undefined.skills.pipelines import PipelineRegistry + + +@pytest.mark.asyncio +async def test_private_message_runs_jm_auto_extract_before_ai_reply( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr( + handlers_module, + "parse_message_content_for_history", + AsyncMock(return_value="jm350234"), + ) + + handler: Any = MessageHandler.__new__(MessageHandler) + handler.config = SimpleNamespace( + bot_qq=10000, + is_private_allowed=lambda _uid: True, + access_control_enabled=lambda: False, + should_process_private_message=lambda: True, + bilibili_auto_extract_enabled=False, + arxiv_auto_extract_enabled=False, + github_auto_extract_enabled=False, + jm_auto_extract_enabled=True, + jm_auto_extract_max_items=1, + is_jm_auto_extract_allowed_private=lambda _uid: True, + ) + handler.onebot = SimpleNamespace( + get_stranger_info=AsyncMock(return_value={"nickname": "测试用户"}), + get_msg=AsyncMock(), + get_forward_msg=AsyncMock(), + ) + handler.sender = SimpleNamespace() + handler.history_manager = SimpleNamespace(add_private_message=AsyncMock()) + handler.ai_coordinator = SimpleNamespace( + model_pool=SimpleNamespace( + handle_private_message=AsyncMock(return_value=False) + ), + handle_private_reply=AsyncMock(), + ) + handler.command_dispatcher = SimpleNamespace( + parse_command=MagicMock(return_value=None) + ) + handler._background_tasks = set() + handler._extract_jm_ids = MagicMock(return_value=["350234"]) + handler._handle_jm_extract = AsyncMock() + handler.pipeline_registry = PipelineRegistry() + handler.pipeline_registry.load_items() + handler._spawn_background_task = MagicMock() + + event = { + "post_type": "message", + "message_type": "private", + "user_id": 20001, + "message_id": 30001, + "message": [{"type": "text", "data": {"text": "jm350234"}}], + "sender": {"user_id": 20001, "nickname": "测试用户"}, + } + + await handler.handle_message(event) + + handler._extract_jm_ids.assert_called_once() + handler._handle_jm_extract.assert_awaited_once_with(20001, ["350234"], "private") + handler._spawn_background_task.assert_not_called() + handler.ai_coordinator.model_pool.handle_private_message.assert_not_called() + handler.command_dispatcher.parse_command.assert_called_once_with("jm350234") + handler.ai_coordinator.handle_private_reply.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_jm_auto_extract_truncates_items_and_reports_failure( + monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture +) -> None: + sent: list[str] = [] + + async def _send(book_id: str, **_kwargs: Any) -> str: + sent.append(book_id) + if book_id == "2222222": + raise RuntimeError("源站 404") + return f"JM{book_id} 已发送" + + monkeypatch.setattr(jm_sender_module, "send_jm_book", _send) + + handler: Any = MessageHandler.__new__(MessageHandler) + handler.config = SimpleNamespace(jm_auto_extract_max_items=2) + handler.sender = SimpleNamespace(send_group_message=AsyncMock()) + + with caplog.at_level("INFO"): + await handler._handle_jm_extract( + 20001, ["1111111", "2222222", "3333333"], "group" + ) + + # max_items=2:第三个车号被丢弃,不再发起下载 + assert sent == ["1111111", "2222222"] + error_calls = handler.sender.send_group_message.await_args_list + assert error_calls, "失败时要给会话一行提示" + assert "JM 提取失败" in error_calls[0].args[1] + assert "源站 404" in error_calls[0].args[1] + assert "RuntimeError" in caplog.text + + +@pytest.mark.asyncio +async def test_jm_auto_extract_passes_custom_sender( + monkeypatch: pytest.MonkeyPatch, +) -> None: + captured: dict[str, Any] = {} + + async def _send(book_id: str, **kwargs: Any) -> str: + captured["book_id"] = book_id + captured["sender"] = kwargs.get("sender") + return "ok" + + monkeypatch.setattr(jm_sender_module, "send_jm_book", _send) + + handler: Any = MessageHandler.__new__(MessageHandler) + handler.config = SimpleNamespace(jm_auto_extract_max_items=3) + handler.sender = SimpleNamespace(send_group_message=AsyncMock()) + address_sender = SimpleNamespace(send_group_message=AsyncMock()) + + await handler._handle_jm_extract(20001, ["350234"], "group", address_sender) + + assert captured["book_id"] == "350234" + assert captured["sender"] is address_sender + + +@pytest.mark.asyncio +async def test_jm_auto_extract_does_nothing_without_ids( + monkeypatch: pytest.MonkeyPatch, +) -> None: + called: list[str] = [] + + async def _send(book_id: str, **_kwargs: Any) -> str: + called.append(book_id) + return "ok" + + monkeypatch.setattr(jm_sender_module, "send_jm_book", _send) + + handler: Any = MessageHandler.__new__(MessageHandler) + handler.config = SimpleNamespace(jm_auto_extract_max_items=3) + handler.sender = SimpleNamespace(send_group_message=AsyncMock()) + + await handler._handle_jm_extract(20001, [], "group") + + assert called == [] diff --git a/tests/test_jm_config.py b/tests/test_jm_config.py new file mode 100644 index 00000000..8517e780 --- /dev/null +++ b/tests/test_jm_config.py @@ -0,0 +1,200 @@ +from __future__ import annotations + +from pathlib import Path + +import pytest + +from Undefined.config.loader import Config + + +def _load_config(tmp_path: Path, extra_toml: str) -> Config: + config_path = tmp_path / "config.toml" + config_path.write_text( + ( + "[core]\n" + "bot_qq = 10001\n" + "superadmin_qq = 20002\n\n" + "[onebot]\n" + 'ws_url = "ws://127.0.0.1:3001"\n\n' + f"{extra_toml}\n" + ), + encoding="utf-8", + ) + return Config.load(config_path=config_path, strict=False) + + +def test_jm_defaults_disable_feature(tmp_path: Path) -> None: + config = _load_config(tmp_path, "") + + assert config.jm_auto_extract_enabled is False + assert config.jm_use_proxy is False + assert config.jm_auto_extract_group_ids == [] + assert config.jm_auto_extract_private_ids == [] + assert config.jm_auto_extract_max_items == 1 + assert config.jm_max_file_size == 100 + assert config.jm_max_chapters == 0 + assert config.jm_pdf_dpi == 150.0 + assert config.jm_image_quality == 95 + assert config.jm_download_concurrency == 8 + assert config.jm_request_timeout == 20.0 + assert config.jm_image_timeout == 60.0 + assert config.jm_session_dir == "" + + +def test_jm_config_clamps_invalid_values(tmp_path: Path) -> None: + config = _load_config( + tmp_path, + ( + "[jm]\n" + "auto_extract_enabled = true\n" + "auto_extract_max_items = 99\n" + "max_file_size = -5\n" + "max_chapters = -1\n" + "pdf_dpi = 0\n" + "image_quality = 0\n" + "download_concurrency = 0\n" + "request_timeout = 0\n" + "image_timeout = 0\n" + 'session_dir = "/tmp/jmcpy-home"\n' + ), + ) + + assert config.jm_auto_extract_enabled is True + assert config.jm_auto_extract_max_items == 5 + assert config.jm_max_file_size == 100 + assert config.jm_max_chapters == 0 + assert config.jm_pdf_dpi == 150.0 + assert config.jm_image_quality == 95 + assert config.jm_download_concurrency == 8 + assert config.jm_request_timeout == 20.0 + assert config.jm_image_timeout == 60.0 + assert config.jm_session_dir == "/tmp/jmcpy-home" + + +def test_jm_config_keeps_explicit_values(tmp_path: Path) -> None: + config = _load_config( + tmp_path, + ( + "[jm]\n" + "auto_extract_enabled = true\n" + "use_proxy = true\n" + "auto_extract_max_items = 3\n" + "max_file_size = 0\n" + "max_chapters = 2\n" + "pdf_dpi = 200.0\n" + "image_quality = 100\n" + "download_concurrency = 32\n" + "request_timeout = 45.0\n" + "image_timeout = 120.0\n" + ), + ) + + assert config.jm_use_proxy is True + assert config.jm_auto_extract_max_items == 3 + assert config.jm_max_file_size == 0 + assert config.jm_max_chapters == 2 + assert config.jm_pdf_dpi == 200.0 + assert config.jm_image_quality == 100 + assert config.jm_download_concurrency == 32 + assert config.jm_request_timeout == 45.0 + assert config.jm_image_timeout == 120.0 + + +def test_jm_auto_extract_allowlist_follows_global_access_when_empty( + tmp_path: Path, +) -> None: + config = _load_config( + tmp_path, + ( + "[access]\n" + 'mode = "allowlist"\n' + "allowed_group_ids = [123456]\n" + "allowed_private_ids = [20003]\n\n" + "[jm]\n" + "auto_extract_enabled = true\n" + ), + ) + + assert config.is_jm_auto_extract_allowed_group(123456) is True + assert config.is_jm_auto_extract_allowed_group(654321) is False + assert config.is_jm_auto_extract_allowed_private(20003) is True + assert config.is_jm_auto_extract_allowed_private(30004) is False + + +def test_jm_auto_extract_allowlist_overrides_global_access_when_non_empty( + tmp_path: Path, +) -> None: + config = _load_config( + tmp_path, + ( + "[access]\n" + 'mode = "allowlist"\n' + "allowed_group_ids = [123456]\n" + "allowed_private_ids = [20003]\n\n" + "[jm]\n" + "auto_extract_enabled = true\n" + "auto_extract_group_ids = [654321]\n" + "auto_extract_private_ids = [30004]\n" + ), + ) + + assert config.is_jm_auto_extract_allowed_group(123456) is False + assert config.is_jm_auto_extract_allowed_group(654321) is True + assert config.is_jm_auto_extract_allowed_private(20003) is False + assert config.is_jm_auto_extract_allowed_private(30004) is True + + +def test_jm_use_proxy_reads_environment( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setenv("JM_USE_PROXY", "true") + config = _load_config(tmp_path, "") + + assert config.jm_use_proxy is True + + +def test_jm_config_clamps_upper_bounds(tmp_path: Path) -> None: + config = _load_config( + tmp_path, + ( + "[jm]\n" + "auto_extract_max_items = 0\n" + "image_quality = 1000\n" + "download_concurrency = 999\n" + "request_timeout = 1000.0\n" + "image_timeout = 100000.0\n" + ), + ) + + assert config.jm_auto_extract_max_items == 1 + assert config.jm_image_quality == 100 + assert config.jm_download_concurrency == 32 + assert config.jm_request_timeout == 120.0 + assert config.jm_image_timeout == 600.0 + + +def test_jm_config_rejects_negative_values(tmp_path: Path) -> None: + """负值必须被回退成默认值,而不是原样传下去。""" + config = _load_config( + tmp_path, + ( + "[jm]\n" + "auto_extract_max_items = -3\n" + "max_file_size = -1\n" + "max_chapters = -9\n" + "pdf_dpi = -150.0\n" + "image_quality = -5\n" + "download_concurrency = -8\n" + "request_timeout = -1.0\n" + "image_timeout = -1.0\n" + ), + ) + + assert config.jm_auto_extract_max_items == 1 + assert config.jm_max_file_size == 100 + assert config.jm_max_chapters == 0 + assert config.jm_pdf_dpi == 150.0 + assert config.jm_image_quality == 95 + assert config.jm_download_concurrency == 8 + assert config.jm_request_timeout == 20.0 + assert config.jm_image_timeout == 60.0 diff --git a/tests/test_jm_downloader.py b/tests/test_jm_downloader.py new file mode 100644 index 00000000..f1d459aa --- /dev/null +++ b/tests/test_jm_downloader.py @@ -0,0 +1,583 @@ +from __future__ import annotations + +from pathlib import Path +from types import SimpleNamespace +from typing import Any, Literal + +import asyncio + +import fitz +import pytest +from jmcpy.imaging import block_count, descramble +from jmcpy import Book +from jmcpy.models import Picture +from PIL import Image + +import Undefined.jm.client as jm_client +import Undefined.jm.downloader as jm_downloader +from Undefined.jm.downloader import JmDownload, download_book_pdf + + +class _FakeChapter: + def __init__(self, chapter_id: int, pages: int) -> None: + self.chapter_id = chapter_id + self.pictures = tuple(f"{index:05d}.webp" for index in range(1, pages + 1)) + + def __len__(self) -> int: + return len(self.pictures) + + +class _FakeArtifact: + def __init__(self, path: Path, size: int, picture: Any) -> None: + self.path = path + self.size = size + self.picture = picture + + +class _FakeDownloadResult: + def __init__(self, paths: list[Path], page_size: int, failures: int = 0) -> None: + self.paths = tuple(paths) + self.failures = tuple(object() for _ in range(failures)) + self._page_size = page_size + self._artifacts = [ + _FakeArtifact(path, page_size, _fake_picture(path)) for path in paths + ] + + def __iter__(self) -> Any: + return iter(self._artifacts) + + +def _fake_picture(path: Path) -> Any: + return SimpleNamespace( + filename=path.name, + chapter_id=111, + scramble_id=0, + suffix=path.suffix, + ) + + +def _fake_book(chapter_ids: list[tuple[int, int]]) -> Any: + return SimpleNamespace( + book_id=1114751, + title="测试 本子/标题", + chapters=tuple( + SimpleNamespace(chapter_id=chapter_id, order=order) + for chapter_id, order in chapter_ids + ), + ) + + +class _FakeClient: + """记录调用参数的 jmcpy Client 替身。""" + + instances: list["_FakeClient"] = [] + + def __init__( + self, settings: Any, *, chapter_ids: list[tuple[int, int]], page_size: int + ) -> None: + self.settings = settings + self.chapter_ids = chapter_ids + self.page_size = page_size + self.downloads: list[dict[str, Any]] = [] + _FakeClient.instances.append(self) + + def __enter__(self) -> "_FakeClient": + return self + + def __exit__(self, *exc_info: object) -> Literal[False]: + return False + + def get_book(self, book_id: str) -> Any: + self.book_id = book_id + return _fake_book(self.chapter_ids) + + def get_chapter(self, chapter_id: int) -> _FakeChapter: + self.chapter_id = chapter_id + # 第一章没有图片,用于验证空章节被跳过 + return _FakeChapter(chapter_id, 0 if chapter_id == 999 else 2) + + def download( + self, + chapter: _FakeChapter, + *, + output: Any, + dest: Path, + decode: bool, + concurrency: int, + ) -> _FakeDownloadResult: + dest.mkdir(parents=True, exist_ok=True) + paths: list[Path] = [] + for name in chapter.pictures: + path = dest / name + path.write_bytes(b"page") + paths.append(path) + self.downloads.append( + { + "chapter_id": chapter.chapter_id, + "output": output, + "dest": dest, + "concurrency": concurrency, + "decode": decode, + } + ) + return _FakeDownloadResult(paths, page_size=self.page_size) + + +@pytest.fixture(autouse=True) +def _patch_env(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + _FakeClient.instances.clear() + monkeypatch.setattr(jm_downloader, "_JM_DOWNLOAD_DIR", tmp_path / "jm") + + +def _install_client( + monkeypatch: pytest.MonkeyPatch, + *, + chapter_ids: list[tuple[int, int]], + page_size: int = 10, +) -> None: + def _factory(settings: Any) -> _FakeClient: + return _FakeClient(settings, chapter_ids=chapter_ids, page_size=page_size) + + monkeypatch.setattr(jm_downloader, "Client", _factory) + + +def _capture_write_pdf(monkeypatch: pytest.MonkeyPatch) -> dict[str, Any]: + captured: dict[str, Any] = {} + + def _write_pdf( + pages: Any, + output: Path, + *, + dpi: float, + quality: int, + password: str | None, + limit_bytes: int | None, + ) -> Any: + captured["pages"] = list(pages) + captured["sources"] = [path for path, _picture in pages] + captured["dpi"] = dpi + captured["quality"] = quality + captured["password"] = password + captured["limit_bytes"] = limit_bytes + output.parent.mkdir(parents=True, exist_ok=True) + output.write_bytes(b"%PDF-1.4") + return jm_downloader._PdfBuild( + "ok", output, len(captured["pages"]), output.stat().st_size, 0 + ) + + monkeypatch.setattr(jm_downloader, "_write_pdf", _write_pdf) + return captured + + +def _config(**overrides: Any) -> Any: + base: dict[str, Any] = { + "jm_session_dir": "", + "jm_request_timeout": 20.0, + "jm_image_timeout": 60.0, + "jm_download_concurrency": 4, + "jm_max_chapters": 0, + "jm_max_file_size": 100, + "jm_pdf_dpi": 150.0, + "jm_image_quality": 95, + } + base.update(overrides) + return SimpleNamespace(**base) + + +@pytest.mark.asyncio +async def test_download_book_pdf_merges_all_chapters_in_order( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_client(monkeypatch, chapter_ids=[(111, 1), (222, 2)]) + captured = _capture_write_pdf(monkeypatch) + + result, task_dir = await download_book_pdf( + "1114751", config=_config(), password="Ab3xK9Qm" + ) + + assert result.status == "ok" + assert result.chapter_count == 2 + assert result.downloaded_chapters == 2 + assert result.page_count == 4 + assert result.pdf_path is not None + assert result.pdf_path.name == "JM1114751 测试 本子_标题.pdf" + assert result.pdf_path.parent == task_dir + # 页序 = 章节顺序 × 章节内页序,且每章写入独立目录 + assert [path.name for path in captured["sources"]] == [ + "00001.webp", + "00002.webp", + "00001.webp", + "00002.webp", + ] + assert captured["password"] == "Ab3xK9Qm" + assert captured["dpi"] == 150.0 + client = _FakeClient.instances[0] + assert [item["chapter_id"] for item in client.downloads] == [111, 222] + assert client.downloads[0]["dest"] != client.downloads[1]["dest"] + assert client.downloads[0]["concurrency"] == 4 + # 取服务端原始字节:解扰与编码交给写 PDF 那一步,避免多一代有损压缩 + assert client.downloads[0]["decode"] is False + + +@pytest.mark.asyncio +async def test_download_book_pdf_sorts_chapters_by_order( + monkeypatch: pytest.MonkeyPatch, +) -> None: + # 服务端返回顺序不可信:按 order 排序后合出的 PDF 才是阅读顺序 + _install_client(monkeypatch, chapter_ids=[(111, 2), (222, 1)]) + captured = _capture_write_pdf(monkeypatch) + + result, _ = await download_book_pdf( + "1114751", config=_config(), password="Ab3xK9Qm" + ) + + assert result.status == "ok" + assert [item["chapter_id"] for item in _FakeClient.instances[0].downloads] == [ + 222, + 111, + ] + assert [path.parent.name for path in captured["sources"]] == [ + "c001", + "c001", + "c002", + "c002", + ] + + +@pytest.mark.asyncio +async def test_download_book_pdf_falls_back_to_book_id_for_single_chapter( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_client(monkeypatch, chapter_ids=[]) + _capture_write_pdf(monkeypatch) + + result, _ = await download_book_pdf( + "1114751", config=_config(), password="Ab3xK9Qm" + ) + + assert result.status == "ok" + assert result.chapter_count == 1 + assert _FakeClient.instances[0].downloads[0]["chapter_id"] == 1114751 + + +@pytest.mark.asyncio +async def test_download_book_pdf_skips_empty_chapters_and_honours_max_chapters( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_client(monkeypatch, chapter_ids=[(999, 1), (111, 2), (222, 3)]) + _capture_write_pdf(monkeypatch) + + result, _ = await download_book_pdf( + "1114751", config=_config(jm_max_chapters=2), password="Ab3xK9Qm" + ) + + # 只取前两章,其中一章没有图片 + assert result.chapter_count == 3 + assert result.downloaded_chapters == 1 + assert result.page_count == 2 + assert [item["chapter_id"] for item in _FakeClient.instances[0].downloads] == [111] + + +@pytest.mark.asyncio +async def test_download_book_pdf_stops_when_images_exceed_limit( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_client(monkeypatch, chapter_ids=[(111, 1), (222, 2)], page_size=600_000) + captured = _capture_write_pdf(monkeypatch) + + result, _ = await download_book_pdf( + "1114751", config=_config(jm_max_file_size=1), password="Ab3xK9Qm" + ) + + assert result.status == "oversize" + assert result.pdf_path is None + assert captured == {} + assert len(_FakeClient.instances[0].downloads) == 1 + + +@pytest.mark.asyncio +async def test_download_book_pdf_reports_empty_result( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_client(monkeypatch, chapter_ids=[(999, 1)]) + captured = _capture_write_pdf(monkeypatch) + + result, _ = await download_book_pdf( + "1114751", config=_config(), password="Ab3xK9Qm" + ) + + assert result.status == "empty" + assert result.pdf_path is None + assert captured == {} + + +def test_build_settings_maps_config(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + jm_client, + "get_request_proxy", + lambda *_args, **_kwargs: "http://127.0.0.1:7890", + ) + + settings = jm_client.build_settings( + _config( + jm_use_proxy=True, + jm_session_dir="/tmp/jmcpy-home", + jm_request_timeout=45.0, + jm_image_timeout=120.0, + jm_download_concurrency=16, + ) + ) + + assert settings.proxy == "http://127.0.0.1:7890" + assert settings.timeout == 45.0 + assert settings.image_timeout == 120.0 + assert settings.concurrency == 16 + assert settings.home == Path("/tmp/jmcpy-home") + + +def test_build_settings_leaves_home_unset_when_session_dir_empty( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(jm_client, "get_request_proxy", lambda *_args, **_kwargs: None) + + settings = jm_client.build_settings(_config(jm_session_dir=" ")) + + assert settings.home is None + assert settings.proxy is None + + +def _page_png(path: Path, size: tuple[int, int], *, band: int = 0) -> Path: + image = Image.new("RGB", size, (255, 255, 255)) + for index in range(0, size[1], 7): + colour = (band, index % 256, 0) + image.paste(colour, (0, index, size[0], min(index + 3, size[1]))) + image.save(path, format="PNG") + return path + + +def test_write_pdf_keeps_one_page_scale_for_every_page(tmp_path: Path) -> None: + pages = [ + (_page_png(tmp_path / "a.png", (600, 900)), _fake_picture(Path("00001.webp"))), + ( + _page_png(tmp_path / "b.png", (900, 600), band=10), + _fake_picture(Path("00002.webp")), + ), + ( + _page_png(tmp_path / "c.png", (600, 900), band=20), + _fake_picture(Path("00003.webp")), + ), + ] + + build = jm_downloader._write_pdf( + pages, + tmp_path / "out.pdf", + dpi=150.0, + quality=95, + password="Ab3xK9Qm", + limit_bytes=None, + ) + + assert build.status == "ok" + assert build.path is not None + assert build.page_count == 3 + assert build.failed_pages == 0 + doc = fitz.open(build.path) + assert doc.needs_pass + assert doc.authenticate("Ab3xK9Qm") > 0 + assert doc.page_count == 3 + # 每页都按同一个 DPI 换算物理尺寸,不再出现第一页 150 DPI、其余 72 DPI 的错位 + expected = [(288, 432), (432, 288), (288, 432)] + for page, (width, height) in zip(doc, expected): + assert (round(page.rect.width), round(page.rect.height)) == (width, height) + info = doc.extract_image(page.get_images(full=True)[0][0]) + assert info["ext"] == "jpeg" + assert (info["width"], info["height"]) == ( + width * 150 // 72, + height * 150 // 72, + ) + + +def test_load_page_descrambles_server_blocks(tmp_path: Path) -> None: + original = Image.new("RGB", (30, 100)) + for index in range(10): + original.paste( + (index * 25 % 256, index * 17 % 256, index * 9 % 256), + (0, index * 10, 30, index * 10 + 10), + ) + # 服务端把竖直分块打乱;10 块时是整块反转,反转两次即还原 + scrambled = descramble(original, 10) + path = tmp_path / "00001.webp" + scrambled.save(path, format="PNG") + picture = Picture( + chapter_id=42, index=1, filename="00001.webp", url="", scramble_id=42 + ) + + restored = jm_downloader._load_page(path, picture) + + assert block_count(42, 42, "00001.webp") == 10 + assert restored.tobytes() == original.tobytes() + + +def test_write_pdf_can_skip_encryption(tmp_path: Path) -> None: + pages = [ + (_page_png(tmp_path / "a.png", (300, 400)), _fake_picture(Path("00001.webp"))) + ] + + build = jm_downloader._write_pdf( + pages, + tmp_path / "plain.pdf", + dpi=150.0, + quality=95, + password=None, + limit_bytes=None, + ) + + assert build.status == "ok" and build.path is not None + doc = fitz.open(build.path) + assert not doc.needs_pass, "uid 模式必须输出未加密 PDF,否则 PDF 解析工具打不开" + + +def test_write_pdf_stops_when_encoded_size_exceeds_limit(tmp_path: Path) -> None: + pages = [ + (_page_png(tmp_path / "a.png", (600, 900)), _fake_picture(Path("00001.webp"))) + ] + + build = jm_downloader._write_pdf( + pages, + tmp_path / "big.pdf", + dpi=150.0, + quality=95, + password=None, + limit_bytes=1, + ) + + assert build.status == "oversize" + assert build.path is None + assert not (tmp_path / "big.pdf").exists(), "提前中止时不应留下半成品 PDF" + + +def test_write_pdf_skips_undecodable_page(tmp_path: Path) -> None: + good = _page_png(tmp_path / "good.png", (300, 400)) + broken = tmp_path / "broken.webp" + broken.write_bytes(b"not an image") + pages = [ + (good, _fake_picture(Path("00001.webp"))), + (broken, _fake_picture(Path("00002.webp"))), + (good, _fake_picture(Path("00003.webp"))), + ] + + build = jm_downloader._write_pdf( + pages, + tmp_path / "partial.pdf", + dpi=150.0, + quality=95, + password=None, + limit_bytes=None, + ) + + assert build.status == "ok" + assert build.page_count == 2, "坏页被跳过,其余页照常出 PDF" + assert build.failed_pages == 1 + + +def test_write_pdf_reports_empty_when_every_page_fails(tmp_path: Path) -> None: + broken = tmp_path / "broken.webp" + broken.write_bytes(b"not an image") + + build = jm_downloader._write_pdf( + [(broken, _fake_picture(Path("00001.webp")))], + tmp_path / "empty.pdf", + dpi=150.0, + quality=95, + password=None, + limit_bytes=None, + ) + + assert build.status == "empty" + assert build.path is None + assert build.failed_pages == 1 + + +@pytest.mark.asyncio +async def test_download_book_pdf_cleans_up_when_download_raises( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + def _boom(*_args: Any, **_kwargs: Any) -> Any: + raise RuntimeError("boom") + + monkeypatch.setattr(jm_downloader, "_download_sync", _boom) + + with pytest.raises(RuntimeError, match="boom"): + await download_book_pdf("1114751", config=_config(), password="Ab3xK9Qm") + + leftovers = list((tmp_path / "jm").glob("*")) + assert leftovers == [], f"异常路径不得留下任务目录: {leftovers}" + + +@pytest.mark.asyncio +async def test_download_book_pdf_counts_failed_pages( + monkeypatch: pytest.MonkeyPatch, +) -> None: + # 每章 2 页,其中 1 页下载失败:失败数应汇总到 JmDownload.failed_pages + _install_client(monkeypatch, chapter_ids=[(111, 1)]) + _capture_write_pdf(monkeypatch) + + original_download = _FakeClient.download + + def _download_with_failure(self: Any, chapter: Any, **kwargs: Any) -> Any: + result = original_download(self, chapter, **kwargs) + result.failures = (object(),) + return result + + monkeypatch.setattr(_FakeClient, "download", _download_with_failure) + + result, _ = await download_book_pdf( + "1114751", config=_config(), password="Ab3xK9Qm" + ) + + assert result.status == "ok" + assert result.failed_pages == 1 + + +@pytest.mark.asyncio +async def test_download_book_pdf_cleans_up_after_cancellation( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + """取消后线程仍在写盘:必须等它结束再清理,不留残留原图/未加密 PDF。""" + import threading + + started = threading.Event() + release = threading.Event() + + def _slow(*_args: Any, **_kwargs: Any) -> Any: + started.set() + release.wait(10) + return JmDownload( + status="ok", + book=Book(book_id=1114751, title="测试本子"), + pdf_path=None, + page_count=0, + size_bytes=0, + chapter_count=1, + downloaded_chapters=0, + failed_pages=0, + ) + + monkeypatch.setattr(jm_downloader, "_download_sync", _slow) + + task = asyncio.create_task( + download_book_pdf("1114751", config=_config(), password=None) + ) + assert await asyncio.to_thread(started.wait, 10), "下载线程没能启动" + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + # 取消时目录还在(线程仍在写),放行线程后由回调清理 + assert list((tmp_path / "jm").glob("*")), "取消瞬间不应删除仍在写入的目录" + release.set() + + for _ in range(100): + if not list((tmp_path / "jm").glob("*")): + break + await asyncio.sleep(0.05) + assert list((tmp_path / "jm").glob("*")) == [], "取消后必须清掉任务目录" diff --git a/tests/test_jm_parser.py b/tests/test_jm_parser.py new file mode 100644 index 00000000..55f03e0a --- /dev/null +++ b/tests/test_jm_parser.py @@ -0,0 +1,143 @@ +from __future__ import annotations + +import json +from typing import Any + +from Undefined.jm.parser import ( + extract_from_json_message, + extract_jm_ids, + normalize_jm_id, +) + + +def test_normalize_jm_id_accepts_book_id_forms() -> None: + assert normalize_jm_id("1114751") == "1114751" + assert normalize_jm_id(1114751) == "1114751" + assert normalize_jm_id("JM1114751") == "1114751" + assert normalize_jm_id("jm1114751") == "1114751" + assert normalize_jm_id("https://18comic.vip/album/1114751") == "1114751" + assert normalize_jm_id("https://jmcomic.me/photo/1114751?x=1") == "1114751" + + +def test_normalize_jm_id_rejects_out_of_range_and_garbage() -> None: + assert normalize_jm_id("1234") is None + assert normalize_jm_id("123456789") is None + assert normalize_jm_id("") is None + assert normalize_jm_id("JM") is None + assert normalize_jm_id("https://example.com/album/1114751") is None + + +def test_extract_jm_ids_accepts_token_forms() -> None: + assert extract_jm_ids("jm1234567") == ["1234567"] + assert extract_jm_ids("看看 JM 1114751 吧") == ["1114751"] + assert extract_jm_ids("jm:1234567") == ["1234567"] + assert extract_jm_ids("jm#1234567") == ["1234567"] + assert extract_jm_ids("jm-123456") == ["123456"] + assert extract_jm_ids("JM1114751 和 jm 1114751") == ["1114751"] + + +def test_extract_jm_ids_respects_digit_bounds() -> None: + # 4 位过短、9 位过长,都不算车号;长数字串也不会被截断成 8 位 + assert extract_jm_ids("jm1234") == [] + assert extract_jm_ids("jm123456789") == [] + assert extract_jm_ids("jm00001234567") == [] + + +def test_extract_jm_ids_requires_prefix_boundary() -> None: + assert extract_jm_ids("xxjm1234567") == [] + assert extract_jm_ids("ajm1234567") == [] + + +def test_extract_jm_ids_does_not_match_bare_numbers() -> None: + assert extract_jm_ids("1114751 是群号") == [] + assert extract_jm_ids("2024-01-01 12:00:00") == [] + + +def test_extract_jm_ids_accepts_jm_links_only() -> None: + text = ( + "https://18comic.vip/album/1114751 " + "https://jmcomic.me/photo/2222222?x=1 " + "https://example.com/album/3333333" + ) + assert extract_jm_ids(text) == ["1114751", "2222222"] + + +def test_extract_jm_ids_strips_url_wrappers() -> None: + assert extract_jm_ids("看这个(https://18comic.vip/album/1114751)。") == [ + "1114751" + ] + + +def test_extract_from_json_message_reads_share_card() -> None: + payload = {"meta": {"detail_1": {"qqdocurl": "https://18comic.vip/album/1114751"}}} + segments: list[dict[str, Any]] = [ + {"type": "text", "data": {"text": "jm2222222"}}, + {"type": "json", "data": {"data": json.dumps(payload)}}, + ] + + assert extract_from_json_message(segments) == ["1114751"] + + +def test_extract_from_json_message_tolerates_broken_payload() -> None: + segments: list[dict[str, Any]] = [{"type": "json", "data": {"data": "{not-json"}}] + assert extract_from_json_message(segments) == [] + + +def test_extract_jm_ids_reads_id_query_links() -> None: + assert extract_jm_ids("https://18comic.vip/?id=350234") == ["350234"] + assert extract_jm_ids("https://jmcomic.me/album/350234?foo=1&id=350234") == [ + "350234" + ] + + +def test_extract_jm_ids_accepts_digit_bounds() -> None: + assert extract_jm_ids("jm10000") == ["10000"] # 5 位下界 + assert extract_jm_ids("jm99999999") == ["99999999"] # 8 位上界 + + +def test_extract_jm_ids_accepts_fullwidth_separators() -> None: + assert extract_jm_ids("jm:350234") == ["350234"] + assert extract_jm_ids("JM 350234") == ["350234"] + + +def test_extract_jm_ids_ignores_url_and_filename_context() -> None: + # 前缀前是 URL/路径/文件名分隔符时不算车号,避免把外部链接当车号下载 + assert extract_jm_ids("https://t.me/jm1234567") == [] + assert extract_jm_ids("video_jm1234567.mp4") == [] + assert extract_jm_ids("https://example.com/x/jm1234567") == [] + assert extract_jm_ids("看这个 jm1234567") == ["1234567"] + + +def test_extract_jm_ids_keeps_original_text_order() -> None: + # 车号在前、链接在后:先出现的必须排在前面(默认只取第一个,顺序错了会下错本) + assert extract_jm_ids("jm1111111 https://18comic.vip/album/2222222") == [ + "1111111", + "2222222", + ] + assert extract_jm_ids("https://18comic.vip/album/2222222 然后 jm1111111") == [ + "2222222", + "1111111", + ] + + +def test_extract_from_json_message_survives_malformed_segment() -> None: + segments: list[dict[str, Any]] = [ + {"type": "json", "data": "not-a-dict"}, + {"type": "json", "data": None}, + {"type": "json"}, + { + "type": "json", + "data": { + "data": '{"meta":{"detail_1":{"qqdocurl":"https://18comic.vip/album/3333333"}}}' + }, + }, + ] + + # 畸形段落不能中断整条消息的检测 + assert extract_from_json_message(segments) == ["3333333"] + + +def test_extract_from_json_message_skips_broken_json_body() -> None: + segments = [{"type": "json", "data": {"data": "{not json"}}] + + assert extract_from_json_message(segments) == [] diff --git a/tests/test_jm_pipeline.py b/tests/test_jm_pipeline.py new file mode 100644 index 00000000..e55e0b3b --- /dev/null +++ b/tests/test_jm_pipeline.py @@ -0,0 +1,114 @@ +from __future__ import annotations + +import json +from types import SimpleNamespace +from typing import Any +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from Undefined.handlers import MessageHandler +from Undefined.skills.pipelines import PipelineRegistry + + +def _handler(config: Any) -> Any: + handler: Any = MessageHandler.__new__(MessageHandler) + handler.config = config + handler.sender = SimpleNamespace() + handler.onebot = SimpleNamespace() + handler._extract_jm_ids = MagicMock(return_value=[]) + handler._handle_jm_extract = AsyncMock() + handler.pipeline_registry = PipelineRegistry() + handler.pipeline_registry.load_items() + return handler + + +def _config(**overrides: Any) -> Any: + base: dict[str, Any] = { + "bilibili_auto_extract_enabled": False, + "bilibili_opus_enabled": False, + "douyin_auto_extract_enabled": False, + "arxiv_auto_extract_enabled": False, + "github_auto_extract_enabled": False, + "jm_auto_extract_enabled": True, + "is_jm_auto_extract_allowed_group": lambda _gid: True, + "is_jm_auto_extract_allowed_private": lambda _uid: True, + "jm_auto_extract_max_items": 1, + } + base.update(overrides) + return SimpleNamespace(**base) + + +@pytest.mark.asyncio +async def test_jm_pipeline_dispatches_to_handler() -> None: + handler = _handler(_config()) + handler._extract_jm_ids = MagicMock(return_value=["1114751"]) + + handled = await handler._run_pipelines( + target_id=20001, + target_type="private", + text="jm1114751", + message_content=[], + ) + + assert handled is True + handler._extract_jm_ids.assert_called_once() + handler._handle_jm_extract.assert_awaited_once_with(20001, ["1114751"], "private") + + +@pytest.mark.asyncio +async def test_jm_pipeline_uses_real_extractor_for_text_and_share_card() -> None: + handler = _handler(_config()) + # 删掉替身,回落到 MessageHandler 上真实的 _extract_jm_ids + del handler._extract_jm_ids + card = {"meta": {"detail_1": {"qqdocurl": "https://18comic.vip/album/2222222"}}} + + handled = await handler._run_pipelines( + target_id=20001, + target_type="private", + text="jm1114751", + message_content=[ + {"type": "text", "data": {"text": "jm1114751"}}, + {"type": "json", "data": {"data": json.dumps(card)}}, + ], + ) + + assert handled is True + handler._handle_jm_extract.assert_awaited_once_with( + 20001, ["1114751", "2222222"], "private" + ) + + +@pytest.mark.asyncio +async def test_jm_pipeline_skips_when_disabled() -> None: + handler = _handler(_config(jm_auto_extract_enabled=False)) + + handled = await handler._run_pipelines( + target_id=20001, + target_type="private", + text="jm1114751", + message_content=[], + ) + + assert handled is False + handler._extract_jm_ids.assert_not_called() + handler._handle_jm_extract.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_jm_pipeline_respects_allowlist() -> None: + handler = _handler( + _config( + is_jm_auto_extract_allowed_group=lambda gid: gid == 123456, + ) + ) + + handled = await handler._run_pipelines( + target_id=654321, + target_type="group", + text="jm1114751", + message_content=[], + ) + + assert handled is False + handler._extract_jm_ids.assert_not_called() diff --git a/tests/test_jm_sender.py b/tests/test_jm_sender.py new file mode 100644 index 00000000..40b78ef4 --- /dev/null +++ b/tests/test_jm_sender.py @@ -0,0 +1,449 @@ +from __future__ import annotations + +from pathlib import Path +from types import SimpleNamespace +from typing import Any, Literal +from unittest.mock import AsyncMock + +import pytest +from jmcpy import Book + +import Undefined.jm.sender as jm_sender +from Undefined.jm.downloader import JmDownload +from Undefined.jm.sender import ( + generate_pdf_password, + send_jm_book, +) + + +def _book() -> Book: + return Book( + book_id=1114751, + title="测试本子", + authors=("作者A", "作者B"), + description="这是一段简介。", + tags=("标签1", "标签2"), + likes=12, + views=3456, + chapters=(), + ) + + +def _result( + tmp_path: Path, + *, + status: Literal["ok", "oversize", "empty"] = "ok", + size_bytes: int | None = None, +) -> JmDownload: + pdf_path = tmp_path / "JM1114751 测试本子.pdf" + if status == "ok": + pdf_path.write_bytes(b"%PDF-1.4") + if size_bytes is None: + size_bytes = 12_345_678 if status == "ok" else None + return JmDownload( + status=status, + book=_book(), + pdf_path=pdf_path if status == "ok" else None, + page_count=0 if status == "empty" else 120, + size_bytes=size_bytes, + chapter_count=3, + downloaded_chapters=3 if status == "ok" else 1, + failed_pages=0, + ) + + +def _sender() -> Any: + return SimpleNamespace( + send_group_message=AsyncMock(), + send_private_message=AsyncMock(), + send_group_forward_message=AsyncMock(), + send_private_forward_message=AsyncMock(), + send_group_file=AsyncMock(), + send_private_file=AsyncMock(), + ) + + +def _patch_download( + monkeypatch: pytest.MonkeyPatch, result: JmDownload, task_dir: Path +) -> dict[str, Any]: + captured: dict[str, Any] = {} + + async def _fake_download( + book_id: str, *, config: Any, password: str + ) -> tuple[JmDownload, Path]: + captured["book_id"] = book_id + captured["password"] = password + captured["config"] = config + return result, task_dir + + monkeypatch.setattr(jm_sender, "download_book_pdf", _fake_download) + cleanup = AsyncMock() + monkeypatch.setattr(jm_sender, "cleanup_download_path", cleanup) + captured["cleanup"] = cleanup + return captured + + +def test_generate_pdf_password_uses_safe_eight_char_alphabet() -> None: + password = generate_pdf_password() + + assert len(password) == 8 + assert all(char in jm_sender._PASSWORD_ALPHABET for char in password) + # 形近字符不入字母表 + assert not set(password) & set("0O1lI") + + +@pytest.mark.asyncio +async def test_send_jm_book_sends_forward_only( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + sender = _sender() + result = _result(tmp_path) + captured = _patch_download(monkeypatch, result, tmp_path / "task") + + status = await send_jm_book( + "1114751", + sender=sender, + target_type="group", + target_id=20001, + config=SimpleNamespace(), + ) + + assert status.startswith("JM1114751 已发送合并转发与 PDF 文件") + pdf_path = result.pdf_path + assert pdf_path is not None + args = sender.send_group_forward_message.await_args + nodes = args.args[1] + # 需求要求 PDF 节点留在转发里;可下载的入口是紧随其后的独立文件消息 + assert [node["data"]["name"] for node in nodes] == ["本子信息", "解密密码", "PDF"] + file_segment = nodes[2]["data"]["content"][0] + assert file_segment["type"] == "file" + assert file_segment["data"]["file"] == f"file://{pdf_path.resolve()}" + assert file_segment["data"]["name"] == pdf_path.name + info_text = nodes[0]["data"]["content"] + assert "JM1114751" in info_text + assert "测试本子" in info_text + # 末尾展示可复制的车号,不再放站点链接 + assert info_text.strip().endswith("JM1114751") + assert "18comic" not in info_text + assert f":{captured['password']}" in nodes[1]["data"]["content"] + assert len(captured["password"]) == 8 + # 文件只在转发里:不外发独立文件消息 + sender.send_group_file.assert_not_awaited() + # 历史摘要不含密码,只说明密码在转发节点里 + history_message = args.kwargs["history_message"] + assert captured["password"] not in history_message + assert "密码见转发节点" in history_message + captured["cleanup"].assert_awaited_once_with(tmp_path / "task") + + +@pytest.mark.asyncio +async def test_send_jm_book_falls_back_to_file_message_when_forward_fails( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + sender = _sender() + result = _result(tmp_path) + sender.send_group_forward_message.side_effect = RuntimeError("forward rejected") + _patch_download(monkeypatch, result, tmp_path / "task") + + await send_jm_book( + "1114751", + sender=sender, + target_type="group", + target_id=20001, + config=SimpleNamespace(), + ) + + # 转发本身发不出去时才退化为独立文件消息,否则用户拿不到 PDF + pdf_path = result.pdf_path + assert pdf_path is not None + sender.send_group_file.assert_awaited_once_with(20001, str(pdf_path), pdf_path.name) + + +@pytest.mark.asyncio +async def test_send_jm_book_falls_back_to_plain_messages( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + sender = _sender() + result = _result(tmp_path) + sender.send_group_forward_message.side_effect = RuntimeError("forward rejected") + captured = _patch_download(monkeypatch, result, tmp_path / "task") + + await send_jm_book( + "1114751", + sender=sender, + target_type="group", + target_id=20001, + config=SimpleNamespace(), + ) + + assert sender.send_group_message.await_count == 2 + password_call = sender.send_group_message.await_args_list[1] + assert captured["password"] in password_call.args[1] + # 普通消息兜底时历史里同样不留密码 + for call in sender.send_group_message.await_args_list: + history_message = call.kwargs["history_message"] + assert history_message is not None + assert captured["password"] not in history_message + sender.send_group_file.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_send_jm_book_does_not_retry_when_delivery_is_uncertain( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + """投递结果未确认时绝不能降级重发(仓库约定)。""" + + class _Uncertain(RuntimeError): + delivery_uncertain = True + + sender = _sender() + result = _result(tmp_path) + sender.send_group_forward_message.side_effect = _Uncertain("timeout") + _patch_download(monkeypatch, result, tmp_path / "task") + + with pytest.raises(_Uncertain): + await send_jm_book( + "1114751", + sender=sender, + target_type="group", + target_id=20001, + config=SimpleNamespace(), + ) + + sender.send_group_message.assert_not_awaited() + sender.send_group_file.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_send_jm_book_does_not_retry_when_file_transfer_failed( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + class _TransferFailed(RuntimeError): + file_transfer_error = True + + sender = _sender() + result = _result(tmp_path) + sender.send_group_forward_message.side_effect = _TransferFailed("prepare failed") + _patch_download(monkeypatch, result, tmp_path / "task") + + with pytest.raises(_TransferFailed): + await send_jm_book( + "1114751", + sender=sender, + target_type="group", + target_id=20001, + config=SimpleNamespace(), + ) + + sender.send_group_message.assert_not_awaited() + sender.send_group_file.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_send_jm_book_reports_when_pdf_upload_also_fails( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + sender = _sender() + result = _result(tmp_path) + sender.send_group_forward_message.side_effect = RuntimeError("forward rejected") + sender.send_group_file.side_effect = RuntimeError("upload failed") + _patch_download(monkeypatch, result, tmp_path / "task") + + status = await send_jm_book( + "1114751", + sender=sender, + target_type="group", + target_id=20001, + config=SimpleNamespace(), + ) + + assert status.startswith("JM1114751 合并转发与 PDF 文件均发送失败"), status + last = sender.send_group_message.await_args_list[-1] + assert "PDF 上传失败" in last.args[1] + # 提示消息不写历史(避免第三条重复摘要):auto_history 是第三个位置参数 + assert last.args[2] is False + + +@pytest.mark.asyncio +async def test_send_jm_book_reports_fallback_status( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + sender = _sender() + result = _result(tmp_path) + sender.send_group_forward_message.side_effect = RuntimeError("forward rejected") + _patch_download(monkeypatch, result, tmp_path / "task") + + status = await send_jm_book( + "1114751", + sender=sender, + target_type="group", + target_id=20001, + config=SimpleNamespace(), + ) + + assert "已改为普通消息发送信息与密码" in status + assert "补发独立 PDF 文件" in status + + +@pytest.mark.asyncio +async def test_send_jm_book_skips_pdf_when_empty( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + sender = _sender() + result = _result(tmp_path, status="empty") + _patch_download(monkeypatch, result, tmp_path / "task") + + status = await send_jm_book( + "1114751", + sender=sender, + target_type="group", + target_id=20001, + config=SimpleNamespace(), + ) + + assert status == "JM1114751 已发送信息(PDF 未发送:没有下载到任何页面)" + nodes = sender.send_group_forward_message.await_args.args[1] + assert "没有下载到任何页面" in nodes[0]["data"]["content"] + sender.send_group_file.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_send_jm_book_oversize_does_not_show_source_bytes_as_pdf_size( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + """预判超限时只有原图字节,信息节点不能把它写成 PDF 大小。""" + sender = _sender() + result = _result(tmp_path, status="oversize", size_bytes=88 * 1024 * 1024) + _patch_download(monkeypatch, result, tmp_path / "task") + + await send_jm_book( + "1114751", + sender=sender, + target_type="group", + target_id=20001, + config=SimpleNamespace(), + ) + + info = sender.send_group_forward_message.await_args.args[1][0]["data"]["content"] + assert "PDF:" not in info + + +@pytest.mark.asyncio +async def test_send_jm_book_skips_pdf_when_oversize( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + sender = _sender() + result = _result(tmp_path, status="oversize") + _patch_download(monkeypatch, result, tmp_path / "task") + + status = await send_jm_book( + "1114751", + sender=sender, + target_type="group", + target_id=20001, + config=SimpleNamespace(), + ) + + assert status == "JM1114751 已发送信息(PDF 未发送:文件超过体积上限)" + nodes = sender.send_group_forward_message.await_args.args[1] + assert [node["data"]["name"] for node in nodes] == ["本子信息", "状态"] + assert "超过体积上限" in nodes[0]["data"]["content"] + sender.send_group_file.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_send_jm_book_uses_private_forward( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + sender = _sender() + result = _result(tmp_path) + _patch_download(monkeypatch, result, tmp_path / "task") + + await send_jm_book( + "1114751", + sender=sender, + target_type="private", + target_id=30001, + config=SimpleNamespace(), + ) + + sender.send_private_forward_message.assert_awaited_once() + sender.send_group_forward_message.assert_not_awaited() + assert sender.send_private_forward_message.await_args.args[0] == 30001 + + +def test_format_jm_book_info_lists_metadata() -> None: + text = jm_sender.format_jm_book_info(_book()) + + assert "「JM1114751 测试本子」" in text + assert "作者: 作者A, 作者B" in text + assert "章节: 1 章" in text + assert "标签: 标签1、标签2" in text + assert "观看 3456 | 点赞 12" in text + assert "车号: JM1114751" in text + assert "18comic.vip/album/1114751" in text + + +@pytest.mark.asyncio +async def test_fetch_jm_book_attachment_registers_unencrypted_pdf( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + result = _result(tmp_path) + captured = _patch_download(monkeypatch, result, tmp_path / "task") + record = SimpleNamespace(uid="file_jm123") + registry = SimpleNamespace(register_local_file=AsyncMock(return_value=record)) + + text = await jm_sender.fetch_jm_book_attachment( + book_id="1114751", + attachment_registry=registry, + scope_key="group:20001", + config=SimpleNamespace(), + ) + + # uid 模式必须输出未加密 PDF,否则 PDF 解析工具打不开 + assert captured["password"] is None + register_args = registry.register_local_file.await_args + assert register_args.args[0] == "group:20001" + assert register_args.kwargs["kind"] == "file" + assert register_args.kwargs["source_kind"] == "jm_book" + assert 'PDF: ' in text + assert "该 PDF 未加密" in text + captured["cleanup"].assert_awaited_once() + + +@pytest.mark.asyncio +async def test_fetch_jm_book_attachment_requires_scope_and_registry( + tmp_path: Path, +) -> None: + assert "attachment_registry" in await jm_sender.fetch_jm_book_attachment( + book_id="1114751", + attachment_registry=None, + scope_key="group:20001", + config=SimpleNamespace(), + ) + assert "作用域" in await jm_sender.fetch_jm_book_attachment( + book_id="1114751", + attachment_registry=SimpleNamespace(register_local_file=AsyncMock()), + scope_key=" ", + config=SimpleNamespace(), + ) + + +@pytest.mark.asyncio +async def test_fetch_jm_book_attachment_reports_oversize( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + result = _result(tmp_path, status="oversize") + captured = _patch_download(monkeypatch, result, tmp_path / "task") + registry = SimpleNamespace(register_local_file=AsyncMock()) + + text = await jm_sender.fetch_jm_book_attachment( + book_id="1114751", + attachment_registry=registry, + scope_key="group:20001", + config=SimpleNamespace(), + ) + + assert "PDF 超过体积上限" in text + registry.register_local_file.assert_not_awaited() + captured["cleanup"].assert_awaited_once() diff --git a/tests/test_jm_tool.py b/tests/test_jm_tool.py new file mode 100644 index 00000000..a89ee514 --- /dev/null +++ b/tests/test_jm_tool.py @@ -0,0 +1,305 @@ +from __future__ import annotations + +from types import SimpleNamespace +from typing import Any +from unittest.mock import AsyncMock + +import pytest +from jmcpy import Book + +import Undefined.skills.shared as skills_shared +import Undefined.skills.tools.jm_book.handler as jm_tool +from Undefined.skills.tools.jm_book.handler import execute + + +def _book() -> Book: + return Book(book_id=350234, title="董卓 上+下", authors=("作者A",), tags=("标签",)) + + +def _context(**overrides: Any) -> dict[str, Any]: + base: dict[str, Any] = { + "runtime_config": SimpleNamespace(), + "sender": SimpleNamespace(), + "request_type": "group", + "group_id": 1074091596, + "attachment_registry": SimpleNamespace(register_local_file=AsyncMock()), + } + base.update(overrides) + return base + + +@pytest.mark.asyncio +async def test_jm_book_info_mode_forwards_normalized_book_id( + monkeypatch: pytest.MonkeyPatch, +) -> None: + info = AsyncMock(return_value="「JM350234 董卓 上+下」\n车号: JM350234") + monkeypatch.setattr(jm_tool, "jm_book_info_text", info) + + result = await execute({"book_id": "JM350234", "output_mode": "info"}, _context()) + + assert result.startswith("「JM350234 董卓 上+下」") + call = info.await_args + assert call is not None + assert call.args[0] == "350234" + + +@pytest.mark.asyncio +async def test_jm_book_info_mode_rejects_bad_book_id( + monkeypatch: pytest.MonkeyPatch, +) -> None: + fetch = AsyncMock() + monkeypatch.setattr(jm_tool, "jm_normalize_book_id", lambda _value: None) + fetch = AsyncMock() + monkeypatch.setattr(jm_tool, "jm_book_info_text", fetch) + + result = await execute({"book_id": "jm123", "output_mode": "info"}, _context()) + + assert "无法解析禁漫车号" in result + fetch.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_jm_book_uid_mode_registers_attachment( + monkeypatch: pytest.MonkeyPatch, +) -> None: + fetch_attachment = AsyncMock(return_value='PDF: ') + monkeypatch.setattr(jm_tool, "jm_book_attachment_text", fetch_attachment) + + result = await execute({"book_id": "350234", "output_mode": "uid"}, _context()) + + assert result == 'PDF: ' + call = fetch_attachment.await_args + assert call is not None + assert call.args[0] == "350234" + + +@pytest.mark.asyncio +async def test_jm_book_send_mode_uses_current_session( + monkeypatch: pytest.MonkeyPatch, +) -> None: + send = AsyncMock(return_value="JM350234 已发送合并转发与 PDF 文件") + monkeypatch.setattr(jm_tool, "jm_send_book", send) + monkeypatch.setattr(jm_tool, "jm_book_attachment_text", AsyncMock()) + + result = await execute({"book_id": "jm350234"}, _context()) + + assert result == "JM350234 已发送合并转发与 PDF 文件" + call = send.await_args + assert call is not None + assert call.kwargs["target_type"] == "group" + assert call.kwargs["target_id"] == 1074091596 + + +@pytest.mark.asyncio +async def test_jm_book_send_mode_accepts_explicit_target( + monkeypatch: pytest.MonkeyPatch, +) -> None: + send = AsyncMock(return_value="ok") + monkeypatch.setattr(jm_tool, "jm_send_book", send) + + await execute( + { + "book_id": "350234", + "target_type": "private", + "target_id": 20001, + }, + _context(request_type="group", group_id=999), + ) + + call = send.await_args + assert call is not None + assert call.kwargs["target_type"] == "private" + assert call.kwargs["target_id"] == 20001 + + +@pytest.mark.asyncio +async def test_jm_book_rejects_unknown_output_mode() -> None: + result = await execute({"book_id": "350234", "output_mode": "pdf"}, _context()) + + assert "output_mode 只能是" in result + + +@pytest.mark.asyncio +async def test_jm_book_reports_missing_runtime_config() -> None: + result = await execute({"book_id": "350234"}, _context(runtime_config=None)) + + assert result == "缺少必要的运行时组件(runtime_config)" + + +def test_jm_book_tool_definition_is_sendable_to_file_agent() -> None: + import json + from pathlib import Path + + tool_dir = ( + Path(__file__).resolve().parents[1] / "src/Undefined/skills/tools/jm_book" + ) + config = json.loads((tool_dir / "config.json").read_text(encoding="utf-8")) + callable_config = json.loads( + (tool_dir / "callable.json").read_text(encoding="utf-8") + ) + + assert config["function"]["name"] == "jm_book" + assert set(config["function"]["parameters"]["required"]) == {"book_id"} + assert callable_config["allowed_callers"] == ["file_analysis_agent"] + + +@pytest.mark.asyncio +async def test_jm_book_uid_mode_requires_runtime_config() -> None: + result = await execute( + {"book_id": "350234", "output_mode": "uid"}, _context(runtime_config=None) + ) + + assert result == "缺少必要的运行时组件(runtime_config)" + + +@pytest.mark.asyncio +async def test_jm_book_send_mode_requires_sender() -> None: + result = await execute({"book_id": "350234"}, _context(sender=None)) + + assert result == "缺少必要的运行时组件(sender)" + + +@pytest.mark.asyncio +async def test_jm_book_send_mode_needs_a_target_session() -> None: + result = await execute( + {"book_id": "350234"}, + _context(request_type=None, group_id=None, user_id=None), + ) + + assert "无法确定目标会话" in result + + +@pytest.mark.asyncio +async def test_jm_book_rejects_invalid_target_arguments() -> None: + bad_type = await execute( + {"book_id": "350234", "target_type": "channel", "target_id": 1}, _context() + ) + assert "target_type 只能是" in bad_type + + bad_id = await execute( + {"book_id": "350234", "target_type": "group", "target_id": "abc"}, _context() + ) + assert "target_id 必须是整数" in bad_id + + +@pytest.mark.asyncio +async def test_jm_book_uid_mode_falls_back_to_scope_from_context( + monkeypatch: pytest.MonkeyPatch, +) -> None: + fetch_attachment = AsyncMock(return_value="ok") + monkeypatch.setattr(jm_tool, "jm_book_attachment_text", fetch_attachment) + monkeypatch.setattr(jm_tool, "jm_scope_key", lambda _ctx: "group:1074091596") + + await execute({"book_id": "350234", "output_mode": "uid"}, _context()) + + call = fetch_attachment.await_args + assert call is not None + assert call.kwargs["scope_key"] == "group:1074091596" + + +@pytest.mark.asyncio +async def test_jm_book_reports_unexpected_errors( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr( + jm_tool, + "jm_book_info_text", + AsyncMock(side_effect=RuntimeError("源站 403")), + ) + + result = await execute({"book_id": "350234", "output_mode": "info"}, _context()) + + assert "禁漫本子处理失败" in result + assert "源站 403" in result + + +# --- skills/shared.py 的 JM 桥接:handler 不越界导入,由这里转发 --- + + +def test_shared_jm_normalize_book_id_accepts_ids_and_links() -> None: + assert skills_shared.jm_normalize_book_id("JM350234") == "350234" + assert skills_shared.jm_normalize_book_id("https://18comic.vip/album/350234") == ( + "350234" + ) + assert skills_shared.jm_normalize_book_id("hello") is None + + +def test_shared_jm_scope_key_prefers_context_value( + monkeypatch: pytest.MonkeyPatch, +) -> None: + assert skills_shared.jm_scope_key({"scope_key": " group:1 "}) == "group:1" + + import Undefined.attachments as attachments_module + + monkeypatch.setattr( + attachments_module, "scope_from_context", lambda _ctx: "private:2" + ) + assert skills_shared.jm_scope_key({"user_id": 2}) == "private:2" + + +@pytest.mark.asyncio +async def test_shared_jm_bridge_forwards_to_domain_functions( + monkeypatch: pytest.MonkeyPatch, +) -> None: + import Undefined.jm.downloader as jm_downloader_module + import Undefined.jm.sender as jm_sender_module + + monkeypatch.setattr( + jm_downloader_module, "fetch_book", AsyncMock(return_value=_book()) + ) + info = await skills_shared.jm_book_info_text("350234", config=SimpleNamespace()) + assert "JM350234" in info + + attachment = AsyncMock(return_value="uid-ok") + monkeypatch.setattr(jm_sender_module, "fetch_jm_book_attachment", attachment) + assert ( + await skills_shared.jm_book_attachment_text( + "350234", + registry="registry", + scope_key="group:1", + config="config", + ) + == "uid-ok" + ) + assert attachment.await_args is not None + assert attachment.await_args.kwargs["attachment_registry"] == "registry" + + send = AsyncMock(return_value="sent") + monkeypatch.setattr(jm_sender_module, "send_jm_book", send) + assert ( + await skills_shared.jm_send_book( + "350234", + sender="sender", + target_type="group", + target_id=20001, + config="config", + ) + == "sent" + ) + assert send.await_args is not None + assert send.await_args.kwargs["target_id"] == 20001 + + +def test_jm_book_handler_has_no_out_of_skills_imports() -> None: + """handler 必须只依赖 skills/ 内部(依赖经 shared 转发)。""" + import ast + from pathlib import Path + + handler_path = ( + Path(__file__).resolve().parents[1] + / "src/Undefined/skills/tools/jm_book/handler.py" + ) + tree = ast.parse(handler_path.read_text(encoding="utf-8")) + modules: set[str] = set() + for node in ast.walk(tree): + if isinstance(node, ast.Import): + modules.update(alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom) and not node.level: + modules.add(node.module or "") + out_of_skills = { + name + for name in modules + if name == "Undefined" + or (name.startswith("Undefined.") and not name.startswith("Undefined.skills")) + } + assert out_of_skills == set() diff --git a/tests/test_proxy_config.py b/tests/test_proxy_config.py index d59af7c7..86227f71 100644 --- a/tests/test_proxy_config.py +++ b/tests/test_proxy_config.py @@ -89,6 +89,7 @@ def test_proxy_switches_default_false_and_legacy_global_ignored( assert cfg.douyin_use_proxy is False assert cfg.arxiv_use_proxy is False assert cfg.github_use_proxy is False + assert cfg.jm_use_proxy is False assert cfg.api.tool_invoke_callback_use_proxy is False assert cfg.naga.use_proxy is False assert cfg.chat_model.use_proxy is False @@ -121,6 +122,7 @@ def test_proxy_switches_parse_per_feature_and_per_model( "douyin": {"use_proxy": True}, "arxiv": {"use_proxy": True}, "github": {"use_proxy": True}, + "jm": {"use_proxy": True}, "api": {"tool_invoke_callback_use_proxy": True}, "naga": {"use_proxy": True}, } @@ -224,6 +226,7 @@ def test_proxy_switches_parse_per_feature_and_per_model( assert cfg.douyin_use_proxy is True assert cfg.arxiv_use_proxy is True assert cfg.github_use_proxy is True + assert cfg.jm_use_proxy is True assert cfg.api.tool_invoke_callback_use_proxy is True assert cfg.naga.use_proxy is True assert cfg.chat_model.use_proxy is True @@ -252,6 +255,7 @@ def test_scoped_proxy_env_vars_enable_scoped_switches( _clear_proxy_env(monkeypatch) monkeypatch.setenv("SEARCH_USE_PROXY", "true") monkeypatch.setenv("GITHUB_USE_PROXY", "true") + monkeypatch.setenv("JM_USE_PROXY", "true") monkeypatch.setenv("CHAT_MODEL_USE_PROXY", "true") monkeypatch.setenv("IMAGE_GEN_MODEL_USE_PROXY", "true") @@ -259,6 +263,7 @@ def test_scoped_proxy_env_vars_enable_scoped_switches( assert cfg.search_use_proxy is True assert cfg.github_use_proxy is True + assert cfg.jm_use_proxy is True assert cfg.chat_model.use_proxy is True assert cfg.models_image_gen.use_proxy is True assert cfg.render_use_proxy is False diff --git a/uv.lock b/uv.lock index c7586776..f00a0657 100644 --- a/uv.lock +++ b/uv.lock @@ -944,6 +944,30 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/20/0c/7bb51e3acfafd16c48875bf3db03607674df16f5b6ef8d056586af7e2b8b/cssselect-1.4.0-py3-none-any.whl", hash = "sha256:c0ec5c0191c8ee39fcc8afc1540331d8b55b0183478c50e9c8a79d44dbceb1d8", size = 18540, upload-time = "2026-01-29T07:00:24.994Z" }, ] +[[package]] +name = "curl-cffi" +version = "0.16.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "cffi" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/82/e1/730125c43e3e331d98e17af3cb310ba526b3f1101b7635ca23d976ebfcf5/curl_cffi-0.16.3.tar.gz", hash = "sha256:d15d0c2a35f2d75bec430c28946c2a833f421c85773bdb0795182cc5c515665b", size = 239020, upload-time = "2026-09-02T11:58:23.266Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/79/7a/ec08ef0665c4ef4ea76b47042eb1c043e4afb374d8b9218e00272c9e73a2/curl_cffi-0.16.3-cp310-abi3-macosx_10_9_x86_64.whl", hash = "sha256:0f1f6878863fba393801e4d59b2f2766d1983b5c9d9dfa11d4becfd6a74cc937", size = 3025646, upload-time = "2026-09-02T11:57:39.326Z" }, + { url = "https://files.pythonhosted.org/packages/4c/86/e21b8ed384db26401a4438f20f01c7bcd9c3a6f8ceede458344e2d62775c/curl_cffi-0.16.3-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:f3b63da797912bc82911e34dfe449725514e4281527fb516931fc457087cfb44", size = 2784023, upload-time = "2026-09-02T11:57:40.986Z" }, + { url = "https://files.pythonhosted.org/packages/97/2d/25b106e64178829be1ce171b6cd45ba354ab7a2a5169001866b38d4c440f/curl_cffi-0.16.3-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d5a4103f2baa1fcf619ec3101b419827d367044ba106b206228137cc71a5a9c5", size = 12834219, upload-time = "2026-09-02T11:57:42.711Z" }, + { url = "https://files.pythonhosted.org/packages/e7/dd/db27a521777d0cf00f9a1554453ae730539dd134bca108d8df256a85c91e/curl_cffi-0.16.3-cp310-abi3-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:f2795f0ef2e8cc0e6d702e52367af6600f5bcf10e44d683e256254adc7e3589f", size = 12655304, upload-time = "2026-09-02T11:57:45.334Z" }, + { url = "https://files.pythonhosted.org/packages/72/01/2bbf141baa0fc3921d31a90de5465b7a94188845a8fe84dee86bf7bd90f1/curl_cffi-0.16.3-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a875a661e2f9a949be29454880bbb9553307a487c4c08819738298cf5c1622e2", size = 13484311, upload-time = "2026-09-02T11:57:47.58Z" }, + { url = "https://files.pythonhosted.org/packages/bb/d4/745ca299a2a223ee18574ec7cff75de620a92ce69b3cb09490db8fba614b/curl_cffi-0.16.3-cp310-abi3-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:0851e710608122a2716bdee35788bbd7e9d4a0fd42899b2bca9181277095af8e", size = 12840616, upload-time = "2026-09-02T11:57:50.016Z" }, + { url = "https://files.pythonhosted.org/packages/5e/bf/98d72d7a081cc155a71ab66bde6a18640d4ac5d4f6766f729a92cb4257c0/curl_cffi-0.16.3-cp310-abi3-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:1d7e553442cefec100dfd1fca4ae7035ab6c094457244bf60a38670b8ac8185d", size = 12612602, upload-time = "2026-09-02T11:57:52.584Z" }, + { url = "https://files.pythonhosted.org/packages/36/cf/2fdaff71fd6f39c5994495af8378e6e26bca8e447d94d2f75a76337908c8/curl_cffi-0.16.3-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:60621b3f561346046dd62be33abfb50c8b88a8007699d6b11d39ad4755312c4a", size = 12588697, upload-time = "2026-09-02T11:57:55.138Z" }, + { url = "https://files.pythonhosted.org/packages/74/55/68c399019bc24ea6ac783c98139a2555f88589631f3d127fe6b9073f019b/curl_cffi-0.16.3-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:20a7b1b473371cfaf2118958034977e457c6fa279fbd11543c9e0ab58be9eedd", size = 13253555, upload-time = "2026-09-02T11:57:57.524Z" }, + { url = "https://files.pythonhosted.org/packages/9b/72/1732a24ef4a2aeba994b80ec163debe8deda403c07e4abbc0443bca078b8/curl_cffi-0.16.3-cp310-abi3-win_amd64.whl", hash = "sha256:fe87b66e324ed7318166698e02169f3208dbda32b872a27d2bc61a9c19b335eb", size = 1978602, upload-time = "2026-09-02T11:58:00.033Z" }, + { url = "https://files.pythonhosted.org/packages/45/bb/67bec3132aeabac99dfe2f299a9b43dcb5de23ad96219ee98516d177fc9c/curl_cffi-0.16.3-cp310-abi3-win_arm64.whl", hash = "sha256:5a2ba880019f9e5a9e8f38ae22de6e4ea4c8d34a51ae4f1a2fce962c7b632006", size = 1713140, upload-time = "2026-09-02T11:58:01.558Z" }, + { url = "https://files.pythonhosted.org/packages/49/e3/b88f9b1a60a1e29b42e9371c1b3f4fdd83bf8177fcf863df67438da12693/curl_cffi-0.16.3-cp313-cp313-android_24_arm64_v8a.whl", hash = "sha256:01c31369b1c8063c7e459152c508c90de7a4218aa66ee3a1f575ae37ce44bc5a", size = 8607348, upload-time = "2026-09-02T11:58:03.095Z" }, +] + [[package]] name = "cycler" version = "0.12.1" @@ -1720,6 +1744,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/67/8a/a342b2f0251f3dac4ca17618265d93bf244a2a4d089126e81e4c1056ac50/jiter-0.13.0-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7bb00b6d26db67a05fe3e12c76edc75f32077fb51deed13822dc648fa373bc19", size = 343768, upload-time = "2026-02-02T12:37:55.055Z" }, ] +[[package]] +name = "jmcpy" +version = "0.1.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cryptography" }, + { name = "curl-cffi" }, + { name = "pillow" }, + { name = "platformdirs" }, + { name = "pypdf" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/fb/53/8d8e8ef0499f5c5b33981353720ef3a56f17ee38faaddda710f35c82e9d3/jmcpy-0.1.3.tar.gz", hash = "sha256:b3cfcb7ec6accbc544fc088fac29c9c9a23b7146d6d8b207214bb0fa4f088963", size = 129421, upload-time = "2026-09-30T15:18:32.223Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8c/bb/dd8c6e805fba67e363f86e59131aa8dd3eddb97e0eb767b164da1cb419b3/jmcpy-0.1.3-py3-none-any.whl", hash = "sha256:88e37ffd67a69a54ed7010b1352ed59d0ca224a119fee8a371820f655aabdaca", size = 84262, upload-time = "2026-09-30T15:18:31.102Z" }, +] + [[package]] name = "joblib" version = "1.5.3" @@ -3565,6 +3605,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl", hash = "sha256:850ba148bd908d7e2411587e247a1e4f0327839c40e2e5e6d05a007ecc69911d", size = 122781, upload-time = "2026-01-21T03:57:55.912Z" }, ] +[[package]] +name = "pypdf" +version = "6.19.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1f/ac/63d71aaedb59acbcdef491e6ca6469165e3771c9c74358204818fd9bc5a6/pypdf-6.19.0.tar.gz", hash = "sha256:bbc43aca292369ccc6cbc8a921991ecf2538a3587ab5a116eff06c321d647155", size = 7033266, upload-time = "2026-09-16T09:32:05.946Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3c/2c/c43c03eaf630435f023f1dc61ec4a4a78951ad5530a62c71cc89bde307b7/pypdf-6.19.0-py3-none-any.whl", hash = "sha256:7e5d6e730e7dae87d560a2cee218b852f6498c8be61966f3cd02ead971e48d14", size = 395480, upload-time = "2026-09-16T09:32:04.087Z" }, +] + [[package]] name = "pyperclip" version = "1.11.0" @@ -4680,7 +4729,7 @@ wheels = [ [[package]] name = "undefined-bot" -version = "3.17.2" +version = "3.18.0" source = { editable = "." } dependencies = [ { name = "aiofiles" }, @@ -4692,6 +4741,7 @@ dependencies = [ { name = "crawl4ai" }, { name = "fastmcp" }, { name = "httpx" }, + { name = "jmcpy" }, { name = "langchain-community" }, { name = "linkify-it-py" }, { name = "lunar-python" }, @@ -4748,6 +4798,7 @@ requires-dist = [ { name = "crawl4ai", specifier = ">=0.8.6" }, { name = "fastmcp", specifier = ">=3.1.1" }, { name = "httpx", specifier = ">=0.27.0" }, + { name = "jmcpy", specifier = ">=0.1.3" }, { name = "langchain-community", specifier = ">=0.3.0" }, { name = "linkify-it-py", specifier = ">=2.0.3" }, { name = "lunar-python", specifier = ">=1.4.8" },