From 83f6e97f38cff8988822dd4b81e08ce2cffdbd32 Mon Sep 17 00:00:00 2001 From: Comui520 Date: Mon, 26 Oct 2026 10:01:06 +0800 Subject: [PATCH 01/17] docs: design variant pricing and media evidence upgrade --- .../2026-10-26-variant-price-media-design.md | 218 ++++++++++++++++++ 1 file changed, 218 insertions(+) create mode 100644 docs/superpowers/specs/2026-10-26-variant-price-media-design.md diff --git a/docs/superpowers/specs/2026-10-26-variant-price-media-design.md b/docs/superpowers/specs/2026-10-26-variant-price-media-design.md new file mode 100644 index 0000000..0fd2fdb --- /dev/null +++ b/docs/superpowers/specs/2026-10-26-variant-price-media-design.md @@ -0,0 +1,218 @@ +# 规格、价格可信度与多媒体分析升级设计 + +日期:2026-10-26 + +基线:`f728ff8` / v0.1.1 +状态:设计待用户批准;尚未开始实现。 + +## 1. 目标与已确认范围 + +用户确认采用方案 A:分阶段实施,完成规格与价格可信度、多媒体分析,再进行全项目检查、性能优化及缺陷修复。 +重点是同一商品链接中的型号/容量组合,以及“标价不代表实际售价、需要询价”的商品。 +继续沿用 Python、FastAPI、SQLAlchemy、SQLite、Playwright、Jinja2/HTMX,保留 uv/Conda 与 Docker 运行方式。 + +核心原则:先确认选项、价格及证据,再判断预算与性价比。未知不是零元,提取失败不是没有瑕疵。 + +非目标:自动询价、自动聊天、下单或支付;绕过登录/验证码/访问控制;穷举商家全部 SKU; +音轨理解和完整动态故障诊断;宣称兼容未经验证的所有闲鱼页面;重写前端或引入分布式服务。 + +## 2. 方案比较与选择 + +- A(用户选择):分阶段建立规格/价格基础,再接入媒体证据,最后测量并优化。每阶段可测试、审查、回退。 +- B(不采用):同时替换全部抓取、数据和分析逻辑,末尾统一测试。集成风险大,难以定位回归。 +- 技术实现采用结构化页面证据优先、确定性规则裁决,LLM 只辅助抽取文本声明与总结,不生成实际售价或虚构 SKU。 + 全靠 LLM 推断型号和价格不能提供可核对证据;一开始绑定某一家原生视频 API 则限制现有模型兼容性。 + +## 3. 当前实现与必要改变 + +- `ListingData/ListingDetail/Listing` 只有图片列表,没有视频媒体契约;品相请求最多发送四张图片。 +- `variants` 仅由价格区间生成“最低价/最高价”,不是真实可购买组合。 +- 搜索标价在详情提取前就被预算过滤;它还用于价格走势、性价比、重推,可能错误比较不同规格。 +- `_fetch_detail` 通常有描述就不刷新;现有商品重评未必更新规格与媒体。 +- 网络、模型调用处于较长数据库会话内;需测量锁占用、浏览器启动、重复模型请求与关系查询次数。 +- “重新上架”判断发生在 status 被设为 active 之后,是需以回归测试验证的疑点;有限搜索结果未见不等于已下架。 +- 当前应用无用户认证。升级不扩大默认网络暴露范围,不借此次改造冒充解决多用户安全问题。 + +## 4. 阶段一:规格与价格可信度 + +### 4.1 数据与职责 + +沿用按任务隔离的 Listing 父记录;保留 `Listing.price` 与旧 PriceSnapshot 为“页面标价/旧标价走势”, +不再把它们直接当作目标规格的成交价。新增规范化的选项和观察记录,不继续向无类型 variants 字典塞业务状态。 + +- WatchTask:新增独立的目标规格文本 `target_variant`(如“iPhone 15 / 256GB / 非扩容”)和 `notify_pending_price`。 +- ListingOffer:`listing_id`、稳定 `offer_key`、原站 SKU ID(如存在)、规范化属性、选项名称、可用状态、 + 当前价格状态/证据、匹配状态、评估结果。唯一约束 `(listing_id, offer_key)`。 +- OfferObservation:关联 offer、原始来源、价格状态、确定价或区间、采集时间、页面位置/原文片段、内容指纹。 + 新价格以人民币分整数保存;原始金额用 Decimal 校验后转换,拒绝负数、非有限值和不支持的币种。 +- ManualQuote:关联 offer、金额、用户说明、确认/失效时间、撤销状态;独立于抓取值,绝不被爬虫覆盖。 +- offer 维度的评估与通知记录:记录使用的 observation/quote、媒体证据和任务条件指纹;通知增加 offer 与事件类型引用。 + +原站 SKU ID 优先作为身份;无 ID 时按完整规范化属性组合生成稳定键,不依赖显示顺序或价格。 +多个不同选项出现相同属性键、或无法确认组合完整性时标记歧义,不把它们合并。 +普通单件无规格商品使用默认 offer;旧 variants 区间仍展示为历史信息,不能迁移为真实 SKU。 +父 Listing 的摘要兼容旧 API,但新增明确的有效价格/规格/状态字段,前端不得再将旧 price 标成确认价。 + +### 4.2 价格状态与裁决 + +| 状态 | 含义 | 允许预算判断与性价比排名 | +| --- | --- | --- | +| exact | 当前具体可用选项有明确价格,且无适用的冲突或非售价声明 | 是 | +| range_or_from | 仅起售价或区间,目标选项未确定 | 否 | +| inquiry | 明示需咨询、标价非售价或占位价 | 否 | +| conflict | 同一选项同时存在互斥的价格声明 | 否 | +| unknown | 抓取失败、信息缺失或映射不明 | 否 | + +另有“人工确认报价”来源,不将它伪装为原站 exact。exact 仅表示页面证据明确,不保证卖家最终履约价。 +stock 状态使用 available/unavailable/unknown;SKU 不可用时不参与正常推荐,未知库存明确展示。 +“起”本身不能推断高价规格售价;“优惠私聊”与“所有标价非实际售价”区分作用范围; +声明作用范围不清时保守待确认,不能让一个疑似数字覆盖其他明确证据。 +来自图片或 LLM 的识别仅为待核对声明:必须关联证据,不独立提升为 exact。 + +### 4.3 采集与目标匹配 + +首先从商品所属 DOM/结构化页面状态读取真实 SKU 组合;只读必要的选项交互,不点击购买/咨询/支付。 +支持的结构化状态格式需要真实样本验证,不预先假定平台存在某个 API。 +最多处理每链接 20 个候选组合,只针对目标匹配选项做后续昂贵分析;限制截断时显示“规格获取不完整”。 +规则负责型号、容量等可确定约束;LLM 可以提出映射候选,但不能把不确定候选标为已匹配。 +没有填写 target_variant 时,普通单件照常处理;多规格商品显示已核对的选项,多个匹配项分别评估,不能默认最低价代表整件。 +明确型号/容量不匹配不推送;目标匹配不确定则留待确认,不进正常排名。 + +搜索阶段只保留非价格的安全过滤与抓取数量上限。预算过滤在 offer 解析后进行,避免起售价提前漏掉目标规格。 +默认每任务每轮最多 30 个详情候选;超过预算进入后续轮次的有界待抓取队列(持久化或可重建),记录未处理数量,不能静默当作不匹配。 +明确要求规格核验而 fetch_detail 关闭时,保留待确认状态并提示该设置不能获得确价,不偷偷启用详情抓取。 +exact 的单价落在任务预算内才进入正常候选;待确认价不以标价判断超预算,不进入性价比模型/本批最优。 +已知品牌或商家身份不作为直接排除条件;包装、配件、整机作为不同选项匹配,避免交叉借用证据。 + +### 4.4 人工报价与通知(下列默认值为本设计提议) + +- 待询价商品默认站内保留,独立的待确认提醒开关默认关闭,避免升级后突然增加外部通知;用户可按任务开启。 +- 手动报价必须绑定明确规格;确认页面同时展示最新抓取的价格证据,默认有效期 7 天,可修改或撤销。 +- 人工报价有效且规格匹配时可用于预算和排名,所有结果注明“人工报价”和确认时间。 + 新抓取的同 SKU 价格/非售价声明有实质变化时,旧报价转为需复核;选项不可用或身份歧义也会使报价不可用于推荐。 +- 当人工报价与页面 exact 同时有效,优先采用用户对照最新页面证据确认的人工报价(人工操作页明确提示该优先级); + 该报价必须记录确认时的页面证据指纹。若页面证据实质变化,先暂停人工报价而不是偷偷切回页面价继续推送。 + 用户复核或撤销后才重新确定有效来源。页面价与人工报价的历史分开比较;两种来源切换仅称“价格来源更新”, + 不标为降价,同来源内的新价格与旧价格可比时才计算降幅。人工报价到期后可采用仍新鲜的页面 exact,注明来源切换。 +- 不同规格分别做同价历史比较;从 A 切换到 B 不能叫“降价”。历史证据保留,不覆盖。 +- 预算、规格或关键需求修改使评估失效;重评不沿用旧目标报价,不复用旧任务条件的评分。 +- 通知以 offer、事件类型、证据/评估版本、渠道记录状态;待确认转明确可触发一次正常提醒。 + 相同待确认状态与证据不重复推送;渠道失败单独记录,不将其他成功渠道重新发送。 + 不声称网络超时情况下能严格 exactly-once;无幂等接口的渠道需提示可能重复,避免无限重试。 +- 未知价格只展示需求和媒体分析的独立结果,不产生可与正常候选混排的综合性价比名次。 + +### 4.5 刷新与页面行为 + +任务表单、API、设置、列表、详情和通知统一使用价格解析服务输出;不在模板另写价格裁决。 +详情提供规格明细、页面标价与有效价格的区分、原文依据、人工报价的新增/撤销操作。 +普通列表仅将确价候选纳入默认性价比排序;待确认独立筛选,预算/价格排序不拿占位价填空。 +SKU 与有效价格最多缓存一个调度间隔,手动重新分析和刷新强制更新;人工报价独立保持。 +下次抓取失败不抹除旧证据,但将其标为旧观察,不把该轮故障视为库存变化或降价。 +价格、规格映射与可用状态的核验有效期为一个任务调度间隔,自最后成功核验的时间计算; +短暂失败不提前缩短这个有效期,但该失败本身不产生新的价格事件或提醒。 +任一必要证据到期后,该 offer 转为待复核并暂停正常排名与正常外部提醒,不删历史记录、不重新认定售罄。 +人工报价的七天有效期只描述报价本身,不能代替新鲜的规格/可用状态证据;暂停推荐不会撤销或缩短报价期限, +后续成功核验无实质变化时可在原有效期内恢复。恢复不凭状态变化重复发送已成功的事件。 + +## 5. 阶段二:视频及图文混合证据 + +### 5.1 媒体契约 + +MediaAsset 记录 listing/可选 offer、image/video/frame 类型、原始来源/页面位置、封面、时间位置、内容 hash、 +采集时间、父视频引用、处理状态与失败原因。没有确切 SKU 关联时标记 listing_shared,不假设属于任意一台机器。 +只采集商品详情主媒体区域,排除推荐流、头像、广告;不能只按整个网页的 video/img 标签抓取。 +视频 URL 可能过期、blob 或为分段流:首版支持可授权获取的直接 HTTP(S) 媒体;blob/DRM/未适配流清楚标注不支持, +不绕过权限。必要时通过正常页面刷新重新获取临时 URL。 + +### 5.2 抽帧与资源边界(设计默认值,可配置) + +使用 FFmpeg/ffprobe 子进程处理本地受限文件,再把帧转为现有视觉接口支持的图片输入;不要求特定厂商的原生视频 API。 +Docker 显式安装 FFmpeg;uv/Conda 检测外部工具,缺失时禁用视频处理并提示安装,不让应用无法启动。 +视频分析为明确开关,默认关闭且显示“未分析视频”;启用后与图片联合分析。每商品最多 1 个视频、8 张原图、6 帧, +每次模型请求总视觉输入最多 8 张(有视频时优先最多 4 张原图 + 最多 4 帧,其余作为详情证据保存)。 +视频上限 100 MiB、180 秒;流式下载最多 60 秒、解码最多 30 秒,超预算中止并删除半成品;媒体处理并发 1。 +有限采样覆盖起、中、末,过滤模糊/重复帧;不宣称未采样片段已检查,也不对音频或动态故障下确定结论。 +下载器验证 URL、DNS 解析地址和每次重定向;禁止回环、内网、链路本地、云 metadata 等目标。 +媒体下载与可配置内网模型地址是不同信任边界;不用媒体请求携带任意域 Cookie。子进程不经 shell, +FFmpeg 仅读本地输入、限制网络协议和执行时间;文件输出在受控缓存根目录内。 + +### 5.3 缓存与证据分析 + +缓存位于 data/media-cache,默认上限 1 GiB、保留 7 天;原视频成功抽帧后删除,失败半成品立即清理。 +按内容 hash/处理参数去重,媒体 URL 改变不直接认定内容改变;无法取得内容时不可凭 URL 缓存成成功。 +已被分析引用的帧若因容量被清理,保留摘要、hash、时间位置并显示“证据文件已清理”;可重新获取,不显示坏链接。 +API 不接受任意本地路径;媒体通过受控 ID 映射提供,不泄露签名 URL 的令牌、Cookie 和宿主文件路径。 +模型请求使用可验证的 data URL 或受支持上传,不把容器本地文件路径传给远程模型;现有 chat/responses 两接口都要覆盖测试。 +分析输出包括证据 ID、来源、视频时间位置、观察到的瑕疵、卖家声明、信息冲突和未验证项。 +结构化校验只允许引用本次输入的证据 ID;拒绝模型虚构的帧、价格、容量或来源。 +共享媒体不能自动生成某一 SKU 的确定品相结论;分析结果保留样品/不保证对应实物提示。 +抓帧或模型失败时保留可用图片分析;无有效视觉证据时不生成“品相达标”结论,正常提醒明确标识未核验。 +最低品相分是明确门槛:任务设置大于 0 时,必须有适用于该 offer 的有效视觉评分达到门槛, +未分析、分析失败、只有不确定关联的共享样品媒体均进入待核验,不参与正常排名或正常外部提醒。 +最低分为 0 时,确价且规格匹配的商品可在明确标注“品相未核验”的情况下按价格等已知信息排名/提醒, +但不能生成品相通过结论。媒体与文字有实质冲突时保留冲突状态,不能用缺失分数的默认半分掩盖。 +关闭视觉而任务最低分大于 0 时提示配置冲突,不能静默放行或修改用户门槛;这是相对于旧放行逻辑的有意行为改变。 +评估缓存包含证据内容、任务条件、模型/提示词版本;变更时失效。支持手动强制重评,避免陈旧结论。 + +## 6. 阶段三:全项目检查、性能与缺陷修复 + +先建立可重复的基线,再按实测瓶颈修复;不承诺“没有任何 bug”或给出未测量的提速比例。 + +- 正确性:重新上架、有效价格变化、规格售罄、部分解析失败、重评、暂停/删除任务、通知失败恢复。 +- 生命周期:有限搜索结果未见只能称“本轮未见/疑似不可用”;只有详情确证下架才标为确定下架。 +- 数据库:外部请求不占写事务;按商品/步骤短事务,迁移失败不留下半套新结构;验证并发读写与 N+1。 +- 浏览器:评估单轮复用浏览器/隔离上下文,取消/异常可靠关闭,不跨任务泄露 Cookie 或页面状态;保留串行任务限速。 +- 模型:同证据不重复分析;有限超时、退避、重试与单任务调用预算;预算耗尽保留待处理状态,不误报否定。 +- 可观测性:每轮记录候选数、详情耗时、媒体耗时、LLM 次数、缓存命中、SQL 次数和失败状态;日志屏蔽秘密。 +- 安全回归:表单先校验后提交、敏感配置与运行时配置边界、媒体下载边界、noVNC 默认关闭、路径穿越。 + +用固定的合成/脱敏商品集和 mock 网络/模型做可重复基准,比较冷启动与第二轮热缓存: +热缓存不重复启动每商品浏览器、不重复相同媒体的模型分析;记录中位数与 P95、查询数、调用数和资源峰值。 +任何优化需证明结果一致、无重复通知、无明显回归;绝不为性能提高抓取频率或取消安全等待。 + +## 7. 迁移、兼容与回退 + +开发和 Docker 验证使用独立数据库、缓存、容器名和未占用的回环端口,禁止挂载生产 data/;默认禁用真实通知与定时任务。 +生产升级前停写并制作可恢复备份;迁移按版本、幂等、事务执行,先验证临时旧库副本,不能运行未备份的破坏性迁移。 +新增表与字段保留原数据;旧评分标记 legacy_unverified,不转换为新规格的确定结论;原有通知历史不触发重发。 +旧任务 target_variant 为空、待询价外部通知关闭;旧单件下次成功详情抓取后生成默认 offer,未知 SKU 不凭区间构造。 +任务删除、商品删除要覆盖新关联记录,防止孤儿数据;同一外部 ID 跨任务仍隔离。 +关闭新视频能力不会影响原有图片功能;已升级数据库不承诺直接降级可写,完整回退采用备份数据库与旧镜像成对恢复。 +保留本机已忽略的 Windows 快捷脚本、私人文档和 .env;不提交凭据、真实数据库、视频缓存和个人资料。 +当前仅提交设计到本地专用分支;实施前审查远程变化,实施完成不擅自更新 main、release、Tag 或 GHCR。 + +## 8. 验收与交付 + +每阶段:先失败测试 → 最小实现 → 单测/集成测试 → 独立代码审查 → 修复 → 再验证;再进入下一阶段。 + +必要验收用例: +1. 配件 99、128GB 1200、256GB 1500:目标 256GB 使用 1500,预算筛选与历史比较正确,绝不拿 99 评分。 +2. 占位价 1 元、明确非售价或需询价:待确认、无性价比/本批最优,开关与去重符合约定。 +3. 冲突价、选项售罄、SKU 身份变更、详情失败、组合超限不被静默视为已核验;人工报价不会被自动覆盖。 +4. 同一规格降价才产生降价事件;不同规格切换、人工报价过期、条件变化均不冒充降价。 +5. 图片、纯视频、混合媒体、样品视频、过期链接、超大小、超时、缺 FFmpeg、SSRF/路径穿越均有回归用例。 +6. 旧库迁移两遍结果一致,任务/历史/快照/通知不丢失;备份恢复演练通过。 +7. Ruff、锁文件检查、全套 pytest、覆盖率不低于现有 80% 门槛;关键新裁决分支需直接覆盖,不能只靠总覆盖率。 +8. Docker 构建、SDK 导入、FFmpeg 检测、独立数据库启动、healthz、表单/详情/提醒预览与 noVNC 模式检查通过。 +9. 提供性能基线与改后对比,未运行项和受外部环境限制的项如实列出,不以合成样本代替真实平台验证。 + +真实平台适配需要合法可访问的多 SKU、询价、纯视频/混合媒体代表样本;采集夹具必须脱敏。 +若缺真实样本可完成领域模型、规则与夹具测试,但不能将真实 SKU 自动提取/视频覆盖宣称完成; +相关能力保持实验性/待验证并明确列为交付限制,不自动启用或发布稳定版。 + +## 9. 待批准的提议与实施者可决定事项 + +待用户随设计一并批准:待询价站内保留但外部提醒默认关闭;人工报价默认 7 天;视频抽帧可选且默认关闭; +分阶段完成三个阶段,真实样本验证不足时明确标为限制而非伪造兼容结论。 +以上默认值均为助手提议,不表示用户此前逐项确认。 +同样待批准的裁决:有效人工报价优先但来源切换不算降价;新鲜度过期暂停推荐、保留报价; +最低品相分大于 0 时未核验不得进入正常推荐。实施计划必须显式覆盖这些变化与回归测试。 + +实施者可决定:内部模块命名、索引具体形式、帧去重算法及可维护的采样细节、测试工具与基准夹具组织; +不得改变不自动联系卖家、不用未知价排名、不触碰真实数据、不默默扩大资源预算等边界。 + +## 10. 独立设计审查记录 + +已由独立工程师审查初稿,发现三个会影响实施的重要歧义:人工报价与页面价的优先级、刷新失败/过期后的推荐资格、 +最低品相分对未核验商品是否构成硬门槛。已在 §4.4、§4.5、§5.3 补充确定规则,作为设计提议交由用户批准。 +审查未涉及实现正确性;具体代码、真实平台样本与性能结果仍须在后续阶段另行验证。 From 53d933ad151ba6b6bf6dd9e460e65897818443db Mon Sep 17 00:00:00 2001 From: Comui520 Date: Mon, 26 Oct 2026 10:20:49 +0800 Subject: [PATCH 02/17] docs: extend upgrade design with bounded pipeline and UX --- .../2026-10-26-variant-price-media-design.md | 217 +++++++++++++++++- 1 file changed, 210 insertions(+), 7 deletions(-) diff --git a/docs/superpowers/specs/2026-10-26-variant-price-media-design.md b/docs/superpowers/specs/2026-10-26-variant-price-media-design.md index 0fd2fdb..c0e2a42 100644 --- a/docs/superpowers/specs/2026-10-26-variant-price-media-design.md +++ b/docs/superpowers/specs/2026-10-26-variant-price-media-design.md @@ -1,8 +1,10 @@ -# 规格、价格可信度与多媒体分析升级设计 +# 规格价格、多媒体证据、受控流水线与交互升级设计 日期:2026-10-26 基线:`f728ff8` / v0.1.1 +版本:修订版 2(追加资源配置、流水线与全站交互要求) + 状态:设计待用户批准;尚未开始实现。 ## 1. 目标与已确认范围 @@ -10,6 +12,8 @@ 用户确认采用方案 A:分阶段实施,完成规格与价格可信度、多媒体分析,再进行全项目检查、性能优化及缺陷修复。 重点是同一商品链接中的型号/容量组合,以及“标价不代表实际售价、需要询价”的商品。 继续沿用 Python、FastAPI、SQLAlchemy、SQLite、Playwright、Jinja2/HTMX,保留 uv/Conda 与 Docker 运行方式。 +用户进一步确认:媒体输入数量由用户设置且展示代价;采用有界分阶段流水线提速,各硬件敏感并发可配置; +提供保守推荐和设备检测建议;全站交互优化但沿用现有简洁风格,不使用前端设计类技能;所有变化保持跨平台。 核心原则:先确认选项、价格及证据,再判断预算与性价比。未知不是零元,提取失败不是没有瑕疵。 @@ -22,6 +26,9 @@ - B(不采用):同时替换全部抓取、数据和分析逻辑,末尾统一测试。集成风险大,难以定位回归。 - 技术实现采用结构化页面证据优先、确定性规则裁决,LLM 只辅助抽取文本声明与总结,不生成实际售价或虚构 SKU。 全靠 LLM 推断型号和价格不能提供可核对证据;一开始绑定某一家原生视频 API 则限制现有模型兼容性。 +- 调度方案比较:仅修补串行队列改动小但模型仍阻塞抓取;全面异步/分布式重写成本高、增加部署负担; + 用户已选择推荐的有界分阶段流水线(§6.1),保留有界队列用于公平性、去重和背压,不引入 Celery/Redis 作为运行前提。 +- 页面继续使用服务端模板及小量 HTMX/JavaScript;改善状态保持与操作反馈,不引入新视觉系统或 SPA 框架。 ## 3. 当前实现与必要改变 @@ -32,6 +39,10 @@ - 网络、模型调用处于较长数据库会话内;需测量锁占用、浏览器启动、重复模型请求与关系查询次数。 - “重新上架”判断发生在 status 被设为 active 之后,是需以回归测试验证的疑点;有限搜索结果未见不等于已下架。 - 当前应用无用户认证。升级不扩大默认网络暴露范围,不借此次改造冒充解决多用户安全问题。 +- 当前队列只有一个消费者,整轮执行完再固定等待 300 秒;抓取前抖动也占着任务执行通道。 + 因此慢模型、媒体处理和纯等待会阻塞其他任务,需要拆分资源通道,而不是简单删除所有队列或无限开线程。 +- `listings_detail.html` 的返回链接固定 `/listings`,卡片未传入来源参数;这解释页面返回按钮丢失筛选。 + 浏览器原生后退、加载更多、HTMX 刷新恢复仍需浏览器回归验证,不能将静态检查等同于已复现全部症状。 ## 4. 阶段一:规格与价格可信度 @@ -128,9 +139,35 @@ MediaAsset 记录 listing/可选 offer、image/video/frame 类型、原始来源 使用 FFmpeg/ffprobe 子进程处理本地受限文件,再把帧转为现有视觉接口支持的图片输入;不要求特定厂商的原生视频 API。 Docker 显式安装 FFmpeg;uv/Conda 检测外部工具,缺失时禁用视频处理并提示安装,不让应用无法启动。 -视频分析为明确开关,默认关闭且显示“未分析视频”;启用后与图片联合分析。每商品最多 1 个视频、8 张原图、6 帧, -每次模型请求总视觉输入最多 8 张(有视频时优先最多 4 张原图 + 最多 4 帧,其余作为详情证据保存)。 -视频上限 100 MiB、180 秒;流式下载最多 60 秒、解码最多 30 秒,超预算中止并删除半成品;媒体处理并发 1。 +视频分析为明确开关,默认关闭且显示“未分析视频”;启用后与图片联合分析。以下均为可见、可配置、服务端校验的资源预算, +不是靠前端隐藏或硬编码数量决定: + +| 设置 | 默认 | 第一版允许范围 | 代价与语义 | +| --- | --- | --- | --- | +| 每商品原图保留数 | 8 | 1–32 | 下载/存储增加,不等于每张均分析 | +| 每商品视频数 | 1 | 1–3 | 每个视频分别占下载/解码预算 | +| 每视频保留采样帧数 | 6 | 1–24 | 解码耗时、磁盘与证据量增加 | +| 每请求原图配额 | 4 | 0–16 | 更多图片通常增加模型用量和延迟 | +| 每请求视频帧配额 | 4 | 0–16 | 同上,抽帧不代表完整理解视频 | +| 每请求总视觉输入上限 | 8 | 1–16 | 必须不大于用户所选模型已知支持限制 | +| 视频文件上限 | 100 MiB | 10–200 MiB | 流式按字节截止,不只信 Content-Length | +| 视频时长上限 | 180 秒 | 15–300 秒 | 超限明确跳过,不偷偷分析前段后冒充全部 | +| 缓存总量 | 1 GiB | 256 MiB–10 GiB | 占用部署机器/容器挂载卷磁盘 | + +混合媒体默认先给原图和视频帧各 4 个名额,按清晰度、去重和覆盖性确定顺序;单一媒体可借用另一类型未使用的名额, +但不能突破总上限。有混合媒体时,某类配额设 0 即用户主动不分析该类,结果必须显示该覆盖限制。 +每类配置配额之和可大于总上限;混合场景按原图/帧交替公平分配至总上限,固定顺序使结果可复现, +默认自然得到 4+4;额外素材保留作证据。无任何可选输入时不发起伪视觉请求。 +仅调整数量不自动拆成多个付费请求、不自动提升并发;执行一次品相评估仅使用一个选定媒体批次。 +模型图像上限未知时保守限定在 8,用户可在高级设置显式确认其模型上限(最高 16); +已知不支持该值时拒绝配置,不伪称所有 OpenAI 兼容服务支持相同图片数。 +提高数量前提示费用/响应延迟/内存/下载及模型超限风险,展示配置前后输入上限变化;无提供商账单信息时不伪造金额。 +默认将模型图片缩放至最长边 1280 像素、限制图片解码像素和请求体大小(默认 16 MiB,最高 32 MiB), +超限时停止该批评估并提示降低设置,不能静默删掉证据后仍标为完全分析。 +这些配置纳入 Settings、设置表单、`.env.example` 与评估缓存指纹,重启后保留。 + +流式下载默认最多 60 秒、解码默认最多 30 秒(均可在高级设置 10–120 秒内调整),超预算中止并删除半成品; +媒体处理默认并发 1,FFmpeg 默认每进程 1 个解码线程;可调范围见 §6.2。 有限采样覆盖起、中、末,过滤模糊/重复帧;不宣称未采样片段已检查,也不对音频或动态故障下确定结论。 下载器验证 URL、DNS 解析地址和每次重定向;禁止回环、内网、链路本地、云 metadata 等目标。 媒体下载与可配置内网模型地址是不同信任边界;不用媒体请求携带任意域 Cookie。子进程不经 shell, @@ -138,7 +175,14 @@ FFmpeg 仅读本地输入、限制网络协议和执行时间;文件输出在 ### 5.3 缓存与证据分析 -缓存位于 data/media-cache,默认上限 1 GiB、保留 7 天;原视频成功抽帧后删除,失败半成品立即清理。 +缓存位于 data/media-cache,默认上限 1 GiB、保留 7 天(可设 1–30 天);原视频成功抽帧后删除,失败半成品立即清理。 +该总上限包含原图、保留帧、下载半成品、处理中视频及其他磁盘中间文件,不另设不受控临时目录。 +媒体任务开始前按有界下载大小和输出文件预算预留空间,所有读写共用配额记账; +先清理无占用的过期/LRU 文件,仍不足则延后任务并显示磁盘受限,不突破用户上限。 +活跃任务的输入/输出保留占用引用,不被清理器删除;结束/失败/进程恢复核对并释放预留。 +小缓存与高并发不能同时满足时降低实际派发并发;若单任务最坏所需空间都无法容纳,拒绝该配置组合或要求降低文件/采样预算, +不进入永远无法启动的等待。设置页解释配置并发与磁盘允许的有效并发,避免“256 MiB 缓存却同时下载两份 200 MiB”。 +同时检查磁盘剩余空间并预留系统安全余量;外部进程占用导致空间不足时失败清理,不能承诺控制整块磁盘的使用量。 按内容 hash/处理参数去重,媒体 URL 改变不直接认定内容改变;无法取得内容时不可凭 URL 缓存成成功。 已被分析引用的帧若因容量被清理,保留摘要、hash、时间位置并显示“证据文件已清理”;可重新获取,不显示坏链接。 API 不接受任意本地路径;媒体通过受控 ID 映射提供,不泄露签名 URL 的令牌、Cookie 和宿主文件路径。 @@ -161,7 +205,7 @@ API 不接受任意本地路径;媒体通过受控 ID 映射提供,不泄露 - 正确性:重新上架、有效价格变化、规格售罄、部分解析失败、重评、暂停/删除任务、通知失败恢复。 - 生命周期:有限搜索结果未见只能称“本轮未见/疑似不可用”;只有详情确证下架才标为确定下架。 - 数据库:外部请求不占写事务;按商品/步骤短事务,迁移失败不留下半套新结构;验证并发读写与 N+1。 -- 浏览器:评估单轮复用浏览器/隔离上下文,取消/异常可靠关闭,不跨任务泄露 Cookie 或页面状态;保留串行任务限速。 +- 浏览器:按 §6.1 的所有权规则复用浏览器/隔离上下文,取消/异常可靠关闭;平台访问限速独立于分析并发。 - 模型:同证据不重复分析;有限超时、退避、重试与单任务调用预算;预算耗尽保留待处理状态,不误报否定。 - 可观测性:每轮记录候选数、详情耗时、媒体耗时、LLM 次数、缓存命中、SQL 次数和失败状态;日志屏蔽秘密。 - 安全回归:表单先校验后提交、敏感配置与运行时配置边界、媒体下载边界、noVNC 默认关闭、路径穿越。 @@ -170,12 +214,154 @@ API 不接受任意本地路径;媒体通过受控 ID 映射提供,不泄露 热缓存不重复启动每商品浏览器、不重复相同媒体的模型分析;记录中位数与 P95、查询数、调用数和资源峰值。 任何优化需证明结果一致、无重复通知、无明显回归;绝不为性能提高抓取频率或取消安全等待。 +### 6.1 有界分阶段流水线 + +APScheduler 仅提交到期信号,不直接执行完整任务;人工重新分析也走同一资源管理器。 +单进程协调器管理持久化 Run/WorkItem(含任务版本、阶段、输入指纹、重试时间、运行租约和状态), +阶段为搜索/详情抓取 → 纯解析与规则裁决 → 可选媒体处理 → 模型分析 → 聚合与写入 → 通知投递。 +配置、凭据不复制到工作项;工作项仅保存必要引用和脱敏错误。线程池适用于网络 I/O,视频用受限子进程, +不为简单 HTML 解析创建大量 CPU 进程,也不使用 OS 专属 fork 实现。 + +- 抓取通道默认单 worker:该线程创建并拥有 Playwright 及浏览器,其他线程不访问这些对象。 + 可配置最多两个抓取 worker,各自拥有完整独立生命周期;每次抓取/任务使用隔离 context,完成后关闭。 + 浏览器按有限寿命与错误情况回收,不跨线程共享页面或持有宿主登录窗口。每账号同时平台导航最多 1, + 即使抓取 worker 增为 2 也不增加同账号并行导航;额外 worker 不是鼓励多账号绕过限制。 +- 平台请求间隔、429/验证码退避在账号级统一生效;默认最小导航间隔 10 秒,高级可在 5–300 秒内设置。 + 不再在整个任务结束后无条件等待 300 秒;等待下一次允许访问时不占用模型或媒体槽位。 + 抖动改为调度器上的 not-before 时间,不用 sleep 占着整个任务执行槽。 + 更低间隔可能增加风控和账号限制风险,需明确提示;平台风险反馈可延长间隔,不能自动缩短用户值。 +- 模型通道独立线程池与全局并发上限;可设 RPM 限额(默认 10,范围 1–60),重试也计入。 + 每请求 timeout、最多 2 次重试,指数退避不占 worker 槽;并发不会绕过 RPM 或提供商返回的 Retry-After。 + 未知厂商限额按保守值处理;已知更低限额优先,不能自动以设备强大为由提高 API 访问量。 +- 解析输出和模型输出为不可变 DTO,线程不传递 ORM 对象/数据库 session。协调器单写入路径, + 写事务短且无外部网络调用;读请求用独立 session。SQL/SQLite 忙等待有限,重试时重新核验任务版本。 +- 所有通道有界:默认全局最多 200 个未完成 WorkItem,每任务最多 60 个;达到上限暂停上游生产。 + 搜索候选和下一轮待处理项去重持久化,超过上限以未完成运行游标/受限剩余 ID 集恢复, + 不把未处理项当作“不匹配/下架”,不让内存无限积压。任务间轮转公平,单任务不会占满所有槽位。 +- 每任务同时仅一个 active run;运行中再次触发只合并为一次后续到期信号,不并行跑同任务。 + 默认同时 active run 数低占用和均衡均为 2(最高 4,可手动降到 1);active 仅表示允许跨任务分阶段推进, + 实际资源占用仍由各通道限制。进程启动按租约恢复已完成与未完成步骤,不整轮盲目重做。 + 等模型/退避的 run 保持 active 身份,但不占抓取槽;默认两个 run 足以使慢模型期间另一任务能抓取。 + 用户主动将 active runs 设为 1 时明确说明会退化为跨任务串行,跨任务重叠验收仅在 active runs 至少 2 时适用。 +- 预算默认每 run 最多 30 个详情、60 个模型 HTTP 尝试(含重试)、15 分钟墙钟期限; + 高级允许分别 1–100、1–200、5–60 分钟。到期后未完成步骤延后至下一次运行,不硬杀共享线程。 + 网络与子进程有独立硬超时;到期/取消后回来的迟到结果不得提交。 +- 暂停任务:停止新调度,取消该任务待执行工作,活跃请求协作结束;恢复后按正常调度重建未完成工作。 + 修改预算/规格等任务条件时递增任务版本:旧工作返回后只可保存为旧证据,不能覆盖新评估或发送通知。 + 删除任务撤销其工作;每次发布结果与写入前用 Run ID + 任务版本 + 租约进行条件检查。 +- 聚合不等待整批所有任务:仅针对当前 run 已就绪、确价、门槛通过的 offer,按批次生成局部比较。 + 每 offer 的正常通知须等该 offer 的需求、价格和品相门槛都定稿;批次“最优”标记须注明比较范围, + 不把未完成候选当作低分或断言全市场最优。 +- 通知先写持久化待投递事件,再在受限 I/O 通道发送;通知投递与业务计算解耦, + 相同业务事件的成功渠道不重发。进程崩溃后的未知投递结果标记 uncertain,默认等待人工重试, + 不无限自动重发不支持幂等键的渠道;新系统不保证外部网络严格 exactly-once。 + +生命周期补充: +- 计算 run 的租约/截止时间约束计算 worker 的结果提交;通知事件有独立投递租约,已合法提交的事件在 run 完成后可继续投递。 + run 超时不撤销此前成功提交的评估事件,也不允许超时后产生新评估事件。 +- 通知每次发送前重新核对任务存在/启用、当前任务版本、评估有效性和必要价格证据新鲜度; + 任务暂停/删除/条件改变时撤销尚未发送事件;证据过期转为 awaiting_revalidation。 + 已开始的外部 HTTP 发送不能保证撤回,明确记录 sent/failed/uncertain,不把已发送历史改成未发生。 +- uncertain 的人工重试也先重新核验;证据已旧则不能直接发出旧推荐。重新核验未实质变化时恢复原业务事件, + 不创建重复通知;证据变化时原事件标为 superseded,并按新证据是否满足条件决定创建新事件。 +- “仅恢复未完成步骤”不代表永远复用已过期结果:跨轮/重启恢复前检查任务版本及各步骤依赖指纹和新鲜度。 + 先刷新过期的价格/规格/可用性;若只是观察时间更新、实质内容未变,复用仍符合内容指纹的媒体/模型结果, + 不重复付费分析;实质内容变化才失效受影响的下游步骤。 + 恢复工作挂入当前有效 run,用当前已核验的观察记录建立评估引用;旧 run 历史保持不可变。 + +### 6.2 用户资源设置与推荐配置 + +任务调度与硬件敏感设置全部可见、服务端有范围校验;资源 profile 只是填写配置的快捷方式,不是第二套隐形配置。 +平台访问间隔、模型 RPM 和每轮预算不是设备检测可自动放宽的设置。 + +| 项目 | 低占用/首次启动默认 | 均衡(用户确认后) | 自定义硬上限 | +| --- | --- | --- | --- | +| active runs | 2(可手动降为 1) | 2 | 4 | +| 抓取 workers | 1 | 1 | 2(每账号导航仍为 1) | +| 模型请求并发 | 1 | 2 | 8 | +| 媒体处理并发 | 1 | 1 | 2 | +| 每个 FFmpeg 线程数 | 1 | 1 | 4 | +| 平台导航最小间隔 | 10 秒 | 10 秒 | 见 §6.1,不随设备调整 | +| 每请求图像上限 | 8 | 8 | 16,且受模型限制 | + +第一版不提供“一键最高性能”,避免把强硬件误认为可忽略平台和模型限流。并发增加会提高浏览器内存、 +请求排队、视频 CPU、模型费用消耗速度与 SQLite 竞争;提高线程数不承诺线性提速。 +资源保护优先级:应用硬上限/容器资源边界 > 用户已确认的资源上限 > profile 默认。 +运行时压力过高只可临时降低有效并发并显示原因,不偷偷改保存值;不会自动提升至未确认值。 + +硬件检测只读、本地执行,不上传硬件标识、不跑压力测试: +- Windows/macOS/Linux 检测 CPU 数、总内存和可用内存;可采用跨平台 psutil,依赖与镜像锁同步。 +- Linux 容器结合 cgroup v1/v2 的 CPU quota、cpuset、memory limit/current 与进程可用资源取较小可用值; + 可获取亲和性时也纳入。无限制、缺文件、权限不足、嵌套容器等情况显示检测依据/不确定性,不能拿宿主全量资源做保证。 +- 首次默认低占用;当有效 CPU 至少 4、有效内存上限至少 6 GiB、当前余量至少 3 GiB 时才建议均衡。 + 不满足或检测不完整建议低占用。保存建议前展示检测时间、容器/本地来源、拟变更字段和代价,用户点击确认后生效。 +- 不检测 GPU 或自动启用硬件解码;GPU/显存不是远程模型推理能力的可用资源。 + +配置变更按版本应用:只影响新派发工作。减少并发先停止派发、让当前请求结束再缩容, +媒体数量/模型变化使后续评估缓存失效,不自动批量付费重跑全部历史商品。 +外部 API 的有效 RPM/超时设置也记录在可见配置中;无有效值时拒绝保存,不以 0 表示无限。 +模型输入数量与资源预算都由同一 schema 供 UI/API/env/worker 使用,避免设置表有值而执行器不读取。 + +### 6.3 全站交互整理(不更换视觉风格) + +沿用现有简洁配色、按钮、字体和服务端模板;不使用前端设计类技能,不增加炫技动画、厚重组件库或 SPA。 +此次重心是导航、状态保持、分类、操作反馈和移动端可用性,不改变产品品牌或信息架构的基本入口。 + +列表上下文以规范化 URL 查询为事实来源,覆盖 task_id、价格可信状态、show、sort、分页/已加载范围。 +点击详情携带经过白名单构造的相对 return_to;详情返回、拉黑、恢复、删除、重新分析后回到原上下文。 +仅允许本站指定列表路径及合法参数,防止开放重定向;无来源时回标准列表,不能直接使用任意 Referer。 +浏览器原生后退/前进、刷新、直接粘贴链接都应得到一致筛选;滚动锚点和加载范围按查询键存在 sessionStorage, +不能跨查询错误恢复。恢复范围有上限,分段加载,商品被删除时返回相近位置而非空白页。 +不承诺动态数据集永远回到同一像素,但必须保留用户筛选和排序;登录 Cookie、密钥不进 URL/浏览器状态。 + +加载更多与 HTMX 刷新共享参数构造器、稳定排序键(加 ID 做同分 tie-breaker)和去重规则。 +自动刷新不重置加载范围/滚动/筛选、不清掉用户已选项,不替换正在编辑的表单; +交互期间可暂停刷新并显示“有新结果,点击更新”,不能靠每两秒整页换 DOM 达到实时。 +全选明确表示“已加载可见结果”,不默默选择整个数据库;危险操作确认显示数量,失败时保留选择及筛选。 + +设置按账号与登录、文本/视觉模型、消息通知、抓取与性能、媒体、部署与诊断分组; +类别使用锚点/轻量 tab(移动端可用),不是视觉重做。支持直接链接到分类,当前分类刷新/返回后保持。 +每分类独立保存,未提交分类不改;复选框仅在该分类的字段白名单内解释未勾选, +密钥留空保持原值,清除须显式确认。未保存更改切换分类/离开时提示,失败保留非敏感输入, +秘密不回显到 URL、localStorage、日志或 HTML 明文。浏览器及服务端都校验,失败不写半套配置。 +部署/启动级字段(数据库路径、绑定地址、容器运行模式等)只读或明确说明需改 env/重启, +不能把所有 Settings 字段无差别当作可从 Web 修改的运行时选项。 + +页面范围与行为验收: +- 仪表盘:区分排队、抓取、分析、待询价、待核验、失败;数字与任务状态一致,不显示虚假进度百分比。 +- 任务列表/详情:新增目标规格和待询价开关;运行按钮防重复提交,暂停/取消语义与 §6.1 一致, + 编辑保存后留在当前任务/原列表位置,清楚反馈是否有旧版本工作被取消。 +- 商品列表/详情:保持筛选与位置,区分标价/确价/人工报价和证据来源;未知价格不展示为 0 或“超值”。 +- 消息通知:保留筛选/分页,区分 pending/sent/failed/uncertain,重试仅影响所选渠道,不能靠前端刷新伪造成功。 +- 设置:分类、局部保存、配置代价、设备建议确认和生效范围;缺视频工具或模型上限不明时有操作指引。 +- 全局:统一加载、空结果、错误/成功反馈;错误 HTTP 响应不能被当作成功后直接 reload;危险操作确认、键盘可达、 + 窄屏表单/导航无横向溢出。无 JS 时核心筛选和表单提交仍能用;恢复滚动等增强可降级。 + +### 6.4 分阶段顺序、基准与发布边界 + +执行前先记录原行为与失败案例、建立隔离测试配置和性能基线;随后: +1. 规格/价格域及兼容迁移(阶段一),先稳定正确性。 +2. 媒体证据与可配置输入预算(阶段二),默认不增加付费视频请求。 +3. 有界流水线与资源配置(阶段三前半),将前两阶段接入统一执行器;旧执行器仅在过渡测试使用, + 交付时不得同时消费相同工作。再完成交互整理、性能对比、全项目缺陷修复与最终验收。 + +使用隔离数据库、固定合成媒体和 fake 平台/模型时钟测试;不以真实平台压测来证明性能。 +关键场景为慢模型下下一任务抓取仍可在平台节流允许后推进、并发峰值永不超过配置、压力降档不丢工作、 +任务版本变化拦截迟到写入/通知、重启仅恢复未完成步骤、挂起任务不会饿死其他任务。 +浏览器交互验证使用 Playwright 等功能测试,不调用前端设计技能;Windows 本地+Linux Docker 为实测目标, +macOS 及其他架构未实际运行须标注验证边界,不能因“跨平台库”就声称所有设备已测过。 +设计与实现均不新增操作系统专属启动前提;Linux 容器只读资源检测是能力增强,本地检测失败必须可降级。 + ## 7. 迁移、兼容与回退 开发和 Docker 验证使用独立数据库、缓存、容器名和未占用的回环端口,禁止挂载生产 data/;默认禁用真实通知与定时任务。 生产升级前停写并制作可恢复备份;迁移按版本、幂等、事务执行,先验证临时旧库副本,不能运行未备份的破坏性迁移。 新增表与字段保留原数据;旧评分标记 legacy_unverified,不转换为新规格的确定结论;原有通知历史不触发重发。 旧任务 target_variant 为空、待询价外部通知关闭;旧单件下次成功详情抓取后生成默认 offer,未知 SKU 不凭区间构造。 +旧安装升级时保留原有每请求最多 4 张的视觉输入预算,视频仍关闭;新安装默认总上限 8、混合配额 4+4。 +迁移记录明确区分已有安装与新初始化,不能根据有没有 API Key 猜测;已有显式预算始终保留。 +旧用户可在媒体设置中查看从 4 提升到 8 的代价并显式确认;启用视频或应用硬件 profile 不隐式提高媒体总上限。 +低占用/均衡快捷配置只修改并发资源字段,媒体数量和视频开关另行确认,保持升级不默默增加模型用量。 任务删除、商品删除要覆盖新关联记录,防止孤儿数据;同一外部 ID 跨任务仍隔离。 关闭新视频能力不会影响原有图片功能;已升级数据库不承诺直接降级可写,完整回退采用备份数据库与旧镜像成对恢复。 保留本机已忽略的 Windows 快捷脚本、私人文档和 .env;不提交凭据、真实数据库、视频缓存和个人资料。 @@ -195,6 +381,11 @@ API 不接受任意本地路径;媒体通过受控 ID 映射提供,不泄露 7. Ruff、锁文件检查、全套 pytest、覆盖率不低于现有 80% 门槛;关键新裁决分支需直接覆盖,不能只靠总覆盖率。 8. Docker 构建、SDK 导入、FFmpeg 检测、独立数据库启动、healthz、表单/详情/提醒预览与 noVNC 模式检查通过。 9. 提供性能基线与改后对比,未运行项和受外部环境限制的项如实列出,不以合成样本代替真实平台验证。 +10. 媒体预算在 UI/API/env 一致,默认混合 4+4,单一媒体借额、不超过总上限;调整时展示代价,非法/超模型上限拒绝。 +11. 调度去重、公平性、背压、队列上限、资源峰值、429 退避、退出重启、暂停取消、迟到结果、未知投递结果均有回归测试。 +12. 容器限额覆盖宿主资源;检测失败推荐低占用;设备推荐不自动应用,也不提高平台/API 限额。 +13. 商品筛选→加载更多→详情→返回、原生后退/前进、刷新、拉黑/删除/重评均保留上下文;设置分类保存不会修改其他分类。 +14. 所有页面的加载/失败反馈、窄屏、无 JS 核心功能可用;保持原有简洁风格,不用前端设计技能。 真实平台适配需要合法可访问的多 SKU、询价、纯视频/混合媒体代表样本;采集夹具必须脱敏。 若缺真实样本可完成领域模型、规则与夹具测试,但不能将真实 SKU 自动提取/视频覆盖宣称完成; @@ -203,10 +394,14 @@ API 不接受任意本地路径;媒体通过受控 ID 映射提供,不泄露 ## 9. 待批准的提议与实施者可决定事项 待用户随设计一并批准:待询价站内保留但外部提醒默认关闭;人工报价默认 7 天;视频抽帧可选且默认关闭; -分阶段完成三个阶段,真实样本验证不足时明确标为限制而非伪造兼容结论。 +分阶段完成三个阶段(含新版流水线和全站交互),真实样本验证不足时明确标为限制而非伪造兼容结论。 以上默认值均为助手提议,不表示用户此前逐项确认。 同样待批准的裁决:有效人工报价优先但来源切换不算降价;新鲜度过期暂停推荐、保留报价; 最低品相分大于 0 时未核验不得进入正常推荐。实施计划必须显式覆盖这些变化与回归测试。 +新版提议默认:首次使用低占用 profile(两个轻量 active run、每资源通道并发仍为 1),设备推荐需确认; +新安装媒体默认总上限 8、混合 4+4 可调,旧安装升级保留总上限 4,显式确认后才提升; +平台节流替代旧任务结束后的固定等待,导航默认间隔 10 秒、模型 RPM 默认 10;资源和输入硬上限如 §5.2/§6.2。 +用户已确认方向与方案,但这些精确默认数值和完整生命周期规则仍随本设计一并批准。 实施者可决定:内部模块命名、索引具体形式、帧去重算法及可维护的采样细节、测试工具与基准夹具组织; 不得改变不自动联系卖家、不用未知价排名、不触碰真实数据、不默默扩大资源预算等边界。 @@ -216,3 +411,11 @@ API 不接受任意本地路径;媒体通过受控 ID 映射提供,不泄露 已由独立工程师审查初稿,发现三个会影响实施的重要歧义:人工报价与页面价的优先级、刷新失败/过期后的推荐资格、 最低品相分对未核验商品是否构成硬门槛。已在 §4.4、§4.5、§5.3 补充确定规则,作为设计提议交由用户批准。 审查未涉及实现正确性;具体代码、真实平台样本与性能结果仍须在后续阶段另行验证。 + +修订版 2 再次独立审查发现五项实施歧义,现已补充: +1. 默认两个 active run、资源 worker 仍为 1;手动设为 1 时明确跨任务串行代价。 +2. 通知采用独立投递生命周期,发送前重新核验任务版本/证据;完成计算不等于通知失效。 +3. 跨轮恢复先校验过期依赖,内容不变复用模型结果,内容变化才重做相关分析。 +4. 旧安装保留原每请求 4 张预算,新安装默认 8,硬件建议不默默改变媒体用量。 +5. 磁盘总预算包含临时文件,预留空间与背压确保配置的缓存上限有意义。 +以上补充均为助手提出的确定规则,随本文一并交用户批准,未开始实现。 From ac8ed1319b9e36d6109a5d352bb6ff38e301fcc0 Mon Sep 17 00:00:00 2001 From: Comui520 Date: Mon, 26 Oct 2026 10:28:52 +0800 Subject: [PATCH 03/17] docs: plan staged offer media and pipeline upgrade --- .../2026-10-26-phase1-offer-pricing-plan.md | 97 ++++++++++++++ .../2026-10-26-phase2-media-evidence-plan.md | 98 ++++++++++++++ ...-10-26-phase3-pipeline-interaction-plan.md | 121 ++++++++++++++++++ .../plans/2026-10-26-upgrade-roadmap.md | 83 ++++++++++++ .../2026-10-26-variant-price-media-design.md | 4 +- 5 files changed, 401 insertions(+), 2 deletions(-) create mode 100644 docs/superpowers/plans/2026-10-26-phase1-offer-pricing-plan.md create mode 100644 docs/superpowers/plans/2026-10-26-phase2-media-evidence-plan.md create mode 100644 docs/superpowers/plans/2026-10-26-phase3-pipeline-interaction-plan.md create mode 100644 docs/superpowers/plans/2026-10-26-upgrade-roadmap.md diff --git a/docs/superpowers/plans/2026-10-26-phase1-offer-pricing-plan.md b/docs/superpowers/plans/2026-10-26-phase1-offer-pricing-plan.md new file mode 100644 index 0000000..08931ab --- /dev/null +++ b/docs/superpowers/plans/2026-10-26-phase1-offer-pricing-plan.md @@ -0,0 +1,97 @@ +# 阶段一:规格与可信价格 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 使预算、排名和提醒只使用匹配规格的可靠价格,支持待询价与独立人工报价,安全迁移旧库。 + +**Architecture:** 纯函数裁决 + 不可变 DTO + 短事务服务;保持原调度入口可用,后续用相同阶段函数接新协调器。 + +**Tech Stack:** Python >=3.11、Pydantic、SQLAlchemy/SQLite、Playwright、pytest。 + +**Spec:** `D:/good-price/docs/superpowers/specs/2026-10-26-variant-price-media-design.md` §3–4、§7–9。 + +## Global Constraints + +继承 `2026-10-26-upgrade-roadmap.md` 全部约束。金额用分整数与 Decimal 转换;人工报价默认 7 天;未知价不排名;最低品相分 >0 且未核验时进入待核验。 + +## Review Focus + +- 测试 import 副作用触碰生产库(P1.1);“256GB”匹配“128GB”或同名组合合并(P1.2/P1.3)。 +- ORM 迁移隐式丢数据、外键孤儿(P1.2);人工价/页面价切换伪降价(P1.3/P1.4);部分表单清空报价(P1.5)。 + +### P1.1:测试隔离与无副作用应用工厂 + +**Files:** 修改 `goodprice/main.py`、`goodprice/config.py`、`goodprice/__main__.py`、`tests/conftest.py`、`tests/test_main.py`;新增 `tests/test_import_isolation.py`、`scripts/benchmark_pipeline.py`。 + +**Interfaces:** `build_app(settings: Settings | None = None, session_factory=None, with_scheduler: bool = True) -> FastAPI` 保持;新增 `create_app() -> FastAPI` 作为部署工厂;uvicorn 改 factory 调用。日志路径从显式 settings/data_root 获取,import 不 mkdir/init_db/backfill。 + +- [ ] 写 `test_import_does_not_initialize_db_or_write_logs`:子进程在临时 cwd 导入 main/入口,不出现项目 data 写入、连接或调度启动;对 Path/engine 禁止生产路径访问。写 `test_explicit_app_uses_only_temporary_store`,断言 app 数据库、日志、通知与 scheduler 隔离。 +- [ ] 运行 `uv run --locked pytest -q tests/test_import_isolation.py`,应因当前 import-time app 初始化而失败;保存失败证据,不运行真实库。 +- [ ] 实现 lazy factory,fixture 在收集前指定临时资源目录,所有模型与平台测试使用 fake/MockTransport;新增基准脚本只构建显式测试工厂,拒绝非临时数据库与真实外部 URL。 +- [ ] 运行上述用例和 `tests/test_main.py`,应通过;运行基准合成 30 件商品,保存原架构耗时/SQL/调用次数到测试生成目录,不提交机器私有报告。 +- [ ] 独立审查无副作用与入口兼容后,提交 `test: isolate app startup and establish pipeline baseline`。 + +### P1.2:规范化规格与幂等迁移 + +**Files:** 新增 `goodprice/offers/__init__.py`、`goodprice/offers/types.py`、`goodprice/offers/matching.py`、`goodprice/migrations.py`、`tests/test_offer_migrations.py`、`tests/test_offer_identity.py`;修改 `models.py`、`db.py`、`services/task_service.py`、`web/routes.py` 的 TaskCreate/任务表单。 + +**Interfaces:** +- `stable_offer_key(platform_sku_id: str | None, attributes: dict[str, str]) -> str | None`;歧义不强制造 key。 +- `match_variant(target: str, attributes: dict[str, str], title: str) -> Literal['matched','mismatched','uncertain']`。 +- PriceClaim:`source: str`、`kind: str`、`text: str`、`amount_cents: int | None`、`min_cents/max_cents: int | None`、`scope_key: str | None`、`observed_at: datetime`、`evidence_id: str`。 +- OfferCandidate:`key: str`、`attributes: dict[str,str]`、`availability: str`、`claims: tuple[PriceClaim,...]`;提供默认单件 offer 而非区间 SKU。 +- 新增 ListingOffer、OfferObservation、ManualQuote、OfferEvaluation、SchemaVersion 表;WatchTask 新字段 target_variant/notify_pending_price/version,Notification 增可空 offer_id/event_key;旧字段保留。 + +- [ ] 写迁移测试:v0.1.1 合成库迁移两次,原任务/通知/快照逐项不变;旧最低/最高价不变成真实 offer,标记 legacy_unverified;默认新字段不触发重发;删除父记录无新孤儿。 +- [ ] 写身份/匹配测试:不同显示顺序 key 不变、相同属性但不同原站 SKU 不合并、256GB 不匹配 128GB、目标含否定约束且无证据时 uncertain。 +- [ ] 运行 `pytest tests/test_offer_migrations.py tests/test_offer_identity.py -q`,确认新接口缺失导致失败。 +- [ ] 实现模型/DTO/带版本事务迁移,识别旧安装须发生在初始化改变表结构之前;备份恢复只在临时库演练。任务 API/表单统一白名单,版本随业务条件改变递增。 +- [ ] 全量验证旧模型/DB/TaskService/API 测试及新测试,范围默认取 Spec;审查后提交 `feat: add versioned offer and price evidence storage`。 + +### P1.3:价格、来源、报价和新鲜度裁决 + +**Files:** 新增 `goodprice/offers/pricing.py`、`goodprice/offers/service.py`、`tests/test_offer_pricing.py`、`tests/test_manual_quotes.py`。 + +**Interfaces:** +- `parse_money_cents(raw: str) -> int`,有限非负人民币金额,精度超过分拒绝,不浮点猜价。 +- QuoteInput:`offer_key: str, amount_cents: int, confirmed_at: datetime, expires_at: datetime, evidence_fingerprint: str, revoked: bool=False`。 +- PriceDecision:`state: str, amount_cents: int | None, source: str | None, evidence_ids: tuple[str,...], reason: str, fresh_until: datetime | None`。 +- `decide_price(offer: OfferCandidate, quote: QuoteInput | None, now: datetime, ttl_seconds: int) -> PriceDecision`。 +- `is_comparable_drop(previous: PriceDecision, current: PriceDecision, same_offer: bool) -> bool`。 +- `OfferService(session_factory)` 的 `record_observation(listing_id: int, offer: OfferCandidate) -> int`、`confirm_quote(offer_id: int, quote: QuoteInput) -> int`、`revoke_quote(quote_id: int) -> None`,校验与保存同事务。 + +- [ ] 写最小断言:`parse_money_cents('1500.00') == 150000`;参数化拒绝 NaN/Infinity/负数/不支持币种/小数三位。 + 最小红绿测试: + ```python + def test_money_uses_integer_cents(): + assert parse_money_cents("1500.00") == 150000 + ``` +- [ ] 写确价、range、询价、冲突、未知、作用范围不明、SKU 售罄/未知、人工价优先/变化暂停/七天过期、一个调度间隔证据过期测试;页面1500/报价1400/预算1450须使用有效报价,不用页面价。 +- [ ] 写 `test_source_switch_is_not_price_drop`、`test_refresh_failure_does_not_extend_freshness`、`test_revoke_keeps_original_observation`,运行新用例确认红。 +- [ ] 实现优先级和新鲜度纯函数,服务在 commit 前校验合并值;指纹排除纯采集时间变化、包含价格/规格/作用范围。保存证据与人工来源分离。 +- [ ] 运行新用例与原 satisfaction/crawl tests;审查后提交 `feat: resolve trustworthy prices and manual quotes`。 + +### P1.4:真实 SKU 采集和价格正确性接入 + +**Files:** 新增 `goodprice/crawler/offers.py`、`goodprice/services/offer_pipeline.py`、`tests/test_offer_parser.py`、`tests/test_offer_pipeline.py`;修改 crawler/base.py、parser.py、xianyu.py、services/crawl_service.py、satisfaction.py;新增脱敏/合成 fixtures。 + +**Interfaces:** `parse_offer_candidates(html: str) -> list[OfferCandidate]`;`OfferPipeline(session_factory)` 的 `evaluate_listing(listing_id: int, task_id: int, candidates: list[OfferCandidate], now: datetime) -> list[int]` 返回已保存评估 ID;网络/模型通过注入客户端执行,不把 session 包在请求周围。 + +- [ ] fixture 包含配件99/128GB1200/256GB1500;测试目标256GB预算1400时不入选、1600时入选、搜索标价99低于任务下限仍抓详情。文本私聊价保留待确认且没有性价比调用。 +- [ ] 写任务 fetch_detail=False、20组合截断、30详情预算、空/异常详情不误判售罄、旧商品刷新、重新上架状态、最低品相分未核验不放行等失败用例。 +- [ ] 运行 `pytest tests/test_offer_parser.py tests/test_offer_pipeline.py -q` 确认红。 +- [ ] 实现页面范围限定解析和有界只读选项交互,确切来源才确价,移走详情前价格过滤;普通单件生成默认 offer。未适配页面保留 unknown,不将平台状态猜成支持格式。 +- [ ] 将已有串行运行接到相同 offer 阶段函数,保存与通知以 offer 事件引用;旧 listing 标价仅兼容展示,去重不按采集时间每轮生成新事件;有限搜索未见改为疑似未见。 +- [ ] 无样本时写明实验性边界,运行 parser/crawler/crawl/notification 全部测试及新回归;审查后提交 `feat: evaluate listings using matching offer prices`。 + +### P1.5:规格/价格交互、通知与阶段验收 + +**Files:** 新增 `goodprice/web/offer_routes.py`、`tests/test_offer_api.py`;修改 routes.py 注册 router,商品卡片/详情/任务表单/通知文案及原 API 测试。 + +**Interfaces:** GET `/api/listings/{id}/offers`,POST `/offers/{id}/quotes`(金额、到期、确认原文),POST `/quotes/{id}/revoke`;响应包含 price_state/effective_price/source/fresh_until/evidence,不复用旧 price 的“确认价”含义。 + +- [ ] 写缺失/错误报价返回422且无写入;不存在 offer 返回404;报价不会被抓取覆盖;待询价开关关闭不外发;同规格相同事件成功渠道不重发、来源切换不叫降价。 +- [ ] 明确兼容测试:旧API仍有原页面标价 `price`,新 effective_price/source 不隐式取旧值;父摘要基于当前任务已匹配offer确定生成,多个offer时显示范围/数量而不伪称统一确价。 +- [ ] 运行新增 API/通知测试确认红;实现模板与路由,状态 unknown/pending 不生成本批最优,不把空价渲染成0;未保存秘密不回显。 +- [ ] 执行 `uv lock --check`、`uv run --locked ruff check .`、`uv run --locked pytest -q --cov --cov-fail-under=80`;临时库验证迁移/恢复、api 与页面渲染。 +- [ ] 阶段独立审查,按指纹/金额/旧库/提示误导检查;红绿回归修复后提交 `feat: expose offer evidence and inquiry workflow`,勾选总计划阶段一记录。 diff --git a/docs/superpowers/plans/2026-10-26-phase2-media-evidence-plan.md b/docs/superpowers/plans/2026-10-26-phase2-media-evidence-plan.md new file mode 100644 index 0000000..8086f5b --- /dev/null +++ b/docs/superpowers/plans/2026-10-26-phase2-media-evidence-plan.md @@ -0,0 +1,98 @@ +# 阶段二:可配置媒体证据 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 安全识别商品图片/视频、受限抽帧,以用户确认的输入预算形成可追溯分析,而非虚构已看完整视频。 + +**Architecture:** 媒体模块从详情提取结果生成规范证据;受控下载器和本地 FFmpeg 处理,配额管理器统一计算临时/永久文件;以 data URL 复用现有 chat/responses 客户端。 + +**Tech Stack:** Python >=3.11、httpx、Pillow、FFmpeg/ffprobe、SQLAlchemy、pytest;Docker 显式安装 FFmpeg,本地缺工具优雅降级。 + +**Spec:** `D:/good-price/docs/superpowers/specs/2026-10-26-variant-price-media-design.md` §5、§6.2、§7–9;前置为阶段一计划完成。 + +## Global Constraints + +继承总计划。视频默认关闭;默认保留8原图/1视频/每视频6帧,请求原图配额4/帧配额4/总上限8,旧安装总上限保留4;不自动多批收费。视频100 MiB/180秒、下载60秒/解码30秒、缓存1 GiB/7天,范围严格按 Spec §5.2/§6.2。 + +## Review Focus + +- 签名 URL 刷新不等于媒体变化,下载失败不缓存成成功(P2.1/P2.3)。 +- DNS 重绑定/重定向内网、图片解码炸弹、本地路径穿越(P2.2/P2.3)。 +- 两个200 MiB任务突破256 MiB缓存、清理器删活跃输入(P2.2)。 +- 模型虚构 evidence ID 或拿样品评分代表具体 SKU(P2.4)。 +- 应用硬件 profile 无意提升旧安装4张上限、视频缺工具导致启动失败(P2.1/P2.5)。 + +### P2.1:媒体契约、数量配置与提取 + +**Files:** 新增 `goodprice/media/{__init__,types,selection}.py`、`tests/test_media_selection.py`、`tests/test_media_parser.py`;修改 config.py、models.py、migrations.py、crawler/base.py/parser.py/xianyu.py、.env.example、设置一致性测试。 + +**Interfaces:** +- MediaCandidate:`key: str, kind: Literal['image','video'], url: str, offer_key: str | None, source_locator: str, poster_url: str | None`。 +- MediaEvidence:`id: str, kind: Literal['image','frame'], content_hash: str, local_path: Path, offer_key: str | None, video_id: str | None, timestamp_ms: int | None`。 +- MediaSelection:`items: tuple[MediaEvidence,...], omitted_ids: tuple[str,...], limitations: tuple[str,...]`。 +- `select_media(images: list[MediaEvidence], frames: list[MediaEvidence], image_quota: int, frame_quota: int, total_limit: int) -> MediaSelection`:混合交替公平分配,单类型可借额,设0的媒体明确未分析。 +- `extract_media_candidates(html: str) -> list[MediaCandidate]`:仅商品主媒体,未适配 blob/HLS/DRM 标记 unsupported,不自行下载。 +- 新 MediaAsset ORM 记录来源、类型、父视频、处理状态、hash、时间位置;新设置名称统一用 `media_*`(数量/上限/超时)及 `video_analysis_enabled`。 + +- [ ] 写 `test_default_mixed_selection_is_four_images_four_frames`、`test_single_type_borrows_unused_quota`、`test_zero_quota_reports_omitted_type`、`test_invalid_or_over_model_limit_rejected`。 + 对含至少4张有效原图和4帧的 fixture,断言: + ```python + selected = select_media(images, frames, image_quota=4, frame_quota=4, total_limit=8) + assert len(selected.items) == 8 + assert sum(item.kind == "frame" for item in selected.items) == 4 + ``` +- [ ] 写纯图/纯视频/混合/推荐流排除/非实际SKU媒体共享与顺序稳定测试;迁移旧配置总上限4且视频关闭,新安装8,既有显式值保留。 +- [ ] 运行新测试确认红;实现类型、表迁移、配置字段和配额选择器,Settings 跨字段验证引用 Spec 精确范围,未知模型上限保守8,显式确认最高16。 +- [ ] 运行媒体/配置/迁移测试,审查后提交 `feat: configure media evidence and input budgets`。 + +### P2.2:统一磁盘配额与安全下载 + +**Files:** 新增 `goodprice/media/budget.py`、`download.py`、`cache.py`、`tests/test_media_budget.py`、`tests/test_media_download.py`。 + +**Interfaces:** +- `MediaBudget(root: Path, max_bytes: int, retention_seconds: int)`,`reserve(key: str, bytes_needed: int)` 返回上下文管理器/预留token;`commit(token: str, paths: list[Path]) -> None`、`release(token: str) -> None`。 +- `MediaDownloader(budget: MediaBudget, transport=None, resolver=None)` 的 `download(candidate: MediaCandidate, max_bytes: int, timeout_seconds: int) -> Path`;缓存命中由内容和处理参数决定,不只由 URL 决定。 +- 请求使用经过验证/绑定的目的地址与原始 hostname TLS 校验;若底层不能防止解析后重绑定则域不安全,不能只 resolve 一次后普通 client.get。 + +- [ ] 写响应无 Content-Length/伪长度/重定向metadata/IPv6内网/解析后重绑定/凭据跨域/慢流/断流测试;失败半成品不存在,异常不含URL令牌和Cookie。 +- [ ] 写256 MiB配额下两个200 MiB任务只有一个可派发,活动输入不被LRU删除,单任务预算不能容纳时明确报错,重启回收预留且不突破缓存根路径。 +- [ ] 执行 `pytest tests/test_media_budget.py tests/test_media_download.py -q` 确认红。 +- [ ] 实现受限流式下载与重定向逐跳校验,临时文件位于同一受控根目录;配额包含全部中间文件,预先保守估算输出,磁盘剩余不足则延后/拒绝,不偷偷删证据继续算成功。 +- [ ] 运行新用例和媒体选择回归,审查网络与路径边界后提交 `feat: bound media downloads and cache disk usage`。 + +### P2.3:受限解码、抽帧与可复现缓存 + +**Files:** 新增 `goodprice/media/frames.py`、`processing.py`、`tests/test_video_frames.py`、`tests/test_media_processing.py`;修改 Dockerfile、pyproject.toml、uv.lock、requirements.lock、.dockerignore。 + +**Interfaces:** `probe_video(path: Path, timeout_seconds: int) -> VideoInfo`(duration_seconds/dimensions);`extract_frames(path: Path, count: int, output_dir: Path, timeout_seconds: int, threads: int) -> list[FrameFile]`(path/timestamp_ms/hash);`prepare_image(path: Path, max_edge: int = 1280) -> Path`;`MediaProcessor.process(candidates: list[MediaCandidate], settings: Settings) -> list[MediaEvidence]`。 + +VideoInfo 与 FrameFile 定义在 media/types.py,字段分别为 `duration_seconds: float, width: int, height: int` 和 `path: Path, timestamp_ms: int, content_hash: str`。配额 `reserve` 返回 `BudgetReservation(token: str, bytes_reserved: int)` 上下文对象;未commit的退出总会release。 + +- [ ] 合成小视频在临时目录生成并记录清晰帧时间位置;假子进程测试缺FFmpeg、进程超时、超180秒、输出超预留配额和非法协议输入;图片测试超解码像素、伪MIME和坏文件。 +- [ ] 测试 content hash 相同只解码一次、签名URL变化内容相同不付费重算、重复/模糊帧不会补造达到数量;缓存被清理显示证据不可用,不返回坏路径。 +- [ ] 运行新测试确认红;实现参数数组启动FFmpeg,禁止shell/网络输入,默认线程1、受控超时、有限输出,Pillow处理前先检查像素上限。 +- [ ] Docker 安装FFmpeg;Pillow加入正式依赖,uv lock后以 `uv export --locked --no-dev --extra jev --no-emit-project --format requirements.txt --output-file requirements.lock` 同步镜像依赖,检查无无关升级。 +- [ ] Windows真实工具可用时跑抽帧;否则标明缺失并在Linux隔离镜像跑真实工具测试。本地缺工具页面可打开,视频状态标为不可用。审查后提交 `feat: extract bounded video frame evidence`。 + +### P2.4:联合视觉分析与证据校验 + +**Files:** 新增 `goodprice/analysis/media_analysis.py`、`tests/test_media_analysis.py`;修改 analysis/llm.py/prompts.py、services/offer_pipeline.py、services/crawl_service.py、评估序列化。 + +**Interfaces:** `analyze_media(offer: OfferCandidate, selection: MediaSelection, client: LLMClient, requirement: str, request_limit_bytes: int) -> dict`;`validate_evidence_references(verdict: dict, allowed_ids: set[str]) -> dict`;结果含 observations、seller_claims、conflicts、unverified、condition_score 可空、evidence引用。 + +- [ ] MockTransport 断言 chat与responses输入使用可解码 data URL、精确选中数量及证据 ID,不含宿主路径;16 MiB上限超出则不发送请求,不静默缩减输入。 +- [ ] 测试模型引用不存在ID被拒绝、共享样品不能产生SKU确定品相、视频失败仅图分析时标明覆盖限制、无输入不产生伪评分、最低分>0未知时待核验。 +- [ ] 测试缓存 key 包含内容/任务条件/选图参数/模型/提示词版本;只刷新采集时间不重复模型调用,调整设置不自动重跑历史。 +- [ ] 运行新用例确认红;实现证据提示词、输入构造、schema校验和精确缓存失效,与价格结果分开,不允许视觉模型反写实际售价。 +- [ ] 跑全部LLM/Jev/offer/media测试及审查后提交 `feat: analyze mixed media with traceable evidence`。 + +### P2.5:媒体设置、证据展示与阶段验收 + +**Files:** 新增 `goodprice/web/media_routes.py`、`tests/test_media_api.py`;修改设置/商品详情模板、router注册、README、.env.example。 + +**Interfaces:** GET `/media/{asset_id}` 仅内部ID映射受控路径;GET `/api/media/capabilities` 返回FFmpeg可用性和有效预算,不返回宿主路径;媒体配置待阶段三分类保存统一管理,字段校验复用Settings。 + +- [ ] 写路径穿越/未知ID/文件清理返回404,页面显示抽帧时间位置/未分析/工具缺失;数量增大显示费用/延迟/内存/存储/模型限制,确认前不扩大旧4张预算。 +- [ ] 运行新用例确认红;实现最小简洁表单和证据展示,生成HTML不泄露签名URL令牌,本地缺FFmpeg提供安装指引而不静默安装系统工具。 +- [ ] 全套Ruff/锁文件/pytest覆盖率>=80%,隔离Docker构建和FFmpeg/SDK检测;媒体下载全为fake或受控测试资源,不挂生产data。 +- [ ] 阶段独立审查安全与资源边界,修复回归并提交 `feat: expose media evidence and explain analysis costs`;记录真实平台未验证项,勾选总计划阶段二。 diff --git a/docs/superpowers/plans/2026-10-26-phase3-pipeline-interaction-plan.md b/docs/superpowers/plans/2026-10-26-phase3-pipeline-interaction-plan.md new file mode 100644 index 0000000..df8bd0e --- /dev/null +++ b/docs/superpowers/plans/2026-10-26-phase3-pipeline-interaction-plan.md @@ -0,0 +1,121 @@ +# 阶段三:有界流水线、资源推荐与交互 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 让抓取、媒体、模型不再互相串行阻塞,在可配置的资源边界内可靠恢复,并修复全站导航和设置交互。 + +**Architecture:** 持久化状态与短事务单写入协调器,有限线程池处理I/O,Playwright由所属线程独占,FFmpeg为受限子进程;模板保留简洁风格。 + +**Tech Stack:** Python >=3.11、APScheduler、线程池、SQLAlchemy/SQLite、Playwright、HTMX、psutil、pytest。 + +**Spec:** `D:/good-price/docs/superpowers/specs/2026-10-26-variant-price-media-design.md` §6–9;前置为阶段一、二。 + +## Global Constraints + +继承总计划;任务同一时刻一个run,最多合并一次后续到期信号。默认active runs=2,抓取/模型/媒体worker各1,FFmpeg线程1;可调上限为4/2/8/2/4。平台间隔默认10秒(5–300),每账号并行导航1;API RPM默认10(1–60)。配置只影响新派发工作,降低并发先排空,设备建议不自动提高配额。 + +## Review Focus + +- 修改/删除/暂停后的迟到结果和重启通知重复(P3.1–P3.3)。 +- 达到工作项上限时游标丢失、单任务饿死其他任务(P3.1/P3.2)。 +- 容器无quota/嵌套限额/低内存误报强硬件(P3.4)。 +- 查询丢失、加载更多重复、未提交设置分类被清空(P3.5/P3.6)。 +- 性能数据改善来自减少分析覆盖而非架构提速(P3.7对比同数据/同输入预算)。 + +### P3.1:持久化工作状态、租约与版本 + +**Files:** 新增 `goodprice/execution/{__init__,types,store}.py`、`tests/test_execution_store.py`;修改models.py/migrations.py、services/task_service.py。 + +**Interfaces:** Stage 枚举 search/detail/parse/media/analyze/aggregate/notify;RunToken:run_id/task_id/task_version/lease_id;WorkResult:token/stage/input_fingerprint/output 的不可变DTO;新增 TaskRun/WorkItem/NotificationEvent/NotificationDelivery ORM。 +`ExecutionStore(session_factory)` 提供 `submit(task_id: int, kind: str, target_id: int | None = None) -> int | None`、`claim(stage: Stage, now: datetime) -> WorkItemDTO | None`、`complete(result: WorkResult, now: datetime) -> bool`、`recover(now: datetime) -> int`、`invalidate_task(task_id: int) -> None`;只传ID/DTO,不向worker泄漏session。 + +所有DTO在execution/types.py:RunToken各ID为int、lease_id为str;WorkItemDTO含 `id: int, token: RunToken, stage: Stage, input_fingerprint: str, payload: dict, attempt: int, deadline_at: datetime`;WorkResult.output为dict。Run/Work状态与通知状态分开,禁止拿NotificationDelivery的sent冒充Run完成。 + +- [ ] 写同任务去重、运行中多次触发只补一次、全局200/每任务60背压、恢复游标、暂停删除和版本变更拒绝迟到complete、租约只被一个worker领取等失败测试。 +- [ ] 写跨轮观察过期先刷新,实质内容不变复用已有模型结果;观察变化使相关下游失效,不把未知当否定。 +- [ ] 运行 `pytest tests/test_execution_store.py -q` 确认红;实现带版本迁移、条件更新和短事务状态机,notification独立租约而非依赖已结束run。 +- [ ] 回归迁移/报价/任务服务;独立审查后提交 `feat: persist bounded task execution state`。 + +### P3.2:资源通道、调度与取消恢复 + +**Files:** 新增 `goodprice/execution/limits.py`、`workers.py`、`coordinator.py`、`tests/test_execution_limits.py`、`tests/test_coordinator.py`;修改scheduler.py、main.py、crawler/xianyu.py、services/crawl_service.py、旧queue测试。 + +**Interfaces:** +- `ResourceLimits.from_settings(settings: Settings) -> ResourceLimits`(全部并发、RPM、间隔、工作项/run预算),含服务端范围验证。 +- `RateGate.reserve(key: str, now: float) -> float` 返回最早允许时间;`WorkerPools.dispatch(stage: Stage, token: RunToken, payload: dict) -> Future[WorkResult]`。 +- `Coordinator(store: ExecutionStore, limits: ResourceLimits, workers: WorkerPools, clock)` 提供 `start()/stop(timeout: float)/submit(task_id: int)/submit_reanalysis(listing_id: int)/pause(task_id: int)/update_limits(limits: ResourceLimits)`。 +- `BrowserWorker` 所在线程负责 playwright/browser/context 创建/使用/关闭;平台节点执行包含SKU只读交互的账号级节流;先前offer/media阶段函数保持相同输入输出。 + +- [ ] 用受控Future/假时钟写慢模型期间第二任务抓取可推进、active=1时明确串行、同账号导航峰值1、不同资源峰值<=设置、429 Retry-After/重试计RPM、等待不占其他槽、轮转公平测试。 +- [ ] 写15分钟墙钟/详情30/HTTP60预算延后、关机恢复只补未完成或过期步骤、取消/改任务后模型迟到不提交、降低并发先排空、高压力暂停派发不丢工作。 +- [ ] 写scheduler修改任务interval后触发器更新、禁用队列任务不再执行、manual重新分析走同一资源池的失败用例,运行确认红。 +- [ ] 实现单进程协调器与按阶段线程所有权;去掉整任务完成后固定300秒等待,保留平台速率/风险退避;APScheduler仅提交信号与not-before。 +- [ ] 确保业务网络调用不在写事务中;分阶段替换原TaskQueue而不双消费,app生命周期可靠关闭浏览器/线程/子进程,debug/reload不启两套scheduler。 +- [ ] 并发/任务/队列/爬虫全部回归并独立审查后提交 `feat: run crawling and analysis through bounded stages`。 + +### P3.3:独立通知投递与可信批次聚合 + +**Files:** 新增 `goodprice/execution/delivery.py`、`tests/test_delivery_lifecycle.py`;修改offer_pipeline、crawl_service的聚合/通知路径、通知历史API/模型。 + +**Interfaces:** `DeliveryService(store, notifiers)` 的 `deliver(event_id: int, now: datetime) -> str`、`retry(delivery_id: int, now: datetime) -> str`;返回 pending/sent/failed/uncertain/awaiting_revalidation/superseded/cancelled。通知发送前核验task版本与价格新鲜度。 + +- [ ] 写run完成后合法pending可发送、run超时后迟到计算不能生成事件、暂停/删除/改条件撤销待发送、证据过期延后、不重复成功渠道、崩溃unknown结果uncertain不得自动重发等失败用例。 +- [ ] 写人工重试也重新核验;确价恢复但实质证据未变复用原业务事件;同一批次只比较当前就绪/确价/门槛通过offer,未完成不算低分,“本批最优”含比较范围。 +- [ ] 新测试红后实现outbox与逐渠道状态,业务事件key排除纯观察时间;外部发送与记录间窗口如实uncertain,不宣称exactly-once。 +- [ ] 全通知/offer/执行回归、审查后提交 `feat: deliver verified offer events independently`。 + +### P3.4:跨平台设备检测、资源profile与运行保护 + +**Files:** 新增 `goodprice/resources.py`、`tests/test_resource_profiles.py`;修改config.py、.env.example、锁文件、设置一致性测试;新增 `goodprice/web/resource_routes.py`。 + +**Interfaces:** `detect_resources() -> ResourceSnapshot`(effective_cpu:float|None、memory_limit_bytes:int|None、available_memory_bytes:int|None、source:str、warnings:tuple[str,...]、detected_at);`recommend_profile(snapshot: ResourceSnapshot) -> dict`;`resource_profile(name: Literal['low','balanced']) -> dict` 仅返回并发字段;POST `/settings/resources/apply-profile` 验证后确认应用,GET `/api/resources/recommendation` 只读。 + +- [ ] mock Windows/macOS/Linux/cgroup v1/v2 CPU quota/cpuset/memory.max与current、无限制/文件缺失/权限异常/祖先更低quota;容器优先有效较小值,不把宿主资源当容器资源。 +- [ ] 检测样本要覆盖cgroup从 `/proc/self/cgroup` 和mountinfo定位到当前进程而不是只读根目录;可用余量考虑宿主available与容器limit-current的较小值,不因cpuset字符串稀疏漏算。 +- [ ] 断言 CPU>=4、limit>=6GiB、available>=3GiB 才建议balanced,否则low;未知来源low;建议GET不改配置/不跑benchmark/不上传机器ID。 +- [ ] 断言profile低占用(2,1,1,1,1)、均衡(2,1,2,1,1),不修改媒体数量、视频开关、平台间隔和RPM;无确认POST不应用。 +- [ ] 测试红后加入psutil正式依赖并同步两锁文件;实现能力可降级检测与运行时仅降档保护,保存值/当前有效值/原因分开显示。 +- [ ] 设备/配置/执行回归及审查后提交 `feat: recommend explicit cross-platform resource budgets`。 + +### P3.5:列表上下文与设置分类保存 + +**Files:** 新增 `goodprice/web/list_context.py`、`settings_sections.py`、`tests/test_list_context.py`、`tests/test_settings_sections.py`;修改routes.py、settings_service.py、列表/详情/设置模板,新增轻量静态交互文件。 + +**Interfaces:** `ListContext.from_query(query) -> ListContext`、`ListContext.url(path: str = '/listings') -> str`、`safe_return_to(raw: str | None, fallback: str) -> str`;`SettingsSection` 白名单仅account/models/notifications/resources/media/deployment_readonly;`save_section(section: str, form: Mapping[str,str]) -> Settings` 在单事务校验合并后保存。 + +- [ ] 写任务/状态/价格状态/排序/加载范围roundtrip;非法/重复/超长参数规范化;外域、协议相对、编码换行/双编码return_to回标准列表,不用任意Referer。 + URL边界的最小断言: + ```python + assert safe_return_to("//example.com/listings", "/listings") == "/listings" + assert safe_return_to("/listings?task_id=7&show=active", "/listings") == "/listings?task_id=7&show=active" + ``` +- [ ] 写仅保存media不改变notification复选框;本分类未勾才关闭;秘密空值保持、显式清除有确认;数据库/容器运行模式等部署字段拒绝Web修改;无效值不提交半套配置。 +- [ ] 新用例红后实现规范URL链接贯通卡片、详情返回及所有操作重定向;分类用锚点/轻量tab保持当前分类,服务端白名单控制写入,不再遍历全部Settings做公共表单。 +- [ ] 分类支持unsaved提示、失败保留非秘密输入,清楚显示新增预算的代价和生效范围;沿用现有CSS/HTMX,无前端设计技能。 +- [ ] API/设置/所有页面渲染回归、审查后提交 `fix: retain listing context and isolate settings sections`。 + +### P3.6:全站功能交互回归 + +**Files:** 新增 `tests/e2e/test_navigation.py`、`test_settings.py`、`test_task_feedback.py`、`tests/e2e/conftest.py`;修改列表/卡片/详情/任务/通知/仪表盘/base模板与必要API。UI功能测试沿用Playwright,不用视觉重设计技能。 + +**Interfaces:** E2E fixture 只启动显式临时Settings、禁用真实scheduler/通知/模型、使用动态未占用回环端口,浏览器退出后清理;sessionStorage按规范查询key保存加载范围与锚点,秘密禁止保存。 + +- [ ] 先写失败流程:筛选→加载更多→详情→页面返回/原生后退→刷新,断言task/status/sort/已加载卡片范围保留;随后删除/拉黑/重评返回相同上下文,被删锚点恢复到相近项。 +- [ ] 测试轮询不清选中项/正在编辑表单/当前滚动;加载更多稳定排序同分用ID,卡片无重复;交互时暂停刷新并给“有新结果,点击更新”。 +- [ ] 测试任务重复运行去重/暂停反馈、通知pending/uncertain状态、失败HTTP不会伪成功、全选只针对已加载项、设置分类返回与unsaved提示。 +- [ ] 各主要页面测试窄屏无横向溢出、键盘可达、禁JS核心GET筛选/POST表单可用;实现最小模板/JS修复,保留风格。 +- [ ] 浏览器回归通过后审查并提交 `fix: preserve navigation and improve simple page interactions`。 + +### P3.7:全项目审计、性能对比与隔离部署验收 + +**Files:** 扩展 `scripts/benchmark_pipeline.py`、`tests/test_pipeline_regressions.py`、`.github/workflows/ci.yml`;修改README、Dockerfile/compose必要参数化、相关发现的具体模块。新增 `docs/validation/variant-media-upgrade.md` 保存脱敏方法/结果,不提交私人机器信息。 + +**Interfaces:** benchmark `--dataset synthetic-30 --rounds 5 --mode cold|warm --output `,输出JSON包含浏览器启动数、SQL、模型HTTP、阶段中位/P95、资源峰值、处理/待处理/失败数量;必须比较同输入预算和相同业务判定结果。 + +- [ ] 每项新发现bug先最小失败测试;重点旧库/启动副作用/后台重评绕过队列/interval更新/下架误判/长事务/N+1/资源泄漏/配置commit前校验,不能仅重构不证明症状。 +- [ ] 运行冷/热两轮对照基准,保证缓存轮不重复相同媒体模型调用、浏览器不逐商品重启;报告无明显回归或如实解释开销,不制造任意提速百分比。 +- [ ] CI按Windows/Linux配置单测与浏览器功能验证;requirements.lock用jev extra重新导出与uv.lock比较,镜像SDK/FFmpeg可用;不在CI中推镜像,release的测试门槛与main保持一致。 +- [ ] 运行 `uv lock --check`、`uv run --locked ruff check .`、`uv run --locked pytest -q --cov --cov-fail-under=80` 和E2E;迁移临时旧库两遍、还原备份演练通过。 +- [ ] Docker以唯一测试镜像/容器名、独立临时卷和空通知配置构建启动;检查healthz、表单、SDK/FFmpeg、视频开关、noVNC开/关,绝不使用生产compose up替换正在运行的服务。 +- [ ] 整分支独立代码审查、修复并完整复验,审查真实样本范围、缓存/预算、生命周期、API/旧库兼容。报告未验证macOS/ARM/平台样本限制,不冒充全部完成。 +- [ ] 提交 `test: validate upgrade performance and deployment compatibility`,填写总计划完成记录;用户选择集成前保持独立分支,不自动push/发版。 diff --git a/docs/superpowers/plans/2026-10-26-upgrade-roadmap.md b/docs/superpowers/plans/2026-10-26-upgrade-roadmap.md new file mode 100644 index 0000000..a11bfb1 --- /dev/null +++ b/docs/superpowers/plans/2026-10-26-upgrade-roadmap.md @@ -0,0 +1,83 @@ +# 规格价格、媒体与流水线升级实施总计划 + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 将单标价图片盯价工具升级为按规格、可信价格和媒体证据判断、可配置资源且保持简洁交互的跨平台应用。 + +**Architecture:** 保留单进程 FastAPI/SQLite/Playwright 与服务端模板。分三个可独立测试的子计划实施,先完成纯领域裁决与迁移,再加入媒体,最后整合有界流水线与全站交互;不引入 Redis/Celery/SPA。 + +**Tech Stack:** Python >=3.11、FastAPI、SQLAlchemy/SQLite、Playwright、Jinja2/HTMX、pytest/Ruff;阶段二新增 Pillow、FFmpeg,阶段三新增 psutil。 + +**Spec:** `D:/good-price/docs/superpowers/specs/2026-10-26-variant-price-media-design.md`(已获用户批准)。 + +## Global Constraints + +- 不使用前端设计类技能;保留简洁的颜色、布局基础与服务端模板,只改善信息与交互行为。 +- Windows/macOS/Linux/NAS/Docker 均不以操作系统专属 shell/fork 为前提;Windows 本地和 Linux Docker 实测,其余未测明示。 +- 不读取/提交真实 Cookie、密钥、私人文档,不挂载生产 data/ 进行测试,不发送真实通知,不自动触发付费模型调用。 +- 不更改/删除本地忽略的 `.cmd`、PowerShell 启动器,不自动 push main/release、创建 Tag 或发布镜像。 +- 价格未知不视为零,不按起售价排名,不自动联系卖家;必要证据过期暂停正常推荐。 +- 待询价外部通知默认关闭,人工报价默认 7 天;视频默认关闭;旧安装总视觉输入保留 4,新安装为 8(混合 4+4)。 +- 低占用首次默认:active runs=2、抓取/模型/媒体 worker 各 1、FFmpeg 线程=1;硬件建议需确认才能应用。 +- 不默默放宽访问和资源预算;导航间隔默认 10 秒(5–300),模型 RPM 默认 10(1–60),并发和大小有服务端硬上限。 +- 默认每轮详情 30、模型 HTTP 尝试 60、墙钟 15 分钟;默认工作项全局 200/每任务 60;取自设计 §6.1。 +- 在新架构完成前保留阶段一/二已通过的交付;独立阶段提交可审查,但不把部分功能误标为整轮完成或正式稳定版。 +- Ruff、锁文件一致性、全套测试与 >=80% 覆盖率、迁移、隔离 Docker 和独立审查均是交付门槛,不靠覆盖率取代行为用例。 + +## Review Focus + +1. import app/pytest 收集意外初始化真实数据库:P1.1 用子进程与拒绝路径访问验证。 +2. 价格来源切换伪造降价、任务目标变更沿用报价:P1.3/P1.4 以 observation/quote 与任务版本测试。 +3. 临时媒体文件绕过磁盘预算、模型输出引用不存在证据:P2.2/P2.4 分别覆盖。 +4. 暂停/到期之后迟到 worker 仍写入或发通知:P3.1/P3.2/P3.3 使用假时钟和受控 Future 验证。 +5. 设置分类遗漏复选框清掉其他分类、return_to 开放重定向:P3.5 做 API 和真实浏览器用例。 + +## 计划拆分与执行次序 + +| 顺序 | 子计划 | 交付 | +| --- | --- | --- | +| 1 | `2026-10-26-phase1-offer-pricing-plan.md` | 测试隔离、规格/可信价/人工报价、旧库迁移、现有流水线接入 | +| 2 | `2026-10-26-phase2-media-evidence-plan.md` | 可配置媒体预算、安全下载/抽帧、证据分析与缓存 | +| 3 | `2026-10-26-phase3-pipeline-interaction-plan.md` | 受控流水线、设备建议、全站交互、综合性能和缺陷修复 | + +每个子计划开始前先阅读本总计划和 Spec;后续计划通过下列契约衔接,不平行改同一大文件。 + +## 共享接口与文件边界 + +- `goodprice/offers/types.py`:不可变 PriceClaim/OfferCandidate/PriceDecision/QuoteInput/OfferEvaluation,不能携带 ORM session。 +- `goodprice/offers/pricing.py`:金额、价格状态、有效来源与新鲜度裁决;`matching.py`:目标组合匹配与稳定身份。 +- `goodprice/offers/service.py`:短事务保存报价、观察与评估;业务规则调用纯函数,模板不重复判断。 +- `goodprice/media/types.py`:MediaCandidate/MediaEvidence/MediaSelection;`budget.py`:输入与磁盘预算,`download.py`:远端边界,`frames.py`:受限本地解码。 +- `goodprice/execution/types.py`:Stage/RunToken/WorkResult;`store.py`:持久化状态;`limits.py`:并发/限流;`coordinator.py`:唯一派发与写入协调。 +- `goodprice/web/list_context.py`:规范化列表状态;`settings_sections.py`:可写分类白名单;拆分新增路由文件,不把所有逻辑继续堆到 routes.py。 +- `goodprice/resources.py`:只读跨平台设备检测、建议与可见有效限制;不代替平台/模型限流。 +- 继续使用原 `models.py`、`db.py` 注册新表与迁移入口。领域和执行逻辑放独立包,避免循环导入。 +- `CrawlService` 阶段一/二是迁移适配层;阶段三接到协调器,最终不允许旧队列与新协调器双消费。 + +## 安全执行与基线 + +- [ ] 实施前读取 using-git-worktrees 技能,使用原生 managed worktree 从已批准设计提交建隔离 checkout;不复制 .env 或 data/。 +- [ ] 检查远程 main 的新变化与本地设计基线差异,仅 fetch 不自动合并社区新代码;必要冲突先解释。 +- [ ] 先执行 P1.1,之后才运行导入 `goodprice.main` 的测试或启动服务。 +- [ ] 基准使用合成页面、媒体、假模型和假时钟,保留请求次数、SQL、浏览器启动次数、中位数/P95及资源峰值;不压测真实平台。 +- [ ] 每任务遵循红→绿→审查→修正→提交;每阶段独立审查并跑全量回归,发现超范围架构问题先更新设计。 + +## 交付记录 + +- [ ] 阶段一完成,记录 commit、命令、结果和真实 SKU 样本验证范围。 +- [ ] 阶段二完成,记录 Linux/Windows FFmpeg 测试、视频适配限制与全部预算行为。 +- [ ] 阶段三完成,记录队列峰值/重启回归、浏览器交互结果、性能对比和剩余问题。 +- [ ] 整分支独立审查与最终复验完成;仅此时总结完成范围,未验证真实平台能力保持实验性。 +- [ ] 用户选择集成/发版方式前,不 push main/release,不升级正在运行的生产容器或数据库。 + +## 当前状态 + +仅设计与实施计划已保存,本清单的实现项均未执行。下一步由用户审阅计划并选择执行方式。 + +## 自审覆盖记录 + +- Spec §4 的规格/金额/报价/来源/过期/迁移由 P1.2–P1.5 实现,§5 的媒体数量/代价/安全/缓存/证据由 P2.1–P2.5 实现。 +- Spec §6.1 的恢复/背压/限流/取消与通知由 P3.1–P3.3,§6.2 的设备/容器限额由 P3.4,§6.3 的全站交互由 P3.5–P3.6 实现。 +- Spec §7–8 的旧库/隔离部署/性能/跨平台验收由 P1.1/P1.2/P3.7 覆盖;私人文件与不发布限制全阶段适用。 +- 接口从 offers/types → media/types → execution/types 顺序定义;任何阶段需要改变前置契约,须同步消费者测试,而非另造平行实现。 +- 默认预算及用户确认边界均引用批准设计;未取得真实多规格/视频样本时,明确保留实验性限制,不视作适配验收通过。 diff --git a/docs/superpowers/specs/2026-10-26-variant-price-media-design.md b/docs/superpowers/specs/2026-10-26-variant-price-media-design.md index c0e2a42..ebc5f5f 100644 --- a/docs/superpowers/specs/2026-10-26-variant-price-media-design.md +++ b/docs/superpowers/specs/2026-10-26-variant-price-media-design.md @@ -5,7 +5,7 @@ 基线:`f728ff8` / v0.1.1 版本:修订版 2(追加资源配置、流水线与全站交互要求) -状态:设计待用户批准;尚未开始实现。 +状态:用户已批准修订设计;正在编写实施计划,尚未开始实现。 ## 1. 目标与已确认范围 @@ -418,4 +418,4 @@ macOS 及其他架构未实际运行须标注验证边界,不能因“跨平 3. 跨轮恢复先校验过期依赖,内容不变复用模型结果,内容变化才重做相关分析。 4. 旧安装保留原每请求 4 张预算,新安装默认 8,硬件建议不默默改变媒体用量。 5. 磁盘总预算包含临时文件,预留空间与背压确保配置的缓存上限有意义。 -以上补充均为助手提出的确定规则,随本文一并交用户批准,未开始实现。 +以上补充均为助手提出的确定规则;用户已在本次对话确认修订设计,未开始实现。 From 6e686bf82522d99d0db037ec872e240cce8cfbab Mon Sep 17 00:00:00 2001 From: Comui520 Date: Mon, 26 Oct 2026 10:53:32 +0800 Subject: [PATCH 04/17] test: isolate app startup and establish pipeline baseline --- .env.example | 2 + goodprice/config.py | 1 + goodprice/main.py | 15 +++--- scripts/benchmark_pipeline.py | 93 ++++++++++++++++++++++++++++++++++ tests/conftest.py | 3 +- tests/test_import_isolation.py | 49 ++++++++++++++++++ 6 files changed, 156 insertions(+), 7 deletions(-) create mode 100644 scripts/benchmark_pipeline.py create mode 100644 tests/test_import_isolation.py diff --git a/.env.example b/.env.example index c25efc0..99788df 100644 --- a/.env.example +++ b/.env.example @@ -63,6 +63,8 @@ SECOND_EYE_IMAGE=second-eye:local # SQLite 数据库文件路径(Docker 部署时会被 compose.yml 的 environment 覆盖为 # sqlite:////app/data/goodprice.db,容器内改这里不生效) DATABASE_URL=sqlite:///data/goodprice.db +# 日志与后续媒体缓存目录;测试/独立部署可指定其他目录。 +DATA_ROOT=data # Docker noVNC:默认关闭。临时登录时设置 ENABLE_NOVNC=1,并在 data/.novnc-password # 写入独立密码;登录完成后恢复为 0 并重启容器。 diff --git a/goodprice/config.py b/goodprice/config.py index feb741a..cb195c1 100644 --- a/goodprice/config.py +++ b/goodprice/config.py @@ -40,6 +40,7 @@ class Settings(BaseSettings): ) app_name: str = "闲鱼盯价助手" + data_root: Path = PROJECT_ROOT / "data" database_url: str = f"sqlite:///{DEFAULT_DB.as_posix()}" xianyu_cookie: str = "" llm_base_url: str = "" diff --git a/goodprice/main.py b/goodprice/main.py index 7f2bc09..615f47e 100644 --- a/goodprice/main.py +++ b/goodprice/main.py @@ -21,8 +21,7 @@ LOG_DIR = Path(__file__).resolve().parent.parent / "data" / "logs" -def _setup_logging() -> None: - LOG_DIR.mkdir(parents=True, exist_ok=True) +def _setup_logging(settings: Optional[Settings] = None) -> None: formatter = logging.Formatter("%(asctime)s %(levelname)s %(name)s: %(message)s") root = logging.getLogger() root.setLevel(logging.INFO) @@ -32,8 +31,10 @@ def _setup_logging() -> None: console.setFormatter(formatter) root.addHandler(console) if not os.environ.get("PYTEST_CURRENT_TEST"): + log_dir = (settings.data_root / "logs") if settings else LOG_DIR + log_dir.mkdir(parents=True, exist_ok=True) file_handler = RotatingFileHandler( - LOG_DIR / "app.log", + log_dir / "app.log", maxBytes=5 * 1024 * 1024, backupCount=3, encoding="utf-8", @@ -42,7 +43,6 @@ def _setup_logging() -> None: root.addHandler(file_handler) -_setup_logging() logger = logging.getLogger(__name__) @@ -101,6 +101,7 @@ def build_app( with_scheduler: bool = True, ) -> FastAPI: settings = settings or get_settings() + _setup_logging(settings) if session_factory is None: init_db(settings.database_url) from goodprice.db import make_session_factory @@ -171,8 +172,10 @@ async def lifespan(app: FastAPI): return app -app = build_app() +def create_app() -> FastAPI: + """ASGI factory: importing the module never opens a database or starts work.""" + return build_app() def main() -> None: - uvicorn.run("goodprice.main:app", host="0.0.0.0", port=8000, reload=False) + uvicorn.run("goodprice.main:create_app", factory=True, host="0.0.0.0", port=8000, reload=False) diff --git a/scripts/benchmark_pipeline.py b/scripts/benchmark_pipeline.py new file mode 100644 index 0000000..1c82d6c --- /dev/null +++ b/scripts/benchmark_pipeline.py @@ -0,0 +1,93 @@ +"""Offline, synthetic baseline; never constructs real network clients or uses .env.""" +import argparse +import json +import socket +import statistics +import sys +import tempfile +import time +import tracemalloc +from pathlib import Path +from types import SimpleNamespace + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from sqlalchemy import event # noqa: E402 + +from goodprice.config import Settings # noqa: E402 +from goodprice.crawler.base import ListingData, ListingDetail # noqa: E402 +from goodprice.db import Base, make_session_factory # noqa: E402 +from goodprice.services.crawl_service import CrawlService # noqa: E402 +from goodprice.services.settings_service import SettingsService # noqa: E402 +from goodprice.services.task_service import TaskService # noqa: E402 + + +class SyntheticAdapter: + def __init__(self): + self.detail_calls = 0 + + def search(self, keyword): + return [ListingData(str(i), f"synthetic {i}", 100, f"https://invalid.test/{i}") + for i in range(30)] + + def fetch_detail(self, url): + self.detail_calls += 1 + return ListingDetail(description="Synthetic offline description") + + +def run(rounds, mode): + # Hard fail if future pipeline changes accidentally attempt real network I/O. + def deny_network(*args, **kwargs): + raise AssertionError("External network prohibited in benchmark") + + original = socket.socket.connect + socket.socket.connect = deny_network + results = [] + try: + with tempfile.TemporaryDirectory(prefix="second-eye-benchmark-") as folder: + for index in range(rounds): + factory = make_session_factory(f"sqlite:///{Path(folder).as_posix()}/{index}.db") + engine = factory.kw["bind"] + Base.metadata.create_all(engine) + settings = Settings(_env_file=None, database_url=str(engine.url), + default_crawl_jitter_minutes=0, vision_enabled=False) + task = TaskService(factory).create_task({"keyword": "synthetic"}) + adapter = SyntheticAdapter() + service = CrawlService(factory, adapter, SimpleNamespace(enabled=False), + SimpleNamespace(enabled=False), [], + SettingsService(factory, settings)) + if mode == "warm": + service.run_task(task.id) + queries = [] + event.listen(engine, "before_cursor_execute", lambda *args, q=queries: q.append(1)) + tracemalloc.start() + start = time.perf_counter() + stats = service.run_task(task.id) + elapsed = time.perf_counter() - start + _, peak = tracemalloc.get_traced_memory() + tracemalloc.stop() + results.append({**stats, "seconds": elapsed, "sql_count": len(queries), + "python_peak_bytes": peak, "detail_calls": adapter.detail_calls, + "model_http_attempts": 0, "browser_starts": 0}) + engine.dispose() + finally: + socket.socket.connect = original + times = sorted(row["seconds"] for row in results) + return {"dataset": "synthetic-30", "mode": mode, "results": results, + "external_network_requests": 0, "median_seconds": statistics.median(times), + "p95_seconds": times[min(len(times) - 1, int(len(times) * .95))], + "limits": "No real browser or model latency is represented; no real-platform speed claim."} + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--dataset", choices=["synthetic-30"], default="synthetic-30") + parser.add_argument("--rounds", type=int, default=5) + parser.add_argument("--mode", choices=["cold", "warm"], default="cold") + parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args() + if not 1 <= args.rounds <= 100: + parser.error("rounds must be between 1 and 100") + args.output.mkdir(parents=True, exist_ok=True) + (args.output / "benchmark.json").write_text( + json.dumps(run(args.rounds, args.mode), ensure_ascii=False, indent=2), encoding="utf-8") diff --git a/tests/conftest.py b/tests/conftest.py index 41761df..de65400 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -22,9 +22,10 @@ def session_factory(tmp_db): @pytest.fixture -def base_settings(tmp_db): +def base_settings(tmp_db, tmp_path): return Settings( database_url=tmp_db, + data_root=tmp_path / "data", _env_file=None, default_crawl_interval_minutes=20, default_crawl_jitter_minutes=0, diff --git a/tests/test_import_isolation.py b/tests/test_import_isolation.py new file mode 100644 index 0000000..71e2cf8 --- /dev/null +++ b/tests/test_import_isolation.py @@ -0,0 +1,49 @@ +import os +import subprocess +import sys +from pathlib import Path + + +def test_import_does_not_initialize_db_or_write_logs(tmp_path): + root = Path(__file__).resolve().parents[1] + source = ''' +from pathlib import Path +import sqlalchemy +def forbidden(*args, **kwargs): + raise AssertionError("import opened a database") +sqlalchemy.create_engine = forbidden +import goodprice.main +import goodprice.__main__ +assert callable(goodprice.main.create_app) +assert not Path("data").exists() +''' + env = {**os.environ, "PYTHONPATH": str(root), "DATABASE_URL": "sqlite:///:memory:"} + result = subprocess.run( + [sys.executable, "-c", source], cwd=tmp_path, env=env, + capture_output=True, text=True, timeout=20, + ) + assert result.returncode == 0, result.stdout + result.stderr + + +def test_explicit_app_uses_only_temporary_store(base_settings, session_factory): + from goodprice.main import build_app + + app = build_app(settings=base_settings, session_factory=session_factory, with_scheduler=False) + assert app.state.session_factory is session_factory + assert app.state.settings_service.get().database_url == base_settings.database_url + + +def test_benchmark_is_offline_and_uses_temporary_database(tmp_path): + import json + + root = Path(__file__).resolve().parents[1] + result = subprocess.run( + [sys.executable, str(root / "scripts/benchmark_pipeline.py"), + "--rounds", "1", "--output", str(tmp_path)], + cwd=root, capture_output=True, text=True, timeout=30, + ) + assert result.returncode == 0, result.stderr + report = json.loads((tmp_path / "benchmark.json").read_text(encoding="utf-8")) + assert report["dataset"] == "synthetic-30" + assert report["results"][0]["found"] == 30 + assert report["external_network_requests"] == 0 From e9b7036dbf9dcd26ba2d27576fba82fd2b9d3620 Mon Sep 17 00:00:00 2001 From: Comui520 Date: Mon, 26 Oct 2026 10:56:52 +0800 Subject: [PATCH 05/17] feat: add versioned offer and price evidence storage --- goodprice/db.py | 22 ++++++++- goodprice/migrations.py | 22 +++++++++ goodprice/models.py | 56 +++++++++++++++++++++++ goodprice/offers/__init__.py | 1 + goodprice/offers/matching.py | 50 ++++++++++++++++++++ goodprice/offers/types.py | 62 +++++++++++++++++++++++++ goodprice/services/task_service.py | 13 +++++- goodprice/web/routes.py | 2 + tests/test_offer_identity.py | 24 ++++++++++ tests/test_offer_migrations.py | 73 ++++++++++++++++++++++++++++++ 10 files changed, 322 insertions(+), 3 deletions(-) create mode 100644 goodprice/migrations.py create mode 100644 goodprice/offers/__init__.py create mode 100644 goodprice/offers/matching.py create mode 100644 goodprice/offers/types.py create mode 100644 tests/test_offer_identity.py create mode 100644 tests/test_offer_migrations.py diff --git a/goodprice/db.py b/goodprice/db.py index 385a3c3..529ff28 100644 --- a/goodprice/db.py +++ b/goodprice/db.py @@ -1,6 +1,6 @@ from pathlib import Path -from sqlalchemy import create_engine, text +from sqlalchemy import create_engine, inspect, text from sqlalchemy.orm import DeclarativeBase, sessionmaker _LISTINGS_DDL = """ @@ -66,7 +66,17 @@ def init_db(database_url: str) -> None: from goodprice import models # noqa: F401 确保模型注册 factory = make_session_factory(database_url) - Base.metadata.create_all(factory().get_bind()) + engine = factory.kw["bind"] + legacy = inspect(engine).has_table("watch_tasks") + Base.metadata.create_all(engine) + # Record installation type before additive table creation hides its provenance. + from goodprice.models import SchemaVersion + + with factory() as session: + if session.get(SchemaVersion, 1) is None: + session.add(SchemaVersion(id=1, version=0, legacy_install=legacy)) + session.commit() + engine.dispose() def migrate_schema(session_factory) -> None: @@ -76,6 +86,9 @@ def migrate_schema(session_factory) -> None: ("fetch_detail", "fetch_detail BOOLEAN DEFAULT 1"), ("min_price", "min_price FLOAT DEFAULT 0"), ("exclude_words", "exclude_words TEXT"), + ("target_variant", "target_variant TEXT DEFAULT ''"), + ("notify_pending_price", "notify_pending_price BOOLEAN DEFAULT 0"), + ("version", "version INTEGER DEFAULT 1"), ], "listings": [ ("description", "description TEXT"), @@ -100,6 +113,8 @@ def migrate_schema(session_factory) -> None: ("blocked", "blocked BOOLEAN DEFAULT 0"), ], "notifications": [ + ("offer_id", "offer_id INTEGER"), + ("event_key", "event_key VARCHAR(128)"), ("title", "title TEXT"), ("content", "content TEXT"), ], @@ -120,6 +135,9 @@ def migrate_schema(session_factory) -> None: session.execute(text(f"ALTER TABLE {table} ADD COLUMN {ddl}")) if "listings" in existing_tables and _listing_unique_columns(session) != _NEW_LISTING_UNIQUE: _rebuild_listings(session) + from goodprice.migrations import migrate_offer_schema + + migrate_offer_schema(session) session.commit() diff --git a/goodprice/migrations.py b/goodprice/migrations.py new file mode 100644 index 0000000..71f542b --- /dev/null +++ b/goodprice/migrations.py @@ -0,0 +1,22 @@ +"""Additive offer migration; the caller owns the transaction.""" +from sqlalchemy import inspect + +from goodprice.models import ( + ListingOffer, + ManualQuote, + OfferEvaluationRecord, + OfferObservation, + SchemaVersion, +) + + +def migrate_offer_schema(session) -> None: + connection = session.connection() + tables = set(inspect(connection).get_table_names()) + legacy = "schema_versions" not in tables and "watch_tasks" in tables + for model in (SchemaVersion, ListingOffer, OfferObservation, ManualQuote, OfferEvaluationRecord): + model.__table__.create(connection, checkfirst=True) + if session.get(SchemaVersion, 1) is None: + session.add(SchemaVersion(id=1, version=1, legacy_install=legacy)) + else: + session.get(SchemaVersion, 1).version = 1 diff --git a/goodprice/models.py b/goodprice/models.py index cedc986..da68933 100644 --- a/goodprice/models.py +++ b/goodprice/models.py @@ -36,6 +36,9 @@ class WatchTask(Base): interval_minutes: Mapped[int] = mapped_column(Integer, default=20) enabled: Mapped[bool] = mapped_column(default=True) fetch_detail: Mapped[bool] = mapped_column(default=True) + target_variant: Mapped[str] = mapped_column(Text, default="") + notify_pending_price: Mapped[bool] = mapped_column(default=False) + version: Mapped[int] = mapped_column(Integer, default=1) created_at: Mapped[datetime] = mapped_column(DateTime, default=_now) last_run_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True) last_error: Mapped[Optional[str]] = mapped_column(Text, nullable=True) @@ -88,6 +91,7 @@ class Listing(Base): notifications: Mapped[list["Notification"]] = relationship( back_populates="listing", cascade="all, delete-orphan" ) + offers: Mapped[list["ListingOffer"]] = relationship(cascade="all, delete-orphan") class PriceSnapshot(Base): @@ -105,6 +109,8 @@ class Notification(Base): __tablename__ = "notifications" id: Mapped[int] = mapped_column(primary_key=True) + offer_id: Mapped[Optional[int]] = mapped_column(Integer, nullable=True) + event_key: Mapped[Optional[str]] = mapped_column(String(128), nullable=True, index=True) listing_id: Mapped[Optional[int]] = mapped_column( ForeignKey("listings.id", ondelete="SET NULL"), nullable=True ) @@ -128,6 +134,56 @@ class AppSetting(Base): value: Mapped[str] = mapped_column(Text, default="") +class SchemaVersion(Base): + __tablename__ = "schema_versions" + id: Mapped[int] = mapped_column(primary_key=True) + version: Mapped[int] = mapped_column(Integer, default=1) + legacy_install: Mapped[bool] = mapped_column(Boolean, default=False) + + +class ListingOffer(Base): + __tablename__ = "listing_offers" + __table_args__ = (UniqueConstraint("listing_id", "offer_key", name="uq_listing_offer"),) + id: Mapped[int] = mapped_column(primary_key=True) + listing_id: Mapped[int] = mapped_column(ForeignKey("listings.id"), index=True) + offer_key: Mapped[str] = mapped_column(String(200)) + name: Mapped[str] = mapped_column(Text, default="") + attributes: Mapped[dict] = mapped_column(JSON, default=dict) + availability: Mapped[str] = mapped_column(String(20), default="unknown") + observations: Mapped[list["OfferObservation"]] = relationship(cascade="all, delete-orphan") + quotes: Mapped[list["ManualQuote"]] = relationship(cascade="all, delete-orphan") + evaluations: Mapped[list["OfferEvaluationRecord"]] = relationship(cascade="all, delete-orphan") + + +class OfferObservation(Base): + __tablename__ = "offer_observations" + id: Mapped[int] = mapped_column(primary_key=True) + offer_id: Mapped[int] = mapped_column(ForeignKey("listing_offers.id"), index=True) + fingerprint: Mapped[str] = mapped_column(String(64)) + payload: Mapped[dict] = mapped_column(JSON, default=dict) + observed_at: Mapped[datetime] = mapped_column(DateTime, default=_now) + + +class ManualQuote(Base): + __tablename__ = "manual_quotes" + id: Mapped[int] = mapped_column(primary_key=True) + offer_id: Mapped[int] = mapped_column(ForeignKey("listing_offers.id"), index=True) + amount_cents: Mapped[int] = mapped_column(Integer) + confirmed_at: Mapped[datetime] = mapped_column(DateTime) + expires_at: Mapped[datetime] = mapped_column(DateTime) + evidence_fingerprint: Mapped[str] = mapped_column(String(64)) + revoked: Mapped[bool] = mapped_column(Boolean, default=False) + + +class OfferEvaluationRecord(Base): + __tablename__ = "offer_evaluations" + id: Mapped[int] = mapped_column(primary_key=True) + offer_id: Mapped[int] = mapped_column(ForeignKey("listing_offers.id"), index=True) + task_version: Mapped[int] = mapped_column(Integer) + payload: Mapped[dict] = mapped_column(JSON, default=dict) + evaluated_at: Mapped[datetime] = mapped_column(DateTime, default=_now) + + class Seller(Base): __tablename__ = "sellers" __table_args__ = ( diff --git a/goodprice/offers/__init__.py b/goodprice/offers/__init__.py new file mode 100644 index 0000000..a5dfebc --- /dev/null +++ b/goodprice/offers/__init__.py @@ -0,0 +1 @@ +"""Offer identity, observed prices and buyer-confirmed quotes.""" diff --git a/goodprice/offers/matching.py b/goodprice/offers/matching.py new file mode 100644 index 0000000..55872f2 --- /dev/null +++ b/goodprice/offers/matching.py @@ -0,0 +1,50 @@ +import hashlib +import json +import re +import unicodedata +from typing import Mapping + +from goodprice.offers.types import MatchState + + +def _normal(value: str) -> str: + return re.sub(r"\s+", "", unicodedata.normalize("NFKC", value).casefold()) + + +def stable_offer_key(platform_sku_id: str | None, attributes: Mapping[str, str]) -> str | None: + if platform_sku_id: + return f"sku:{platform_sku_id.strip()}" + normalized = {_normal(k): _normal(v) for k, v in attributes.items() if k.strip() and v.strip()} + if not normalized: + return None + raw = json.dumps(normalized, sort_keys=True, ensure_ascii=False).encode() + return "attrs:" + hashlib.sha256(raw).hexdigest()[:32] + + +def match_variant(target: str, attributes: Mapping[str, str], title: str) -> MatchState: + """Only attest explicit attributes, not broad listing titles or guessed negatives.""" + if not target.strip(): + return "matched" + values = {_normal(v) for v in attributes.values()} + if not values: + return "uncertain" + uncertain = False + for requirement in re.split(r"[/,,;;]+", target): + term = _normal(requirement) + if not term: + continue + if term.startswith(("非", "不要", "不含", "无")): + uncertain = True + elif term in values: + continue + elif re.fullmatch(r"\d+(?:\.\d+)?(?:gb|tb)", term): + if any(re.fullmatch(r"\d+(?:\.\d+)?(?:gb|tb)", v) for v in values): + return "mismatched" + uncertain = True + elif any(k in ("型号", "model") for k in attributes): + model = next((v for k, v in attributes.items() if k in ("型号", "model")), "") + if _normal(model) != term: + return "mismatched" + else: + uncertain = True + return "uncertain" if uncertain else "matched" diff --git a/goodprice/offers/types.py b/goodprice/offers/types.py new file mode 100644 index 0000000..0ba37c9 --- /dev/null +++ b/goodprice/offers/types.py @@ -0,0 +1,62 @@ +from dataclasses import dataclass, field +from datetime import datetime +from types import MappingProxyType +from typing import Literal, Mapping + +PriceState = Literal["exact", "range_or_from", "inquiry", "conflict", "unknown"] +MatchState = Literal["matched", "mismatched", "uncertain"] + + +@dataclass(frozen=True) +class PriceClaim: + source: str + kind: PriceState + text: str + observed_at: datetime + evidence_id: str + amount_cents: int | None = None + min_cents: int | None = None + max_cents: int | None = None + scope_key: str | None = None + + +@dataclass(frozen=True) +class OfferCandidate: + key: str + attributes: Mapping[str, str] = field(default_factory=dict) + availability: Literal["available", "unavailable", "unknown"] = "unknown" + claims: tuple[PriceClaim, ...] = () + name: str = "" + + def __post_init__(self): + object.__setattr__(self, "attributes", MappingProxyType(dict(self.attributes))) + object.__setattr__(self, "claims", tuple(self.claims)) + + +@dataclass(frozen=True) +class QuoteInput: + offer_key: str + amount_cents: int + confirmed_at: datetime + expires_at: datetime + evidence_fingerprint: str + revoked: bool = False + + +@dataclass(frozen=True) +class PriceDecision: + state: PriceState + amount_cents: int | None = None + source: str | None = None + evidence_ids: tuple[str, ...] = () + reason: str = "" + fresh_until: datetime | None = None + + +@dataclass(frozen=True) +class OfferEvaluation: + offer_key: str + match: MatchState + price: PriceDecision + eligible: bool + reason: str = "" diff --git a/goodprice/services/task_service.py b/goodprice/services/task_service.py index 84c3369..f7e302d 100644 --- a/goodprice/services/task_service.py +++ b/goodprice/services/task_service.py @@ -4,7 +4,7 @@ from typing import Optional from goodprice.constants import SCORE_MAX -from goodprice.models import WatchTask +from goodprice.models import Listing, Notification, WatchTask def normalize_task_data(data: dict) -> dict: @@ -48,6 +48,8 @@ def number(name: str) -> float: "platform": str(data.get("platform") or "xianyu").strip() or "xianyu", "interval_minutes": interval_minutes, "fetch_detail": bool(data.get("fetch_detail", True)), + "target_variant": str(data.get("target_variant") or "").strip(), + "notify_pending_price": bool(data.get("notify_pending_price", False)), "enabled": bool(data.get("enabled", True)), } @@ -64,6 +66,8 @@ def number(name: str) -> float: "platform", "interval_minutes", "fetch_detail", + "target_variant", + "notify_pending_price", "enabled", ) @@ -108,6 +112,7 @@ def toggle_task(self, task_id: int) -> Optional[WatchTask]: task = session.get(WatchTask, task_id) if task: task.enabled = not task.enabled + task.version += 1 session.commit() session.refresh(task) return task @@ -117,6 +122,8 @@ def delete_task(self, task_id: int) -> bool: task = session.get(WatchTask, task_id) if not task: return False + session.query(Listing).filter_by(task_id=task_id).update({"task_id": None}) + session.query(Notification).filter_by(task_id=task_id).update({"task_id": None}) session.delete(task) session.commit() return True @@ -133,6 +140,10 @@ def update_task(self, task_id: int, data: dict) -> Optional[WatchTask]: values = normalize_task_data( {**{field: getattr(task, field) for field in TASK_FIELDS}, **data} ) + for field in TASK_FIELDS: + if getattr(task, field) != values[field]: + task.version += 1 + break for field in TASK_FIELDS: setattr(task, field, values[field]) session.commit() diff --git a/goodprice/web/routes.py b/goodprice/web/routes.py index 942e9d2..591be51 100644 --- a/goodprice/web/routes.py +++ b/goodprice/web/routes.py @@ -58,6 +58,8 @@ def healthz(request: Request): class TaskCreate(BaseModel): + target_variant: str = "" + notify_pending_price: bool = False name: str = "" keyword: str = Field(min_length=1) max_price: float = Field(default=0, ge=0, allow_inf_nan=False) diff --git a/tests/test_offer_identity.py b/tests/test_offer_identity.py new file mode 100644 index 0000000..4c04c67 --- /dev/null +++ b/tests/test_offer_identity.py @@ -0,0 +1,24 @@ +from goodprice.offers.matching import match_variant, stable_offer_key + + +def test_stable_key_uses_sku_not_price_or_attribute_order(): + assert stable_offer_key("sku-1", {"容量": "256GB"}) == "sku:sku-1" + assert stable_offer_key(None, {"容量": "256 GB", "型号": "iPhone 15"}) == stable_offer_key( + None, {"型号": "iphone15", "容量": "256GB"}) + assert stable_offer_key(None, {}) is None + + +def test_capacity_and_model_are_not_substring_matches(): + attrs = {"型号": "iPhone 15", "容量": "256GB"} + assert match_variant("iPhone 15 / 256GB", attrs, "") == "matched" + assert match_variant("iPhone 15 / 128GB", attrs, "") == "mismatched" + assert match_variant("iPhone 15 / 256GB", {"型号": "iPhone 15 Pro", "容量": "256GB"}, "") == "mismatched" + + +def test_unknown_or_unproved_negative_is_not_matched(): + assert match_variant("非扩容", {"容量": "256GB"}, "好机器") == "uncertain" + assert match_variant("iPhone 15 / 256GB", {}, "iPhone 15 256GB") == "uncertain" + + +def test_empty_target_does_not_invent_restrictions(): + assert match_variant("", {"容量": "256GB"}, "") == "matched" diff --git a/tests/test_offer_migrations.py b/tests/test_offer_migrations.py new file mode 100644 index 0000000..664ea4c --- /dev/null +++ b/tests/test_offer_migrations.py @@ -0,0 +1,73 @@ +from sqlalchemy import inspect, text + +from goodprice.db import init_db, make_session_factory, migrate_schema +from goodprice.models import Listing, ListingOffer, OfferObservation, SchemaVersion +from goodprice.services.task_service import TaskService + + +def test_old_database_migration_is_idempotent_and_preserves_legacy_values(tmp_db): + factory = make_session_factory(tmp_db) + with factory.kw["bind"].begin() as conn: + conn.execute(text("CREATE TABLE watch_tasks (id INTEGER PRIMARY KEY, keyword TEXT)")) + conn.execute(text("INSERT INTO watch_tasks VALUES (1, 'iphone')")) + conn.execute(text("CREATE TABLE listings (id INTEGER PRIMARY KEY, external_id TEXT, " + "platform TEXT, title TEXT, price FLOAT, url TEXT)")) + conn.execute(text("INSERT INTO listings VALUES (2, 'external', 'xianyu', 'old', 99, 'url')")) + migrate_schema(factory) + migrate_schema(factory) + assert "listing_offers" in inspect(factory.kw["bind"]).get_table_names() + with factory() as session: + assert session.execute(text("SELECT keyword FROM watch_tasks")).scalar() == "iphone" + assert session.execute(text("SELECT external_id FROM listings")).scalar() == "external" + assert session.query(ListingOffer).count() == 0 + marker = session.get(SchemaVersion, 1) + assert marker.legacy_install is True + assert marker.version == 1 + + +def test_task_version_changes_only_with_meaningful_updates(session_factory): + service = TaskService(session_factory) + task = service.create_task({"keyword": "x", "target_variant": "256GB"}) + assert task.target_variant == "256GB" + assert service.update_task(task.id, {"keyword": "x"}).version == task.version + assert service.update_task(task.id, {"target_variant": "128GB"}).version == task.version + 1 + + +def test_init_db_remembers_legacy_install_before_creating_new_tables(tmp_db): + factory = make_session_factory(tmp_db) + with factory.kw["bind"].begin() as conn: + conn.execute(text("CREATE TABLE watch_tasks (id INTEGER PRIMARY KEY, keyword TEXT)")) + init_db(tmp_db) + migrate_schema(factory) + with factory() as session: + assert session.get(SchemaVersion, 1).legacy_install is True + + +def test_task_delete_preserves_listing_history_without_dangling_task_id(session_factory): + service = TaskService(session_factory) + task = service.create_task({"keyword": "x"}) + with session_factory() as session: + listing = Listing(platform="xianyu", external_id="e", title="t", price=1, + url="u", task_id=task.id) + session.add(listing) + session.commit() + listing_id = listing.id + assert service.delete_task(task.id) + with session_factory() as session: + assert session.get(Listing, listing_id).task_id is None + + +def test_listing_delete_cascades_offer_observations(session_factory): + with session_factory() as session: + listing = Listing(platform="xianyu", external_id="e", title="t", price=1, url="u") + session.add(listing) + session.flush() + offer = ListingOffer(listing_id=listing.id, offer_key="sku:1", name="256GB") + session.add(offer) + session.flush() + session.add(OfferObservation(offer_id=offer.id, fingerprint="f", payload={})) + session.commit() + session.delete(listing) + session.commit() + assert session.query(ListingOffer).count() == 0 + assert session.query(OfferObservation).count() == 0 From 798713a7f5c7726b5d8fb961dbeca0a22dc47985 Mon Sep 17 00:00:00 2001 From: Comui520 Date: Mon, 26 Oct 2026 10:58:14 +0800 Subject: [PATCH 06/17] feat: resolve trustworthy prices and manual quotes --- goodprice/offers/pricing.py | 79 +++++++++++++++++++++++++++++++++++ goodprice/offers/service.py | 76 +++++++++++++++++++++++++++++++++ tests/test_manual_quotes.py | 43 +++++++++++++++++++ tests/test_offer_pricing.py | 83 +++++++++++++++++++++++++++++++++++++ 4 files changed, 281 insertions(+) create mode 100644 goodprice/offers/pricing.py create mode 100644 goodprice/offers/service.py create mode 100644 tests/test_manual_quotes.py create mode 100644 tests/test_offer_pricing.py diff --git a/goodprice/offers/pricing.py b/goodprice/offers/pricing.py new file mode 100644 index 0000000..ee6cf76 --- /dev/null +++ b/goodprice/offers/pricing.py @@ -0,0 +1,79 @@ +import hashlib +import json +import re +from datetime import datetime, timedelta, timezone +from decimal import Decimal, InvalidOperation + +from goodprice.offers.types import OfferCandidate, PriceDecision, QuoteInput + + +def utc(value: datetime) -> datetime: + return value.replace(tzinfo=timezone.utc) if value.tzinfo is None else value.astimezone(timezone.utc) + + +def parse_money_cents(raw: str) -> int: + text = str(raw).strip().removeprefix("¥").removeprefix("¥").removesuffix("元").strip() + if not re.fullmatch(r"(?:\d+|\d{1,3}(?:,\d{3})+)(?:\.\d{1,2})?", text): + raise ValueError("价格必须是非负人民币金额,最多两位小数") + try: + cents = Decimal(text.replace(",", "")) * 100 + if not cents.is_finite() or cents < 0 or cents > 10**12: + raise ValueError("价格超出支持范围") + return int(cents) + except InvalidOperation as exc: + raise ValueError("无效价格") from exc + + +def offer_fingerprint(offer: OfferCandidate) -> str: + # Capture time is deliberately excluded: renewing an unchanged observation + # must not invalidate a quote or cause another paid analysis/notification. + claims = sorted((c.source, c.kind, c.text, c.amount_cents, c.min_cents, c.max_cents, + c.scope_key or "") for c in offer.claims) + raw = {"key": offer.key, "attributes": dict(offer.attributes), + "availability": offer.availability, "claims": claims} + return hashlib.sha256(json.dumps(raw, sort_keys=True, ensure_ascii=False).encode()).hexdigest() + + +def decide_price(offer: OfferCandidate, quote: QuoteInput | None, + now: datetime, ttl_seconds: int) -> PriceDecision: + now = utc(now) + claims = [c for c in offer.claims if c.scope_key in (None, "", offer.key)] + ids = tuple(c.evidence_id for c in claims) + + def result(state, reason, amount=None, source=None, until=None): + return PriceDecision(state, amount, source, ids, reason, until) + + if ttl_seconds <= 0 or not claims: + return result("unknown", "无有效价格证据") + if offer.availability == "unavailable": + return result("unknown", "该规格不可购买") + until = min(utc(c.observed_at) for c in claims) + timedelta(seconds=ttl_seconds) + if any(utc(c.observed_at) > now + timedelta(seconds=5) for c in claims) or until <= now: + return result("unknown", "价格或规格证据已过期,等待重新核验") + if quote and not quote.revoked and utc(quote.expires_at) > now: + if quote.offer_key != offer.key or quote.evidence_fingerprint != offer_fingerprint(offer): + return result("conflict", "页面证据已变化,请复核人工报价") + if utc(quote.confirmed_at) <= now and type(quote.amount_cents) is int and quote.amount_cents > 0: + return result("exact", "人工确认报价", quote.amount_cents, "manual", + min(until, utc(quote.expires_at))) + return result("unknown", "无效的人工报价") + for state in ("conflict", "inquiry"): + if any(c.kind == state for c in claims): + return result(state, "价格声明存在冲突" if state == "conflict" else "需联系卖家询价") + exact = [c for c in claims if c.kind == "exact" and c.source in {"sku", "dom", "description"} + and type(c.amount_cents) is int and 0 < c.amount_cents <= 10**12] + amounts = {c.amount_cents for c in exact} + if len(amounts) > 1: + return result("conflict", "同一规格有多个不一致价格") + if len(amounts) == 1: + return result("exact", "该规格有明确页面价格", exact[0].amount_cents, exact[0].source, until) + if any(c.kind == "range_or_from" for c in claims): + return result("range_or_from", "只有起售价或价格区间") + return result("unknown", "缺少可核验的明确价格") + + +def is_comparable_drop(previous: PriceDecision, current: PriceDecision, same_offer: bool) -> bool: + return bool(same_offer and previous.state == current.state == "exact" + and previous.source == current.source + and previous.amount_cents is not None and current.amount_cents is not None + and 0 < current.amount_cents < previous.amount_cents) diff --git a/goodprice/offers/service.py b/goodprice/offers/service.py new file mode 100644 index 0000000..3a045a2 --- /dev/null +++ b/goodprice/offers/service.py @@ -0,0 +1,76 @@ +from datetime import datetime, timezone + +from goodprice.models import Listing, ListingOffer, ManualQuote, OfferObservation, WatchTask +from goodprice.offers.pricing import offer_fingerprint, utc +from goodprice.offers.types import OfferCandidate, PriceClaim, QuoteInput + + +def encode_offer(offer: OfferCandidate) -> dict: + return {"key": offer.key, "attributes": dict(offer.attributes), "name": offer.name, + "availability": offer.availability, + "claims": [{**vars(c), "observed_at": utc(c.observed_at).isoformat()} for c in offer.claims]} + + +def decode_offer(payload: dict) -> OfferCandidate: + return OfferCandidate(payload["key"], payload["attributes"], payload["availability"], + tuple(PriceClaim(**{**c, "observed_at": datetime.fromisoformat(c["observed_at"])}) + for c in payload["claims"]), payload.get("name", "")) + + +class OfferService: + def __init__(self, session_factory): + self._session_factory = session_factory + + def record_observation(self, listing_id: int, offer: OfferCandidate) -> int: + if not offer.key or not offer.claims: + raise ValueError("规格或证据不能为空") + with self._session_factory() as session: + if session.get(Listing, listing_id) is None: + raise LookupError("商品不存在") + row = session.query(ListingOffer).filter_by(listing_id=listing_id, offer_key=offer.key).first() + if row is None: + row = ListingOffer(listing_id=listing_id, offer_key=offer.key) + session.add(row) + session.flush() + row.name, row.attributes, row.availability = offer.name, dict(offer.attributes), offer.availability + session.add(OfferObservation(offer_id=row.id, fingerprint=offer_fingerprint(offer), + payload=encode_offer(offer), + observed_at=min(utc(c.observed_at) for c in offer.claims))) + session.commit() + return row.id + + def confirm_quote(self, offer_id: int, quote: QuoteInput) -> int: + now = datetime.now(timezone.utc) + if (type(quote.amount_cents) is not int or not 0 < quote.amount_cents <= 10**12 + or utc(quote.expires_at) <= now or utc(quote.confirmed_at) > now + or utc(quote.expires_at) <= utc(quote.confirmed_at) or quote.revoked): + raise ValueError("报价金额、确认时间或有效期无效") + with self._session_factory() as session: + offer = session.get(ListingOffer, offer_id) + if offer is None: + raise LookupError("规格不存在") + latest = session.query(OfferObservation).filter_by(offer_id=offer_id).order_by( + OfferObservation.observed_at.desc(), OfferObservation.id.desc()).first() + listing = session.get(Listing, offer.listing_id) + task = session.get(WatchTask, listing.task_id) if listing.task_id else None + ttl = (task.interval_minutes if task else 20) * 60 + if (latest is None or latest.fingerprint != quote.evidence_fingerprint + or offer.offer_key != quote.offer_key or offer.availability == "unavailable" + or (now - utc(latest.observed_at)).total_seconds() >= ttl): + raise ValueError("证据已变化或过期,请刷新后重新确认") + for old in session.query(ManualQuote).filter_by(offer_id=offer_id, revoked=False): + old.revoked = True + row = ManualQuote(offer_id=offer_id, amount_cents=quote.amount_cents, + confirmed_at=utc(quote.confirmed_at), expires_at=utc(quote.expires_at), + evidence_fingerprint=quote.evidence_fingerprint, revoked=False) + session.add(row) + session.commit() + return row.id + + def revoke_quote(self, quote_id: int) -> None: + with self._session_factory() as session: + quote = session.get(ManualQuote, quote_id) + if quote is None: + raise LookupError("报价不存在") + quote.revoked = True + session.commit() diff --git a/tests/test_manual_quotes.py b/tests/test_manual_quotes.py new file mode 100644 index 0000000..bcd2b87 --- /dev/null +++ b/tests/test_manual_quotes.py @@ -0,0 +1,43 @@ +from datetime import datetime, timedelta, timezone + +import pytest + +from goodprice.models import Listing, ListingOffer, ManualQuote, OfferObservation +from goodprice.offers.pricing import offer_fingerprint +from goodprice.offers.service import OfferService +from goodprice.offers.types import OfferCandidate, PriceClaim, QuoteInput + + +def test_quote_validation_before_commit_and_independent_storage(session_factory): + now = datetime.now(timezone.utc) + with session_factory() as session: + listing = Listing(platform="xianyu", external_id="1", title="t", price=1, url="u") + session.add(listing) + session.commit() + listing_id = listing.id + service = OfferService(session_factory) + offer = OfferCandidate("sku:1", {"容量": "256GB"}, "available", ( + PriceClaim("sku", "exact", "1500元", now, "p", amount_cents=150000),)) + offer_id = service.record_observation(listing_id, offer) + bad = QuoteInput("sku:1", -1, now, now + timedelta(days=7), offer_fingerprint(offer)) + with pytest.raises(ValueError): + service.confirm_quote(offer_id, bad) + with session_factory() as session: + assert session.query(ManualQuote).count() == 0 + quote = QuoteInput("sku:1", 140000, now, now + timedelta(days=7), offer_fingerprint(offer)) + quote_id = service.confirm_quote(offer_id, quote) + service.record_observation(listing_id, offer) + with session_factory() as session: + assert session.get(ManualQuote, quote_id).amount_cents == 140000 + assert session.query(ListingOffer).count() == 1 + service.revoke_quote(quote_id) + with session_factory() as session: + assert session.get(ManualQuote, quote_id).revoked is True + assert session.query(OfferObservation).count() >= 1 + + +def test_stale_confirmation_cannot_be_saved(session_factory): + now = datetime.now(timezone.utc) + service = OfferService(session_factory) + with pytest.raises((ValueError, LookupError)): + service.confirm_quote(999, QuoteInput("x", 1, now, now, "old")) diff --git a/tests/test_offer_pricing.py b/tests/test_offer_pricing.py new file mode 100644 index 0000000..522b4b7 --- /dev/null +++ b/tests/test_offer_pricing.py @@ -0,0 +1,83 @@ +from dataclasses import replace +from datetime import datetime, timedelta, timezone + +import pytest + +from goodprice.offers.pricing import ( + decide_price, + is_comparable_drop, + offer_fingerprint, + parse_money_cents, +) +from goodprice.offers.types import OfferCandidate, PriceClaim, QuoteInput + +NOW = datetime(2026, 10, 26, 0, 0, tzinfo=timezone.utc) + + +def candidate(kind="exact", cents=150000, text="256GB 1500元", source="sku"): + return OfferCandidate("sku:256", {"容量": "256GB"}, "available", ( + PriceClaim(source, kind, text, NOW, "price-1", amount_cents=cents),)) + + +def test_money_uses_integer_cents(): + assert parse_money_cents("1500.00") == 150000 + assert parse_money_cents("¥1,500.50元") == 150050 + + +@pytest.mark.parametrize("raw", ["NaN", "Infinity", "-1", "1.001", "USD 3", "3起", "1e3"]) +def test_invalid_money_rejected(raw): + with pytest.raises(ValueError): + parse_money_cents(raw) + + +@pytest.mark.parametrize("kind", ["range_or_from", "inquiry", "conflict", "unknown"]) +def test_non_exact_claim_is_never_usable_price(kind): + result = decide_price(candidate(kind), None, NOW, 1200) + assert result.state == kind + assert result.amount_cents is None + + +def test_exact_price_and_freshness_boundary(): + result = decide_price(candidate(), None, NOW, 1200) + assert (result.state, result.amount_cents) == ("exact", 150000) + assert decide_price(candidate(), None, NOW + timedelta(seconds=1200), 1200).state == "unknown" + + +def test_unsupported_evidence_cannot_assert_price(): + assert decide_price(candidate(source="llm"), None, NOW, 1200).amount_cents is None + assert decide_price(replace(candidate(), availability="unavailable"), None, NOW, 1200).amount_cents is None + + +def test_global_disclaimer_overrides_sku_and_different_prices_conflict(): + offer = candidate() + disclaimer = PriceClaim("description", "inquiry", "所有标价非售价", NOW, "desc") + assert decide_price(replace(offer, claims=(*offer.claims, disclaimer)), None, NOW, 1200).state == "inquiry" + other = replace(offer.claims[0], amount_cents=160000, evidence_id="price-2") + assert decide_price(replace(offer, claims=(*offer.claims, other)), None, NOW, 1200).state == "conflict" + + +def test_manual_quote_priority_expiry_change_and_source_switch(): + offer = candidate() + quote = QuoteInput(offer.key, 140000, NOW, NOW + timedelta(days=7), offer_fingerprint(offer)) + manual = decide_price(offer, quote, NOW, 1200) + assert manual.amount_cents == 140000 + assert manual.source == "manual" + page = decide_price(offer, replace(quote, revoked=True), NOW, 1200) + assert not is_comparable_drop(page, manual, same_offer=True) + assert decide_price(candidate(cents=160000), quote, NOW, 1200).state == "conflict" + refreshed = replace(offer, claims=(replace(offer.claims[0], observed_at=NOW + timedelta(days=8)),)) + assert offer_fingerprint(refreshed) == offer_fingerprint(offer) + assert decide_price(refreshed, quote, NOW + timedelta(days=8), 1200).source == "sku" + + +def test_refresh_failure_does_not_extend_quote_recommendation(): + offer = candidate() + quote = QuoteInput(offer.key, 140000, NOW, NOW + timedelta(days=7), offer_fingerprint(offer)) + assert decide_price(offer, quote, NOW + timedelta(hours=1), 1200).state == "unknown" + + +def test_same_source_and_same_offer_required_for_drop(): + previous = decide_price(candidate(), None, NOW, 1200) + current = decide_price(candidate(cents=130000), None, NOW, 1200) + assert is_comparable_drop(previous, current, same_offer=True) + assert not is_comparable_drop(previous, current, same_offer=False) From 1c91a00f2e9d5d78a8cbc136050ef56733c974eb Mon Sep 17 00:00:00 2001 From: Comui520 Date: Mon, 26 Oct 2026 11:42:45 +0800 Subject: [PATCH 07/17] feat: evaluate and present trustworthy offer prices --- Dockerfile | 6 + README.md | 17 + .../validation/2026-10-26-phase1-milestone.md | 44 ++ goodprice/crawler/base.py | 3 + goodprice/crawler/offers.py | 120 +++++ goodprice/crawler/parser.py | 2 + goodprice/db.py | 7 + goodprice/main.py | 2 + goodprice/models.py | 7 + goodprice/offers/matching.py | 4 +- goodprice/offers/pricing.py | 6 + goodprice/offers/service.py | 22 +- goodprice/services/crawl_service.py | 409 +----------------- goodprice/services/offer_crawl.py | 396 +++++++++++++++++ goodprice/services/offer_pipeline.py | 132 ++++++ goodprice/web/offer_routes.py | 64 +++ goodprice/web/routes.py | 49 ++- goodprice/web/templates/listings_cards.html | 4 +- goodprice/web/templates/listings_detail.html | 25 +- goodprice/web/templates/tasks.html | 1 + goodprice/web/templates/tasks_detail.html | 3 + tests/test_api.py | 15 +- tests/test_crawl_service.py | 26 +- tests/test_judge.py | 4 +- tests/test_offer_api.py | 55 +++ tests/test_offer_identity.py | 4 + tests/test_offer_parser.py | 46 ++ tests/test_offer_pipeline.py | 137 ++++++ tests/test_offer_pricing.py | 8 + tests/test_offer_regressions.py | 158 +++++++ 30 files changed, 1361 insertions(+), 415 deletions(-) create mode 100644 docs/validation/2026-10-26-phase1-milestone.md create mode 100644 goodprice/crawler/offers.py create mode 100644 goodprice/services/offer_crawl.py create mode 100644 goodprice/services/offer_pipeline.py create mode 100644 goodprice/web/offer_routes.py create mode 100644 tests/test_offer_api.py create mode 100644 tests/test_offer_parser.py create mode 100644 tests/test_offer_pipeline.py create mode 100644 tests/test_offer_regressions.py diff --git a/Dockerfile b/Dockerfile index e53a13e..11eae84 100644 --- a/Dockerfile +++ b/Dockerfile @@ -15,6 +15,12 @@ ARG HTTPS_PROXY # not invalidate the large Chromium download. COPY requirements.lock ./ +# Use authenticated HTTPS transport and bounded retries for Debian downloads. +# Keep repository signatures enabled; transient HTTP CDN 502s are not a reason +# to bypass apt authentication. +RUN sed -i 's|http://deb.debian.org|https://deb.debian.org|g' /etc/apt/sources.list.d/debian.sources \ + && printf 'Acquire::Retries "3";\n' > /etc/apt/apt.conf.d/80-second-eye-retries + RUN env HTTP_PROXY="${HTTP_PROXY}" HTTPS_PROXY="${HTTPS_PROXY}" \ http_proxy="${HTTP_PROXY}" https_proxy="${HTTPS_PROXY}" \ sh -c 'pip install --no-cache-dir --retries 10 --timeout 120 -r requirements.lock && playwright install --with-deps chromium' diff --git a/README.md b/README.md index 78c178f..cfaea46 100644 --- a/README.md +++ b/README.md @@ -210,6 +210,23 @@ Conda 不是运行本项目的必需条件;新用户优先选择 Docker 或 uv ## 配置 +### 开发分支:规格与价格证据(实验性) + +监控任务可在详情页填写目标型号/容量(例如 `iPhone 15 / 256GB`)。 +商品页面标价保留为原始观察,不代表目标规格的实际售价;仅明确规格价格才用于预算与性价比判断。 +价格私聊、区间价、信息冲突或详情提取失败的商品保留为待确认,不以占位价推荐。 +“待询价提醒”需在任务中主动开启,默认不增加外部通知。 + +商品详情可单独确认/撤销人工报价,默认有效期 7 天;人工报价优先,但证据变化需复核, +过期证据不继续作为推荐依据。人工报价与页面价的来源切换不当作降价。 +设置最低品相分后,未核验的图片或多规格共享样品不会绕过门槛。 + +当前支持可明确识别的商品范围 DOM 数据与 Schema.org Product/Offer,尚未取得真实多规格页面样本来验证所有闲鱼控件, +不能识别时显示待核验而不是猜测。视频分析、分阶段并发调度与完整交互改造仍属后续工作,不能将本开发分支当作已完成的稳定发布。 + +应用现使用无导入副作用的工厂;自定义 ASGI 部署请使用 `uvicorn goodprice.main:create_app --factory`, +原有 `python -m goodprice` 启动方式不变。升级旧库前先停止服务并备份数据库,回退使用旧镜像与备份库成对恢复。 + 配置文件: ~~~text diff --git a/docs/validation/2026-10-26-phase1-milestone.md b/docs/validation/2026-10-26-phase1-milestone.md new file mode 100644 index 0000000..4166681 --- /dev/null +++ b/docs/validation/2026-10-26-phase1-milestone.md @@ -0,0 +1,44 @@ +# 升级阶段一核心功能验证记录 + +状态:阶段一核心实现已落地;整个升级未完成,未合并、推送或发版。 + +## 已落地 + +- 应用采用无 import 副作用工厂;原 `python -m goodprice` 保持可用。 +- 规格身份、整分金额、明确价/区间/询价/冲突/未知价裁决。 +- 观察记录与人工报价分开保存;报价默认 7 天、支持撤销,证据过期不继续推荐。 +- 搜索页起售价不提前过滤目标规格,品相分析和性价比使用所匹配规格的确价。 +- 最低品相分不被未知评分绕过;有限搜索未见不冒充确定下架。 +- 实验性 DOM/Schema.org 提取、商品证据展示、任务目标规格和待询价提醒。 +- 单渠道失败不会被其他成功渠道遮蔽;未知投递结果不自动反复重试。 +- 旧库幂等迁移、旧通知基线兼容;用户运行数据未用于开发测试。 + +## 证据 + +- `uv lock --check`:通过。 +- `uv run --locked ruff check .`:通过。 +- `uv run --locked pytest -q --cov --cov-fail-under=80`:295 passed,覆盖率 88.44%。 +- 一项已知的上游 Starlette/httpx 弃用警告,不是测试失败。 +- 独立审查发现六项 Important,逐项编写失败用例后修复;另覆盖媒体移除和视觉请求错误使用标价的回归。 +- 最终清洁 Docker 构建成功,测试镜像 `second-eye:upgrade-phase1`;SDK 导入为 typesafe-sdk 0.7.2。 +- Linux 容器使用临时 tmpfs 数据、动态回环端口、禁用测试任务;healthz、五个主要页面、规格 API、确认和撤销报价通过。 +- 测试容器已删除,未替换现有应用镜像标签/生产容器,未挂载真实 data。 + +## 设计裁决与修复范围 + +- 阶段一使用串行适配器复用 offer 阶段契约;移除旧的重复价格/通知私有实现,避免两条业务规则并存。 + 代价:外部代码若调用旧私有方法需迁移,公开 run_task 入口保留。 +- 删除任务保留商品和报价历史、清除任务关联,而不是删除用户历史。代价:会保留无法再调度的历史商品。 +- 旧“详情失败继续按标价推荐/有限搜索三轮未见就确定下架”的测试按已批准语义更新,而非保留不可信行为。 +- Docker 两次构建遇到官方 Debian HTTP CDN 502,改为官方 HTTPS 源并启用三次有限重试;签名验证未关闭。 + 若 HTTPS 也不可达则仍明确构建失败,不做不安全回退。 + +## 未完成/未验证 + +- 尚无真实多 SKU/视频代表样本;不宣称已适配全部闲鱼 SKU 控件。 +- 有界持久化的跨轮候选续抓、阶段调度、资源保护与设备推荐仍需阶段三接入;当前串行适配器只报告超过30候选的 deferred 数量。 +- 视频采集、下载/解码配额、抽帧、媒体数量设置尚未实施。 +- 全站筛选返回状态、设置分类、局部保存、交互与性能对比尚未实施。 +- 列表证据投影目前有逐商品读取,阶段三需批量查询并实测 N+1;本记录不宣称已提升总体性能。 +- macOS、ARM 与真实平台端到端未测试;未修改真实数据库或正在使用的环境来模拟验收。 +- 阶段一尚不能视为整个17任务计划完成;后续执行须从进度记录继续,不能重复前三个已提交任务。 diff --git a/goodprice/crawler/base.py b/goodprice/crawler/base.py index 14a0749..e45803b 100644 --- a/goodprice/crawler/base.py +++ b/goodprice/crawler/base.py @@ -2,6 +2,8 @@ from datetime import datetime from typing import Optional +from goodprice.offers.types import OfferCandidate + @dataclass class ListingData: @@ -20,6 +22,7 @@ class ListingDetail: description: str = "" image_urls: list[str] = field(default_factory=list) variants: list[dict] = field(default_factory=list) + offers: list[OfferCandidate] = field(default_factory=list) seller_uid: Optional[str] = None seller_name: Optional[str] = None credit_label: Optional[str] = None diff --git a/goodprice/crawler/offers.py b/goodprice/crawler/offers.py new file mode 100644 index 0000000..fd30ecd --- /dev/null +++ b/goodprice/crawler/offers.py @@ -0,0 +1,120 @@ +"""Conservative, bounded offer parsing. Unsupported platform state stays unknown. + +The data-* fixture format and Schema.org Product/Offer are supported adapters, +not a claim that every current Goofish SKU widget exposes these attributes. +""" +import json +import re +from dataclasses import replace +from datetime import datetime, timezone + +from bs4 import BeautifulSoup + +from goodprice.crawler import selectors as sel +from goodprice.offers.matching import stable_offer_key +from goodprice.offers.pricing import parse_money_cents +from goodprice.offers.types import OfferCandidate, PriceClaim + +_INQUIRY = re.compile(r"(?:标价.{0,5}(?:非|不代表|不为).{0,5}(?:售|价)|(?:价格|报价).{0,8}(?:私聊|咨询|面议)|(?:私聊|咨询).{0,8}(?:价格|报价))") + + +def _claim(raw, source, now, evidence_id, scope=None): + text = str(raw or "").strip()[:1000] + kind, amount, low, high = "unknown", None, None, None + if _INQUIRY.search(text): + kind = "inquiry" + elif "起" in text or re.search(r"\d\s*[-~至]\s*\d", text): + kind = "range_or_from" + numbers = re.findall(r"\d+(?:\.\d{1,2})?", text) + if numbers: + low = parse_money_cents(numbers[0]) + high = parse_money_cents(numbers[1]) if len(numbers) > 1 else None + else: + try: + amount = parse_money_cents(text) + kind = "exact" if amount > 0 else "unknown" + except ValueError: + pass + return PriceClaim(source, kind, text, now, evidence_id, amount, low, high, scope) + + +def parse_offer_candidates(html: str) -> list[OfferCandidate]: + now = datetime.now(timezone.utc) + soup = BeautifulSoup(html, "html.parser") + root = soup.select_one("[data-item-detail], main, [class*='detail-container--']") + result = [] + if root: + description = root.select_one("[data-item-description], " + sel.DETAIL_DESC) + disclaimer = description.get_text(" ", strip=True) if description else "" + global_claims = ((_claim(disclaimer, "description", now, "description"),) + if _INQUIRY.search(disclaimer) else ()) + nodes = root.select("[data-sku-id]") + for i, node in enumerate(nodes[:21]): + if i == 20: + result = [replace(o, claims=(*o.claims, PriceClaim( + "dom", "unknown", "规格获取不完整", now, "truncated"))) for o in result] + break + try: + attrs = json.loads(node.get("data-attributes", "{}")) + if not isinstance(attrs, dict) or any(not isinstance(v, str) for v in attrs.values()): + continue + key = stable_offer_key(node.get("data-sku-id"), attrs) + if not key: + continue + claim = _claim(node.get("data-price"), "sku", now, f"{key}:price", key) + available = "unavailable" if node.get("data-available") == "false" else "unknown" + if node.get("data-available") == "true": + available = "available" + result.append(OfferCandidate(key, attrs, available, (claim, *global_claims), + node.get("data-name", ""))) + except (ValueError, TypeError): + continue + if not nodes: + price = root.select_one("[data-item-price], " + sel.DETAIL_PRICE_RANGE) + if price or global_claims: + result.append(OfferCandidate("default", {}, "unknown", ( + _claim(price.get_text(" ", strip=True) if price else "", "dom", now, "detail:price"), + *global_claims), "默认商品")) + if not result: + products = [] + for script in soup.select('script[type="application/ld+json"]')[:10]: + try: + data = json.loads(script.string or "") + except (ValueError, TypeError): + continue + items = data if isinstance(data, list) else [data] + products.extend(d for d in items if isinstance(d, dict) and d.get("@type") == "Product") + if len(products) == 1: + product = products[0] + offers = product.get("offers", []) + offers = [offers] if isinstance(offers, dict) else offers + if isinstance(offers, list): + for i, data in enumerate(offers[:20]): + if not isinstance(data, dict) or data.get("priceCurrency", "CNY") != "CNY": + continue + attrs = data.get("itemOffered", {}).get("additionalProperty", []) if isinstance(data.get("itemOffered", {}), dict) else [] + attrs = {str(a["name"]): str(a["value"]) for a in attrs + if isinstance(a, dict) and "name" in a and "value" in a} if isinstance(attrs, list) else {} + key = stable_offer_key(data.get("sku"), attrs) or ("default" if len(offers) == 1 else None) + if key: + claim = _claim(data.get("price", ""), "sku", now, f"jsonld:{i}", key) + if len(offers) > 20: + claim = replace(claim, kind="unknown", amount_cents=None, text="规格获取不完整") + available = "unavailable" if "OutOfStock" in str(data.get("availability")) else "unknown" + if "InStock" in str(data.get("availability")): + available = "available" + description = str(product.get("description", "")) + disclaimers = (_claim(description, "description", now, "jsonld:description"),) if _INQUIRY.search(description) else () + result.append(OfferCandidate(key, attrs, available, (claim, *disclaimers), str(data.get("name", "")))) + if not result: + result = [OfferCandidate("default", claims=(PriceClaim( + "dom", "unknown", "未找到可核验的规格价格", now, "missing"),))] + merged = {} + for offer in result: + if offer.key in merged: + old = merged[offer.key] + merged[offer.key] = replace(old, claims=(*old.claims, *offer.claims, PriceClaim( + "dom", "conflict", "相同规格身份存在重复记录", now, "duplicate"))) + else: + merged[offer.key] = offer + return list(merged.values()) diff --git a/goodprice/crawler/parser.py b/goodprice/crawler/parser.py index 99a1adf..71b8bd8 100644 --- a/goodprice/crawler/parser.py +++ b/goodprice/crawler/parser.py @@ -7,6 +7,7 @@ from goodprice.crawler import selectors as sel from goodprice.crawler.base import ListingData, ListingDetail, SellerData from goodprice.crawler.constants import GOOFISH_ORIGIN +from goodprice.crawler.offers import parse_offer_candidates BASE_URL = GOOFISH_ORIGIN _PRICE_RE = re.compile(r"(\d+(?:\.\d+)?)") @@ -113,6 +114,7 @@ def parse_detail_html(html: str) -> ListingDetail: credit_el = soup.select_one(sel.DETAIL_CREDIT_LABEL) credit_label = credit_el.get_text(strip=True) if credit_el else None return ListingDetail( + offers=parse_offer_candidates(html), description=desc[:2000], image_urls=images[:8], variants=variants, diff --git a/goodprice/db.py b/goodprice/db.py index 529ff28..b849dcc 100644 --- a/goodprice/db.py +++ b/goodprice/db.py @@ -9,6 +9,9 @@ external_id VARCHAR(200) NOT NULL, title VARCHAR(500) NOT NULL, price FLOAT NOT NULL, + offer_price_cents INTEGER, + offer_verified_until DATETIME, + offer_task_version INTEGER, url TEXT NOT NULL, image_urls JSON, seller VARCHAR(200), @@ -91,6 +94,9 @@ def migrate_schema(session_factory) -> None: ("version", "version INTEGER DEFAULT 1"), ], "listings": [ + ("offer_price_cents", "offer_price_cents INTEGER"), + ("offer_verified_until", "offer_verified_until DATETIME"), + ("offer_task_version", "offer_task_version INTEGER"), ("description", "description TEXT"), ("requirement_match", "requirement_match BOOLEAN"), ("requirement_reason", "requirement_reason TEXT"), @@ -113,6 +119,7 @@ def migrate_schema(session_factory) -> None: ("blocked", "blocked BOOLEAN DEFAULT 0"), ], "notifications": [ + ("event_context", "event_context JSON"), ("offer_id", "offer_id INTEGER"), ("event_key", "event_key VARCHAR(128)"), ("title", "title TEXT"), diff --git a/goodprice/main.py b/goodprice/main.py index 615f47e..0e5d92b 100644 --- a/goodprice/main.py +++ b/goodprice/main.py @@ -16,6 +16,7 @@ from goodprice.services.settings_service import SettingsService from goodprice.services.task_queue import DEFAULT_GAP_SECONDS, TaskQueue from goodprice.services.task_service import TaskService +from goodprice.web.offer_routes import router as offer_router from goodprice.web.routes import router LOG_DIR = Path(__file__).resolve().parent.parent / "data" / "logs" @@ -169,6 +170,7 @@ async def lifespan(app: FastAPI): app.state.task_queue = task_queue app.state.login_session = login_session app.include_router(router) + app.include_router(offer_router) return app diff --git a/goodprice/models.py b/goodprice/models.py index da68933..49c8554 100644 --- a/goodprice/models.py +++ b/goodprice/models.py @@ -56,6 +56,9 @@ class Listing(Base): external_id: Mapped[str] = mapped_column(String(200)) title: Mapped[str] = mapped_column(String(500)) price: Mapped[float] = mapped_column(Float) + offer_price_cents: Mapped[Optional[int]] = mapped_column(Integer, nullable=True) + offer_verified_until: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True) + offer_task_version: Mapped[Optional[int]] = mapped_column(Integer, nullable=True) url: Mapped[str] = mapped_column(Text) image_urls: Mapped[list[str]] = mapped_column(JSON, default=list) seller: Mapped[Optional[str]] = mapped_column(String(200), nullable=True) @@ -111,6 +114,7 @@ class Notification(Base): id: Mapped[int] = mapped_column(primary_key=True) offer_id: Mapped[Optional[int]] = mapped_column(Integer, nullable=True) event_key: Mapped[Optional[str]] = mapped_column(String(128), nullable=True, index=True) + event_context: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True) listing_id: Mapped[Optional[int]] = mapped_column( ForeignKey("listings.id", ondelete="SET NULL"), nullable=True ) @@ -150,6 +154,9 @@ class ListingOffer(Base): name: Mapped[str] = mapped_column(Text, default="") attributes: Mapped[dict] = mapped_column(JSON, default=dict) availability: Mapped[str] = mapped_column(String(20), default="unknown") + last_notified_score: Mapped[Optional[float]] = mapped_column(Float, nullable=True) + last_notified_amount: Mapped[Optional[int]] = mapped_column(Integer, nullable=True) + last_notified_source: Mapped[Optional[str]] = mapped_column(String(30), nullable=True) observations: Mapped[list["OfferObservation"]] = relationship(cascade="all, delete-orphan") quotes: Mapped[list["ManualQuote"]] = relationship(cascade="all, delete-orphan") evaluations: Mapped[list["OfferEvaluationRecord"]] = relationship(cascade="all, delete-orphan") diff --git a/goodprice/offers/matching.py b/goodprice/offers/matching.py index 55872f2..de07438 100644 --- a/goodprice/offers/matching.py +++ b/goodprice/offers/matching.py @@ -29,7 +29,9 @@ def match_variant(target: str, attributes: Mapping[str, str], title: str) -> Mat if not values: return "uncertain" uncertain = False - for requirement in re.split(r"[/,,;;]+", target): + capacities = re.findall(r"\d+(?:\.\d+)?\s*(?:gb|tb)", target, re.I) + remainder = re.sub(r"\d+(?:\.\d+)?\s*(?:gb|tb)", "/", target, flags=re.I) + for requirement in [*re.split(r"[/,,;;]+", remainder), *capacities]: term = _normal(requirement) if not term: continue diff --git a/goodprice/offers/pricing.py b/goodprice/offers/pricing.py index ee6cf76..2d2ae3c 100644 --- a/goodprice/offers/pricing.py +++ b/goodprice/offers/pricing.py @@ -47,6 +47,10 @@ def result(state, reason, amount=None, source=None, until=None): return result("unknown", "无有效价格证据") if offer.availability == "unavailable": return result("unknown", "该规格不可购买") + if any(c.evidence_id == "duplicate" for c in claims): + return result("conflict", "规格身份存在歧义,人工价格不能代替身份核验") + if any(c.kind == "unknown" and c.evidence_id == "truncated" for c in claims): + return result("unknown", "价格或规格证据不完整") until = min(utc(c.observed_at) for c in claims) + timedelta(seconds=ttl_seconds) if any(utc(c.observed_at) > now + timedelta(seconds=5) for c in claims) or until <= now: return result("unknown", "价格或规格证据已过期,等待重新核验") @@ -60,6 +64,8 @@ def result(state, reason, amount=None, source=None, until=None): for state in ("conflict", "inquiry"): if any(c.kind == state for c in claims): return result(state, "价格声明存在冲突" if state == "conflict" else "需联系卖家询价") + if any(c.kind == "unknown" for c in claims): + return result("unknown", "价格或规格证据不完整") exact = [c for c in claims if c.kind == "exact" and c.source in {"sku", "dom", "description"} and type(c.amount_cents) is int and 0 < c.amount_cents <= 10**12] amounts = {c.amount_cents for c in exact} diff --git a/goodprice/offers/service.py b/goodprice/offers/service.py index 3a045a2..347c1a2 100644 --- a/goodprice/offers/service.py +++ b/goodprice/offers/service.py @@ -1,6 +1,13 @@ from datetime import datetime, timezone -from goodprice.models import Listing, ListingOffer, ManualQuote, OfferObservation, WatchTask +from goodprice.models import ( + Listing, + ListingOffer, + ManualQuote, + Notification, + OfferObservation, + WatchTask, +) from goodprice.offers.pricing import offer_fingerprint, utc from goodprice.offers.types import OfferCandidate, PriceClaim, QuoteInput @@ -25,11 +32,16 @@ def record_observation(self, listing_id: int, offer: OfferCandidate) -> int: if not offer.key or not offer.claims: raise ValueError("规格或证据不能为空") with self._session_factory() as session: - if session.get(Listing, listing_id) is None: + listing = session.get(Listing, listing_id) + if listing is None: raise LookupError("商品不存在") row = session.query(ListingOffer).filter_by(listing_id=listing_id, offer_key=offer.key).first() if row is None: row = ListingOffer(listing_id=listing_id, offer_key=offer.key) + legacy_sent = session.query(Notification).filter_by( + listing_id=listing_id, offer_id=None, status="sent").first() + if legacy_sent and offer.key == "default": + row.last_notified_source = "legacy_baseline" session.add(row) session.flush() row.name, row.attributes, row.availability = offer.name, dict(offer.attributes), offer.availability @@ -52,6 +64,9 @@ def confirm_quote(self, offer_id: int, quote: QuoteInput) -> int: latest = session.query(OfferObservation).filter_by(offer_id=offer_id).order_by( OfferObservation.observed_at.desc(), OfferObservation.id.desc()).first() listing = session.get(Listing, offer.listing_id) + candidate = decode_offer(latest.payload) if latest else None + if candidate and any(c.evidence_id in ("duplicate", "truncated") for c in candidate.claims): + raise ValueError("规格身份不明确或获取不完整,不能用报价代替核验") task = session.get(WatchTask, listing.task_id) if listing.task_id else None ttl = (task.interval_minutes if task else 20) * 60 if (latest is None or latest.fingerprint != quote.evidence_fingerprint @@ -64,6 +79,7 @@ def confirm_quote(self, offer_id: int, quote: QuoteInput) -> int: confirmed_at=utc(quote.confirmed_at), expires_at=utc(quote.expires_at), evidence_fingerprint=quote.evidence_fingerprint, revoked=False) session.add(row) + listing.offer_verified_until = None session.commit() return row.id @@ -73,4 +89,6 @@ def revoke_quote(self, quote_id: int) -> None: if quote is None: raise LookupError("报价不存在") quote.revoked = True + offer = session.get(ListingOffer, quote.offer_id) + session.get(Listing, offer.listing_id).offer_verified_until = None session.commit() diff --git a/goodprice/services/crawl_service.py b/goodprice/services/crawl_service.py index a305eab..ddc9d03 100644 --- a/goodprice/services/crawl_service.py +++ b/goodprice/services/crawl_service.py @@ -1,19 +1,11 @@ import logging -import random import threading -import time from datetime import datetime -from typing import Any, Optional +from typing import Any -from goodprice.crawler.base import ListingData +from goodprice.crawler.base import ListingData, ListingDetail from goodprice.crawler.parser import is_product_image -from goodprice.models import Listing, Notification, PriceSnapshot, WatchTask -from goodprice.notify.base import NotificationMessage -from goodprice.services.satisfaction import ( - DROP_PCT_DISPLAY_THRESHOLD, - compute_satisfaction, - drop_pct_from_snapshots, -) +from goodprice.models import Listing, PriceSnapshot, WatchTask logger = logging.getLogger(__name__) @@ -71,164 +63,14 @@ def run_task(self, task_id: int) -> dict[str, Any]: logger.info("任务 %s 已在运行,忽略重复触发", task_id) return {"found": 0, "new": 0, "notified": 0, "skipped": "already_running"} try: - return self._run_impl(task_id) - finally: - self.guard.finish(task_id) + from goodprice.services.offer_crawl import OfferCrawlRunner - def _run_impl(self, task_id: int) -> dict[str, Any]: - stats = { - "found": 0, - "new": 0, - "notified": 0, - "backfilled": 0, - "reevaluated": 0, - "gone": 0, - } - settings = self.settings_service.get() - self._judger = self._build_judger(settings) - jitter = int(settings.default_crawl_jitter_minutes) - if jitter: - time.sleep(random.uniform(0, jitter * 60)) - with self._session_factory() as session: - task = session.get(WatchTask, task_id) - if task is None: - raise RuntimeError(f"任务 {task_id} 不存在") - task.last_run_at = datetime.now() - task.last_error = None - task.last_run_count = (task.last_run_count or 0) + 1 - session.commit() - try: - items = self.adapter.search(task.keyword) + return OfferCrawlRunner(self).run(task_id) except Exception as exc: - self._record_error(task_id, f"抓取失败: {exc}") + self._record_error(task_id, f"处理商品时出错: {exc}"[:1000]) raise - stats["found"] = len(items) - logger.info("任务 %s 搜索命中 %s 条", task_id, len(items)) - batch_rows: list[dict] = [] - pending: list[tuple] = [] # (task, listing, old_price, is_renotify) - seen_ids: set[int] = set() - with self._session_factory() as session: - task = session.get(WatchTask, task_id) - try: - for data in items: - if task.max_price and data.price > task.max_price: - logger.info("任务 %s 跳过(超最高价 %s): %s ¥%s", task_id, task.max_price, data.title[:30], data.price) - continue - if task.min_price and data.price < task.min_price: - logger.info("任务 %s 跳过(低于价格下限 %s): %s ¥%s", task_id, task.min_price, data.title[:30], data.price) - continue - if self._excluded(task, data.title): - logger.info("任务 %s 跳过(命中排除词): %s", task_id, data.title[:40]) - continue - old_price: Optional[float] = None - existing = ( - session.query(Listing) - .filter( - Listing.platform == task.platform, - Listing.external_id == data.external_id, - Listing.task_id == task.id, - ) - .first() - ) - if existing is not None and abs(existing.price - data.price) > 0.001: - old_price = existing.price - listing, is_new = self._upsert_listing(session, task, data) - seen_ids.add(listing.id) - if self._is_blocked(session, listing): - session.commit() - continue - listing.status = "active" - listing.missed_count = 0 - if is_new: - stats["new"] += 1 - logger.info("任务 %s 新品 %s:%s ¥%s", task_id, data.external_id, data.title[:30], data.price) - if task.fetch_detail: - self._fetch_detail(session, listing) - reason = self._evaluate_and_queue( - session, task, listing, old_price, batch_rows, pending, False - ) - if reason: - logger.info("任务 %s %s,不收录:%s", task_id, reason, listing.title[:30]) - session.commit() - continue - else: - changed = old_price is not None or listing.status == "gone" - if changed: - stats["reevaluated"] += 1 - logger.info("任务 %s 重评 %s:%s(价格变化或重新上架)", task_id, data.external_id, data.title[:30]) - reason = self._evaluate_and_queue( - session, task, listing, old_price, batch_rows, pending, True - ) - if reason: - session.commit() - continue - else: - if self._backfill(session, listing, task): - stats["backfilled"] += 1 - session.commit() - # 下架软标记:本轮未出现的该任务商品 - for other in ( - session.query(Listing) - .filter(Listing.task_id == task.id) - .filter(~Listing.id.in_(seen_ids)) - ): - other.missed_count = (other.missed_count or 0) + 1 - if other.missed_count >= GONE_THRESHOLD and other.status != "gone": - other.status = "gone" - stats["gone"] += 1 - logger.info("任务 %s 商品 %s 标记为已下架", task_id, other.title[:30]) - session.commit() - # 批性价比:本批通过筛选的商品统一横向对比 - if batch_rows and self._value_client() is not None: - self._batch_value(session, batch_rows) - session.commit() - # 通知:新品在批性价比后统一发出;重评仅满意度提高时发出 - for task, listing, old_price, is_renotify in pending: - satisfaction = self._satisfaction(listing) - listing.satisfaction = satisfaction - if ( - is_renotify - and listing.last_notified_satisfaction is not None - and satisfaction <= listing.last_notified_satisfaction - ): - continue - self._notify( - session, - task, - listing, - satisfaction, - old_price=old_price, - is_renotify=is_renotify, - ) - stats["notified"] += 1 - session.commit() - except Exception as exc: - task.last_error = f"处理商品时出错: {exc}"[:1000] - session.commit() - raise - session.commit() - return stats - - def _evaluate_and_queue( - self, - session, - task: WatchTask, - listing: Listing, - old_price: Optional[float], - batch_rows: list[dict], - pending: list[tuple], - is_renotify: bool, - ) -> Optional[str]: - """需求→品相→门槛→卖家评估并入队;不通过时返回原因(调用方需 commit 后跳过)。""" - if not self._requirement_pass(session, listing, task): - return "需求不匹配" - self._condition_analysis(session, listing, task) - if self._condition_gate_fails(task, listing): - return "品相分低于门槛" - self._seller_check(session, listing, task) - batch_rows.append(self._batch_row(listing, task.condition_requirement or "")) - pending.append((task, listing, old_price, is_renotify)) - return None + finally: + self.guard.finish(task_id) def _upsert_listing(self, session, task: WatchTask, data: ListingData): listing = ( @@ -280,7 +122,7 @@ def _is_blocked(self, session, listing: Listing) -> bool: return True return False - def _fetch_detail(self, session, listing: Listing, force: bool = False) -> None: + def _fetch_detail(self, session, listing: Listing, force: bool = False) -> ListingDetail | None: if not listing.url: return if not force and listing.description: @@ -292,48 +134,28 @@ def _fetch_detail(self, session, listing: Listing, force: bool = False) -> None: return if detail.description: listing.description = detail.description - merged = [u for u in (listing.image_urls or []) if is_product_image(u)] + merged = [] for url in detail.image_urls: if url not in merged and is_product_image(url): merged.append(url) listing.image_urls = merged[:8] - if detail.variants: - listing.variants = detail.variants[:6] + listing.variants = detail.variants[:6] if detail.seller_uid: listing.seller_uid = detail.seller_uid listing.seller_name = detail.seller_name or listing.seller_name listing.seller_risk = { + **(listing.seller_risk or {}), "credit_label": detail.credit_label, "positive_rate": detail.positive_rate, "sold_count": detail.sold_count, } + return detail def reanalyze_listing(self, listing_id: int) -> dict[str, Any]: - """手动重新分析单个商品:重抓详情并重跑需求/品相/性价比/卖家,不发送通知。""" - stats = {"updated": 0} - self._judger = self._build_judger(self.settings_service.get()) - with self._session_factory() as session: - listing = session.get(Listing, listing_id) - if listing is None: - raise RuntimeError(f"商品 {listing_id} 不存在") - task = ( - session.get(WatchTask, listing.task_id) - if listing.task_id is not None - else WatchTask(keyword="", condition_requirement="", min_condition_score=0) - ) - if listing.url: - self._fetch_detail(session, listing, force=True) - self._requirement_pass(session, listing, task) - self._condition_analysis(session, listing, task) - self._seller_check(session, listing, task) - if self._value_client() is not None: - self._batch_value( - session, [self._batch_row(listing, task.condition_requirement or "")] - ) - listing.satisfaction = self._satisfaction(listing) - session.commit() - stats["updated"] = 1 - return stats + """Revalidate offer evidence before analysis, without sending notifications.""" + from goodprice.services.offer_crawl import OfferCrawlRunner + + return OfferCrawlRunner(self).reanalyze(listing_id) def _seller_check(self, session, listing: Listing, task: WatchTask) -> None: if not listing.seller_uid or self.seller_service is None: @@ -430,9 +252,11 @@ def _build_judger(self, settings): ) return judger - def _condition_analysis(self, session, listing: Listing, task: WatchTask) -> None: + def _condition_analysis(self, session, listing: Listing, task: WatchTask, + price: float | None = None) -> None: if not self.vision.enabled: return + listing.condition_score = None valid = [u for u in (listing.image_urls or []) if is_product_image(u)] if not valid: listing.condition_detail = {"error": "无有效商品图"} @@ -442,7 +266,7 @@ def _condition_analysis(self, session, listing: Listing, task: WatchTask) -> Non try: verdict = self.vision.analyze_condition( title=listing.title, - price=listing.price, + price=price if price is not None else listing.price, description=listing.description or "", requirement=task.condition_requirement or "", image_urls=valid, @@ -455,13 +279,6 @@ def _condition_analysis(self, session, listing: Listing, task: WatchTask) -> Non return listing.condition_detail = {"error": str(last_exc)[:200]} - def _condition_gate_fails(self, task: WatchTask, listing: Listing) -> bool: - return bool( - task.min_condition_score - and listing.condition_score is not None - and listing.condition_score < task.min_condition_score - ) - def _excluded(self, task: WatchTask, title: str) -> bool: """排除词按空白/中英文逗号拆分,标题命中任一即跳过。""" words = ( @@ -497,190 +314,6 @@ def _value_client(self): return client return None - def _drop_pct(self, listing: Listing) -> float: - return drop_pct_from_snapshots(listing) - - def _satisfaction(self, listing: Listing) -> float: - return compute_satisfaction( - listing, - vision_enabled=self.vision.enabled, - price_drop_pct=self._drop_pct(listing), - ) - - def _batch_value(self, session, rows: list[dict]) -> None: - now = datetime.now() - judger = self._judger - if judger is not None and judger.enabled: - try: - result = judger.analyze_batch_value(rows[:MAX_BATCH_VALUE_ITEMS]) - except Exception as exc: - logger.warning("Jev 批量性价比失败,回落原路径: %s", exc) - result = None - if result is not None: - scores = result.get("scores") or {} - best = result.get("best") - logger.info( - "Jev 批量性价比完成:%s 个商品,本批最优 %s", len(rows), best - ) - for row in rows: - listing = session.get(Listing, row["listing_id"]) - if listing is None: - continue - score = scores.get(row["external_id"]) - if score is not None: - listing.value_score = score - listing.value_batch_at = now - listing.best_of_batch = best == row["external_id"] - return - client = self._value_client() - if client is None: - return - try: - result = client.analyze_batch_value(rows[:MAX_BATCH_VALUE_ITEMS]) - except Exception as exc: - logger.warning("批量性价比分析失败,跳过: %s", exc) - return - scores = result.get("scores") or {} - best = result.get("best") - logger.info("批量性价比完成:%s 个商品,本批最优 %s", len(rows), best) - for row in rows: - listing = session.get(Listing, row["listing_id"]) - if listing is None: - continue - score = scores.get(row["external_id"]) - if score is not None: - listing.value_score = score - listing.value_batch_at = now - listing.best_of_batch = best == row["external_id"] - - def _backfill(self, session, listing: Listing, task: WatchTask) -> bool: - changed = False - requirement = (task.condition_requirement or "").strip() - if requirement and self.llm.enabled and listing.requirement_match is None: - verdict = self._requirement_verdict(listing, requirement) - if verdict is not None: - listing.requirement_match, listing.requirement_reason = verdict - changed = True - else: - try: - result = self.llm.analyze_requirement( - title=listing.title, - description=listing.description or "", - requirement=requirement, - ) - listing.requirement_match = result["matched"] - listing.requirement_reason = result["reason"] - changed = True - except Exception as exc: - logger.warning("回填需求分析失败: %s", exc) - if self.vision.enabled and listing.condition_score is None: - try: - verdict = self.vision.analyze_condition( - title=listing.title, - price=listing.price, - description=listing.description or "", - requirement=requirement, - image_urls=listing.image_urls, - ) - listing.condition_score = verdict["condition_score"] - listing.condition_detail = verdict - changed = True - except Exception as exc: - logger.warning("回填品相分析失败: %s", exc) - return changed - - def _notify( - self, - session, - task: WatchTask, - listing: Listing, - satisfaction: float, - old_price: Optional[float] = None, - is_renotify: bool = False, - ) -> None: - def _fmt(value: float) -> str: - return format(value, "g") - - lines = [f"价格:{_fmt(listing.price)} 元"] - drop_pct = self._drop_pct(listing) - if drop_pct >= DROP_PCT_DISPLAY_THRESHOLD: - lines.append(f"较首见降价 {drop_pct:.0%}") - if is_renotify and old_price is not None: - lines.append(f"价格更新重推:{_fmt(old_price)} → {_fmt(listing.price)} 元") - if listing.requirement_match is not None: - status = "是" if listing.requirement_match else "否" - reason = listing.requirement_reason or "" - line = f"需求匹配:{status}" - if reason: - line += f"({reason})" - lines.append(line) - if listing.condition_score is not None: - lines.append(f"品相分:{listing.condition_score}") - elif self.vision.enabled: - err = "" - if isinstance(listing.condition_detail, dict): - err = listing.condition_detail.get("error") or "" - lines.append(f"品相分:未评估({err or '分析失败'})") - else: - lines.append("品相分:视觉模型未启用,未评估") - if listing.value_score is not None: - lines.append(f"性价比:{listing.value_score}/10") - else: - lines.append("性价比:未评估") - if listing.best_of_batch: - lines.append("本批最优") - if listing.variants: - parts = " · ".join( - f"{v.get('name')} {_fmt(v.get('price'))} 元" for v in listing.variants[:6] - ) - lines.append(f"规格:{parts}") - if listing.seller_risk: - risk = listing.seller_risk - name = risk.get("nickname") or "卖家" - level = risk.get("risk_level") - reason = risk.get("risk_reason") or "" - rate = risk.get("positive_rate") - rate_txt = f"好评率 {rate * 100:.0f}%" if isinstance(rate, (int, float)) else "" - lines.append(f"卖家:{name} {rate_txt} · 风险{level}({reason})") - if isinstance(listing.condition_detail, dict): - extra = listing.condition_detail.get("reason", "") - if extra: - lines.append(extra) - message = NotificationMessage( - title=f"[{task.keyword}] {listing.title}", - content="\n".join(lines), - url=listing.url, - ) - for channel, notifier in self.notifiers: - try: - notifier.send(message) - logger.info("通知[%s] 已发送:%s", channel, listing.title[:30]) - session.add( - Notification( - listing_id=listing.id, - task_id=task.id, - channel=channel, - status="sent", - title=message.title, - content=message.content, - ) - ) - except Exception as exc: - logger.warning("通知[%s]失败: %s", channel, exc) - session.add( - Notification( - listing_id=listing.id, - task_id=task.id, - channel=channel, - status="failed", - detail=str(exc), - title=message.title, - content=message.content, - ) - ) - listing.notified_at = datetime.now() - listing.last_notified_satisfaction = satisfaction - def _record_error(self, task_id: int, message: str) -> None: with self._session_factory() as session: task = session.get(WatchTask, task_id) diff --git a/goodprice/services/offer_crawl.py b/goodprice/services/offer_crawl.py new file mode 100644 index 0000000..8beb813 --- /dev/null +++ b/goodprice/services/offer_crawl.py @@ -0,0 +1,396 @@ +"""Serial adapter for offer stages; the coordinator will call the same stages. + +Keep page prices unchanged. Only explicit, fresh offer evidence can enter value +analysis and notifications. External fetches happen before observation writes. +""" +import hashlib +import json +import logging +from datetime import datetime, timezone +from types import SimpleNamespace + +import httpx + +from goodprice.models import ( + Listing, + ListingOffer, + Notification, + OfferEvaluationRecord, + OfferObservation, + WatchTask, +) +from goodprice.notify.base import NotificationMessage +from goodprice.offers.pricing import decide_price, offer_fingerprint, utc +from goodprice.offers.service import decode_offer +from goodprice.offers.types import OfferCandidate, PriceClaim +from goodprice.services.offer_pipeline import OfferPipeline +from goodprice.services.satisfaction import compute_satisfaction + +logger = logging.getLogger(__name__) + + +class OfferCrawlRunner: + def __init__(self, crawl): + self.crawl = crawl + self.factory = crawl._session_factory + self.pipeline = OfferPipeline(self.factory, vision_enabled=crawl.vision.enabled) + + def run(self, task_id: int) -> dict: + now = datetime.now(timezone.utc) + stats = dict.fromkeys(("found", "new", "notified", "backfilled", "reevaluated", "gone"), 0) + with self.factory() as session: + task = session.get(WatchTask, task_id) + if task is None: + raise RuntimeError(f"任务 {task_id} 不存在") + if not task.enabled: + return {**stats, "skipped": "disabled"} + task.last_run_at = now + task.last_run_count = (task.last_run_count or 0) + 1 + task.last_error = None + session.commit() + self.crawl._judger = self.crawl._build_judger(self.crawl.settings_service.get()) + self._retry_failed(task, now) + try: + items = self.crawl.adapter.search(task.keyword) + except Exception as exc: + self.crawl._record_error(task_id, f"抓取失败: {exc}") + raise + stats["found"] = len(items) + pending, seen = [], set() + for data in items[:30]: + if self.crawl._excluded(task, data.title): + continue + with self.factory() as session: + listing, is_new = self.crawl._upsert_listing(session, task, data) + was_missing = listing.status != "active" + listing.status, listing.missed_count = "active", 0 + seen.add(listing.id) + blocked = self.crawl._is_blocked(session, listing) + session.commit() + listing_id = listing.id + if blocked: + continue + stats["new"] += int(is_new) + with self.factory() as session: + previous = session.query(OfferObservation).join(ListingOffer).filter( + ListingOffer.listing_id == listing_id).order_by(OfferObservation.id.desc()).first() + old_fingerprint = previous.fingerprint if previous else None + previous_evaluations = {o["key"]: o["evaluation"] for o in self.pipeline.latest(listing_id)} + candidates = self._fetch(listing_id, task) + now = datetime.now(timezone.utc) + changed = is_new or was_missing or len(candidates) > 1 or any( + offer_fingerprint(c) != old_fingerprint for c in candidates) + # Freshness renews each observation; content identity does not. + ids = self.pipeline.evaluate_listing(listing_id, task_id, candidates, now) + with self.factory() as session: + records = [session.get(OfferEvaluationRecord, i) for i in ids] + potential = any(r.payload["state"] == "exact" and r.payload["match"] == "matched" + and r.payload["amount_cents"] is not None + and r.payload["amount_cents"] >= round(task.min_price * 100) + and (not task.max_price or r.payload["amount_cents"] <= round(task.max_price * 100)) + for r in records) + analysis_price = next((r.payload["amount_cents"] / 100 for r in records + if r.payload["state"] == "exact" and r.payload["match"] == "matched"), None) + for candidate, record in zip(candidates, records, strict=False): + old = previous_evaluations.get(candidate.key) + if old is None or any(old.get(k) != record.payload.get(k) for k in ("state", "source", "amount_cents", "task_version")): + changed = True + listing = session.get(Listing, listing_id) + settings = self.crawl.settings_service.get() + inputs = {"description": listing.description, "images": listing.image_urls, + "requirement": task.condition_requirement, "target": task.target_variant, + "vision": self.crawl.vision.enabled, "model": settings.llm_model, + "vision_model": settings.vision_model, "prompt": "offer-v1", + "llm_format": settings.llm_api_format, "vision_format": settings.vision_api_format, + "llm_url": settings.llm_base_url, "vision_url": settings.vision_base_url} + analysis_fp = hashlib.sha256(json.dumps(inputs, sort_keys=True).encode()).hexdigest() + content_fp = hashlib.sha256(json.dumps( + {"description": listing.description, "images": listing.image_urls}, sort_keys=True).encode()).hexdigest() + unchanged_content = all((previous_evaluations.get(c.key) or {}).get("analysis_content_fingerprint") == content_fp + for c in candidates) + needs_backfill = (not changed and unchanged_content and self.crawl.vision.enabled + and listing.condition_score is None) + if any((previous_evaluations.get(c.key) or {}).get("analysis_fingerprint") != analysis_fp + for c in candidates): + changed = True + if needs_backfill: + changed = False + if potential and (changed or needs_backfill): + with self.factory() as session: + listing = session.get(Listing, listing_id) + if self.crawl._requirement_pass(session, listing, task): + if len(candidates) == 1: + self.crawl._condition_analysis(session, listing, task, price=analysis_price) + else: + listing.condition_score = None + listing.condition_detail = {"error": "多规格共享媒体,品相待核验"} + self.crawl._seller_check(session, listing, task) + session.commit() + ids = self.pipeline.evaluate_listing(listing_id, task_id, candidates, now) + stats["reevaluated"] += int(not is_new and changed) + stats["backfilled"] += int(needs_backfill) + for record_id in ids: + with self.factory() as session: + row = session.get(OfferEvaluationRecord, record_id) + row.payload = {**row.payload, "analysis_fingerprint": analysis_fp, + "analysis_content_fingerprint": content_fp} + offer = session.get(ListingOffer, row.offer_id) + listing = session.get(Listing, listing_id) + if row.payload["eligible"] and not changed: + old = previous_evaluations.get(offer.offer_key) or {} + row.payload = {**row.payload, **{key: old[key] for key in + ("value_score", "best_of_batch", "satisfaction") if key in old}} + listing.offer_price_cents = row.payload["amount_cents"] + listing.offer_verified_until = datetime.fromisoformat(row.payload["fresh_until"]) + listing.offer_task_version = task.version + session.commit() + if row.payload["eligible"] and changed: + key = listing.external_id if len(candidates) == 1 else f"{listing.external_id}:{offer.offer_key}" + pending.append((record_id, { + **self.crawl._batch_row(listing, task.condition_requirement), + "external_id": key, "offer_id": offer.id, + "price": row.payload["amount_cents"] / 100, + "title": f"{listing.title} {offer.name} {dict(offer.attributes)}", + })) + elif not row.payload["eligible"]: + listing.value_score, listing.best_of_batch = None, False + listing.offer_verified_until = None + if task.notify_pending_price and row.payload["state"] != "exact" and row.payload["match"] != "mismatched": + self._notify_pending(session, task, listing, offer, row.payload) + session.commit() + session.commit() + self._finish(task, pending, now, stats) + with self.factory() as session: + for listing in session.query(Listing).filter_by(task_id=task.id): + if listing.id not in seen: + listing.missed_count += 1 + if listing.missed_count >= 3: + listing.status = "unseen" # A limited search is not proof of delisting. + session.commit() + stats["deferred"] = max(0, len(items) - 30) + return stats + + def _retry_failed(self, task, now): + notifiers = dict(self.crawl.notifiers) + with self.factory() as session: + rows = session.query(Notification).filter_by(task_id=task.id, status="failed").order_by( + Notification.id.desc()).limit(100).all() + seen = set() + for row in rows: + pair = (row.event_key, row.channel) + ctx = row.event_context or {} + if pair in seen or not row.event_key or row.channel not in notifiers or ctx.get("task_version") != task.version: + continue + seen.add(pair) + with self.factory() as session: + already_sent = session.query(Notification.id).filter_by( + event_key=row.event_key, channel=row.channel, status="sent").first() + attempts = session.query(Notification).filter_by(event_key=row.event_key, channel=row.channel).count() + if already_sent or attempts >= 3: + continue + snapshots = self.pipeline.snapshot(row.listing_id, now) + current = next((s for s in snapshots if s["id"] == row.offer_id), None) + if current is None or current["fingerprint"] != ctx.get("fingerprint"): + continue + if not ctx.get("pending") and (current["price_state"] != "exact" + or current["source"] != ctx.get("source") + or current["effective_price_cents"] != ctx.get("amount_cents")): + continue + status, detail = "sent", None + try: + with self.factory() as session: + listing = session.get(Listing, row.listing_id) + url = listing.url + notifiers[row.channel].send(NotificationMessage(row.title, row.content, url)) + except Exception as exc: + status = "uncertain" if isinstance(exc, (TimeoutError, httpx.TimeoutException)) else "failed" + detail = str(exc)[:500] + with self.factory() as session: + session.add(Notification(listing_id=row.listing_id, task_id=task.id, offer_id=row.offer_id, + event_key=row.event_key, event_context=ctx, channel=row.channel, status=status, + title=row.title, content=row.content, detail=detail)) + session.commit() + + def _notify_pending(self, session, task, listing, offer, payload): + event_key = "pending:" + hashlib.sha256( + f"{task.id}:{task.version}:{offer.id}:{payload['fingerprint']}".encode()).hexdigest() + message = NotificationMessage(f"[待询价] {listing.title}", + f"规格:{offer.name or dict(offer.attributes) or '待核验'}\n{payload['reason']}\n" + "价格尚未确认,不参与正常性价比排名;请自行联系卖家核实。", listing.url) + for channel, notifier in self.crawl.notifiers: + if session.query(Notification).filter_by(event_key=event_key, channel=channel, status="sent").first(): + continue + status, detail = "sent", None + try: + notifier.send(message) + except Exception as exc: + status, detail = "failed", str(exc)[:500] + session.add(Notification(listing_id=listing.id, task_id=task.id, offer_id=offer.id, + event_key=event_key, channel=channel, status=status, detail=detail, + event_context={"task_version": task.version, "fingerprint": payload["fingerprint"], "pending": True}, + title=message.title, content=message.content)) + + def _fetch(self, listing_id, task): + detail = None + if task.fetch_detail: + with self.factory() as session: + listing = session.get(Listing, listing_id) + detail = self.crawl._fetch_detail(session, listing, force=True) + session.commit() + if detail and detail.offers: + return detail.offers + if detail is None: + # Failure is not a new observation. Reuse original timestamps so a + # transient fault neither revokes a fresh quote nor renews stale data. + with self.factory() as session: + rows = session.query(OfferObservation).join(ListingOffer).filter( + ListingOffer.listing_id == listing_id).order_by(OfferObservation.id.desc()).all() + cached = {} + for row in rows: + cached.setdefault(row.offer_id, decode_offer(row.payload)) + if cached: + return list(cached.values()) + return [OfferCandidate("default", claims=(PriceClaim( + "dom", "unknown", "详情价格未核验", datetime.now(timezone.utc), "detail:missing"),))] + + def reanalyze(self, listing_id): + with self.factory() as session: + listing = session.get(Listing, listing_id) + if listing is None: + raise RuntimeError(f"商品 {listing_id} 不存在") + task = session.get(WatchTask, listing.task_id) if listing.task_id else None + if task is None: + raise RuntimeError("原监控任务已删除,无法按旧条件重新分析") + candidates = self._fetch(listing_id, task) + now = datetime.now(timezone.utc) + ids = self.pipeline.evaluate_listing(listing_id, task.id, candidates, now) + with self.factory() as session: + listing = session.get(Listing, listing_id) + potential = any(session.get(OfferEvaluationRecord, i).payload["state"] == "exact" for i in ids) + analysis_price = next((session.get(OfferEvaluationRecord, i).payload["amount_cents"] / 100 + for i in ids if session.get(OfferEvaluationRecord, i).payload["state"] == "exact"), None) + if potential and self.crawl._requirement_pass(session, listing, task): + if len(candidates) == 1: + self.crawl._condition_analysis(session, listing, task, price=analysis_price) + else: + listing.condition_score = None + listing.condition_detail = {"error": "多规格共享媒体,品相待核验"} + self.crawl._seller_check(session, listing, task) + if not potential: + listing.value_score, listing.best_of_batch = None, False + session.commit() + ids = self.pipeline.evaluate_listing(listing_id, task.id, candidates, datetime.now(timezone.utc)) + pending = [] + with self.factory() as session: + listing = session.get(Listing, listing_id) + for record_id in ids: + row = session.get(OfferEvaluationRecord, record_id) + if row.payload["eligible"]: + offer = session.get(ListingOffer, row.offer_id) + pending.append((record_id, {**self.crawl._batch_row(listing, task.condition_requirement), + "external_id": listing.external_id if len(candidates) == 1 else f"{listing.external_id}:{offer.offer_key}", + "offer_id": offer.id, "price": row.payload["amount_cents"] / 100})) + self.crawl._judger = self.crawl._build_judger(self.crawl.settings_service.get()) + self._finish(task, pending, now, {"notified": 0}, notify=False) + return {"updated": 1} + + def _finish(self, task, pending, now, stats, notify=True): + result = {"scores": {}, "best": None, "reasons": {}} + client = self.crawl._value_client() + judger = self.crawl._judger + typed = None + if pending and judger is not None and judger.enabled: + try: + typed = judger.analyze_batch_value([r for _, r in pending]) + except Exception as exc: + logger.warning("规格 Jev 性价比失败,回落原模型: %s", exc) + if typed is not None: + result = typed + elif client and pending: + try: + result = client.analyze_batch_value([r for _, r in pending]) + except Exception as exc: + logger.warning("规格性价比分析失败: %s", exc) + for record_id, item in pending: + with self.factory() as session: + current_task = session.get(WatchTask, task.id) + if current_task is None or not current_task.enabled or current_task.version != task.version: + continue + record = session.get(OfferEvaluationRecord, record_id) + listing = session.get(Listing, item["listing_id"]) + offer = session.get(ListingOffer, item["offer_id"]) + if record is None or listing is None or offer is None: + continue + payload = dict(record.payload) + if payload["fresh_until"] and datetime.fromisoformat(payload["fresh_until"]) <= datetime.now(timezone.utc): + continue + value = result.get("scores", {}).get(item["external_id"]) + best = result.get("best") == item["external_id"] + first = session.query(OfferObservation).filter_by(offer_id=offer.id).order_by(OfferObservation.id).first() + first_price = decide_price(decode_offer(first.payload), None, utc(first.observed_at), task.interval_minutes * 60) + drop = 0.0 + if first_price.source == payload["source"] and first_price.amount_cents: + drop = max(0.0, (first_price.amount_cents - payload["amount_cents"]) / first_price.amount_cents) + score = compute_satisfaction(SimpleNamespace(requirement_match=listing.requirement_match, + condition_score=listing.condition_score, value_score=value, seller_risk=listing.seller_risk), + vision_enabled=self.crawl.vision.enabled, price_drop_pct=drop) + payload.update(value_score=value, best_of_batch=best, satisfaction=score) + record.payload = payload + listing.value_score, listing.best_of_batch, listing.satisfaction = value, best, score + listing.offer_price_cents = payload["amount_cents"] + listing.offer_verified_until = datetime.fromisoformat(payload["fresh_until"]) + listing.offer_task_version = task.version + if not notify: + session.commit() + continue + if offer.last_notified_source == "legacy_baseline": + offer.last_notified_score, offer.last_notified_amount = score, payload["amount_cents"] + offer.last_notified_source = payload["source"] + session.commit() + continue + if offer.last_notified_score is not None and score <= offer.last_notified_score: + session.commit() + continue + event_key = hashlib.sha256(f"{task.id}:{task.version}:{offer.id}:{payload['fingerprint']}:{score}".encode()).hexdigest() + previous_amount = offer.last_notified_amount + renotify = (previous_amount is not None and offer.last_notified_source == payload["source"] + and previous_amount != payload["amount_cents"]) + lines = [f"规格:{offer.name or dict(offer.attributes) or '默认商品'}", + f"价格:{payload['amount_cents'] / 100:g} 元({'人工报价' if payload['source'] == 'manual' else '页面确价'})", + f"需求匹配:{'否' if listing.requirement_match is False else '是' if listing.requirement_match else '未核验'}", + f"品相分:{listing.condition_score if listing.condition_score is not None else '未评估'}" + + ("(视觉模型未启用)" if not self.crawl.vision.enabled else ""), + f"性价比:{str(value) + '/10' if value is not None else '未评估'}"] + if renotify: + lines.append(f"价格更新重推:{previous_amount / 100:g} → {payload['amount_cents'] / 100:g} 元") + if best: + lines.append(f"本批最优(本次已就绪 {len(pending)} 个规格)") + if listing.seller_risk: + risk = listing.seller_risk + rate = risk.get("positive_rate") + lines.append(f"卖家:{risk.get('nickname', '卖家')} · 风险{risk.get('risk_level', '未知')} " + + (f"好评率 {rate * 100:.0f}%" if isinstance(rate, (int, float)) else "")) + message = NotificationMessage(f"[{task.keyword}] {listing.title}", "\n".join(lines), listing.url) + success = False + for channel, notifier in self.crawl.notifiers: + if session.query(Notification).filter_by(event_key=event_key, channel=channel, status="sent").first(): + success = True + continue + status, error = "sent", None + try: + notifier.send(message) + success = True + except Exception as exc: + status = "uncertain" if isinstance(exc, (TimeoutError, httpx.TimeoutException)) else "failed" + error = str(exc)[:500] + session.add(Notification(listing_id=listing.id, task_id=task.id, offer_id=offer.id, + event_key=event_key, channel=channel, status=status, detail=error, + event_context={"task_version": task.version, "fingerprint": payload["fingerprint"], + "source": payload["source"], "amount_cents": payload["amount_cents"]}, + title=message.title, content=message.content)) + if success: + offer.last_notified_score, offer.last_notified_amount = score, payload["amount_cents"] + offer.last_notified_source = payload["source"] + listing.notified_at, listing.last_notified_satisfaction = now, score + stats["notified"] += 1 + session.commit() diff --git a/goodprice/services/offer_pipeline.py b/goodprice/services/offer_pipeline.py new file mode 100644 index 0000000..32187da --- /dev/null +++ b/goodprice/services/offer_pipeline.py @@ -0,0 +1,132 @@ +"""Offer decisions shared by serial execution and the future stage coordinator.""" +from dataclasses import asdict +from datetime import datetime + +from goodprice.models import ( + Listing, + ListingOffer, + ManualQuote, + OfferEvaluationRecord, + OfferObservation, + WatchTask, +) +from goodprice.offers.matching import match_variant +from goodprice.offers.pricing import decide_price, utc +from goodprice.offers.service import OfferService +from goodprice.offers.types import OfferCandidate, QuoteInput + + +class OfferPipeline: + def __init__(self, session_factory, vision_enabled: bool = True): + self._session_factory = session_factory + self.vision_enabled = vision_enabled + self.offers = OfferService(session_factory) + + def evaluate_listing(self, listing_id: int, task_id: int, + candidates: list[OfferCandidate], now: datetime) -> list[int]: + # Network/AI callers must finish their work before this short write phase. + ids = [self.offers.record_observation(listing_id, o) for o in candidates[:20]] + result = [] + with self._session_factory() as session: + listing, task = session.get(Listing, listing_id), session.get(WatchTask, task_id) + if listing is None or task is None or listing.task_id != task.id: + raise LookupError("任务或商品不存在/关联不一致") + for candidate, offer_id in zip(candidates, ids, strict=False): + quote = session.query(ManualQuote).filter_by(offer_id=offer_id, revoked=False).order_by( + ManualQuote.id.desc()).first() + value = None if quote is None else QuoteInput(candidate.key, quote.amount_cents, + utc(quote.confirmed_at), utc(quote.expires_at), quote.evidence_fingerprint, quote.revoked) + decision = decide_price(candidate, value, now, task.interval_minutes * 60) + match = match_variant(task.target_variant, candidate.attributes, listing.title) + eligible = (match == "matched" and decision.state == "exact" + and candidate.availability != "unavailable") + reason = decision.reason if not eligible else "" + if decision.amount_cents is not None: + amount = decision.amount_cents + if amount < round(task.min_price * 100) or (task.max_price and amount > round(task.max_price * 100)): + eligible, reason = False, "该规格价格不在预算内" + if match != "matched": + eligible, reason = False, "目标规格不匹配" if match == "mismatched" else "规格待核验" + if task.min_condition_score and (not self.vision_enabled or len(candidates) != 1 or listing.condition_score is None + or listing.condition_score < task.min_condition_score): + eligible, reason = False, "品相未核验或低于门槛" + if listing.requirement_match is False or listing.blocked: + eligible, reason = False, "需求不匹配或商品已拉黑" + payload = {**asdict(decision), "match": match, "eligible": eligible, "reason": reason} + payload["fresh_until"] = decision.fresh_until.isoformat() if decision.fresh_until else None + latest = session.query(OfferObservation).filter_by(offer_id=offer_id).order_by( + OfferObservation.id.desc()).first() + payload["fingerprint"] = latest.fingerprint + payload["observation_id"] = latest.id + payload["task_version"] = task.version + row = OfferEvaluationRecord(offer_id=offer_id, task_version=task.version, + payload=payload, evaluated_at=now) + session.add(row) + session.flush() + result.append(row.id) + session.commit() + return result + + def latest(self, listing_id: int) -> list[dict]: + with self._session_factory() as session: + result = [] + for offer in session.query(ListingOffer).filter_by(listing_id=listing_id).order_by(ListingOffer.id): + row = session.query(OfferEvaluationRecord).filter_by(offer_id=offer.id).order_by( + OfferEvaluationRecord.id.desc()).first() + result.append({"id": offer.id, "key": offer.offer_key, "name": offer.name, + "attributes": offer.attributes, "availability": offer.availability, + "evaluation": row.payload if row else None}) + return result + + def snapshot(self, listing_id: int, now: datetime) -> list[dict]: + """Read-time validity: stale stored recommendations must not stay eligible.""" + from goodprice.offers.service import decode_offer + + result = self.latest(listing_id) + with self._session_factory() as session: + listing = session.get(Listing, listing_id) + if listing is None: + raise LookupError("商品不存在") + task = session.get(WatchTask, listing.task_id) if listing.task_id else None + for item in result: + observation = session.query(OfferObservation).filter_by(offer_id=item["id"]).order_by( + OfferObservation.observed_at.desc(), OfferObservation.id.desc()).first() + quote = session.query(ManualQuote).filter_by(offer_id=item["id"], revoked=False).order_by(ManualQuote.id.desc()).first() + candidate = decode_offer(observation.payload) + value = None if quote is None else QuoteInput(candidate.key, quote.amount_cents, + utc(quote.confirmed_at), utc(quote.expires_at), quote.evidence_fingerprint, quote.revoked) + decision = decide_price(candidate, value, now, (task.interval_minutes if task else 20) * 60) + item.update(fingerprint=observation.fingerprint, price_state=decision.state, + effective_price_cents=decision.amount_cents, source=decision.source, + fresh_until=decision.fresh_until.isoformat() if decision.fresh_until else None, + reason=decision.reason, quote_id=quote.id if quote else None) + evaluation = dict(item.get("evaluation") or {}) + valid = (task is not None and task.enabled and not listing.blocked + and decision.state == "exact" + and evaluation.get("task_version") == task.version + and evaluation.get("fingerprint") == observation.fingerprint + and evaluation.get("source") == decision.source + and evaluation.get("amount_cents") == decision.amount_cents + and (not task.min_condition_score or self.vision_enabled)) + if not valid: + evaluation.update(eligible=False, value_score=None, best_of_batch=False, satisfaction=None) + item["evaluation"] = evaluation + return result + + +def present_listings(listings, factory, vision_enabled=True): + from types import SimpleNamespace + + pipeline = OfferPipeline(factory, vision_enabled=vision_enabled) + result = [] + for listing in listings: + view = SimpleNamespace(**{col.name: getattr(listing, col.name) for col in Listing.__table__.columns}) + offers = pipeline.snapshot(listing.id, datetime.now().astimezone()) + eligible = [o for o in offers if (o.get("evaluation") or {}).get("eligible")] + exact = [o for o in offers if o["price_state"] == "exact"] + view.price_confirmed = bool(eligible) + view.effective_price_cents = min((o["effective_price_cents"] for o in exact), default=None) + if not eligible: + view.value_score, view.best_of_batch, view.satisfaction = None, False, 0.0 + result.append(view) + return result diff --git a/goodprice/web/offer_routes.py b/goodprice/web/offer_routes.py new file mode 100644 index 0000000..31ada68 --- /dev/null +++ b/goodprice/web/offer_routes.py @@ -0,0 +1,64 @@ +from datetime import datetime, timedelta, timezone + +from fastapi import APIRouter, HTTPException, Request +from fastapi.responses import RedirectResponse +from pydantic import BaseModel, Field, ValidationError + +from goodprice.models import ListingOffer +from goodprice.offers.pricing import parse_money_cents +from goodprice.offers.service import OfferService +from goodprice.offers.types import QuoteInput +from goodprice.services.offer_pipeline import OfferPipeline + +router = APIRouter() + + +class QuoteForm(BaseModel): + amount: str = Field(min_length=1, max_length=40) + fingerprint: str = Field(min_length=1, max_length=64) + valid_days: int = Field(default=7, ge=1, le=30) + + +@router.get("/api/listings/{listing_id}/offers") +def list_offers(request: Request, listing_id: int): + try: + return OfferPipeline(request.app.state.session_factory).snapshot(listing_id, datetime.now(timezone.utc)) + except LookupError as exc: + raise HTTPException(404, str(exc)) from exc + + +@router.post("/offers/{offer_id}/quotes") +async def confirm_quote(request: Request, offer_id: int): + factory = request.app.state.session_factory + with factory() as session: + offer = session.get(ListingOffer, offer_id) + if offer is None: + raise HTTPException(404, "规格不存在") + key, listing_id = offer.offer_key, offer.listing_id + json_request = "application/json" in request.headers.get("content-type", "") + try: + form = QuoteForm.model_validate(await request.json() if json_request else dict(await request.form())) + now = datetime.now(timezone.utc) + row_id = OfferService(factory).confirm_quote(offer_id, QuoteInput( + key, parse_money_cents(form.amount), now, now + timedelta(days=form.valid_days), form.fingerprint)) + except (ValidationError, ValueError, TypeError) as exc: + raise HTTPException(422, "报价无效或价格证据已变化,请刷新后确认") from exc + if json_request: + return {"id": row_id} + return RedirectResponse(f"/listings/{listing_id}", status_code=303) + + +@router.post("/quotes/{quote_id}/revoke") +async def revoke_quote(request: Request, quote_id: int): + from goodprice.models import ManualQuote + + factory = request.app.state.session_factory + with factory() as session: + quote = session.get(ManualQuote, quote_id) + if quote is None: + raise HTTPException(404, "报价不存在") + listing_id = session.get(ListingOffer, quote.offer_id).listing_id + OfferService(factory).revoke_quote(quote_id) + if "application/json" in request.headers.get("content-type", ""): + return {"revoked": True} + return RedirectResponse(f"/listings/{listing_id}", status_code=303) diff --git a/goodprice/web/routes.py b/goodprice/web/routes.py index 591be51..e0e5455 100644 --- a/goodprice/web/routes.py +++ b/goodprice/web/routes.py @@ -8,7 +8,7 @@ from fastapi.responses import HTMLResponse, RedirectResponse from fastapi.templating import Jinja2Templates from pydantic import BaseModel, Field, field_validator, model_validator -from sqlalchemy import func, text +from sqlalchemy import and_, case, func, select, text from goodprice.constants import SCORE_MAX @@ -30,7 +30,9 @@ def _apply_listing_filters(query, *, task_id, show, sort): show 的取值由模板下拉决定;未知值(含 "all")不追加过滤条件。 """ - from goodprice.models import Listing + from datetime import datetime, timezone + + from goodprice.models import Listing, WatchTask if task_id: query = query.filter(Listing.task_id == task_id) @@ -40,13 +42,18 @@ def _apply_listing_filters(query, *, task_id, show, sort): query = query.filter(Listing.status == "gone") elif show == "blocked": query = query.filter(Listing.blocked.is_(True)) + version = select(WatchTask.version).where(WatchTask.id == Listing.task_id).correlate(Listing).scalar_subquery() + valid = and_(Listing.offer_verified_until > datetime.now(timezone.utc), + Listing.offer_price_cents.is_not(None), Listing.offer_task_version == version) + query = query.order_by(case((valid, 0), else_=1)) + price = case((valid, Listing.offer_price_cents), else_=None) if sort == "price_asc": - return query.order_by(Listing.price.asc()) + return query.order_by(price.asc(), Listing.id.desc()) if sort == "price_desc": - return query.order_by(Listing.price.desc()) + return query.order_by(price.desc(), Listing.id.desc()) if sort == "newest": return query.order_by(Listing.first_seen_at.desc()) - return query.order_by(Listing.satisfaction.desc(), Listing.first_seen_at.desc()) + return query.order_by(case((valid, Listing.satisfaction), else_=0).desc(), Listing.first_seen_at.desc(), Listing.id.desc()) @router.get("/healthz", include_in_schema=False) @@ -278,6 +285,8 @@ def api_list_notifications(request: Request): @router.post("/tasks") def create_task_form( request: Request, + target_variant: str = Form(""), + notify_pending_price: Optional[int] = Form(None), keyword: str = Form(...), name: str = Form(""), max_price: float = Form(0), @@ -293,6 +302,8 @@ def create_task_form( try: task_service.create_task( { + "target_variant": target_variant, + "notify_pending_price": bool(notify_pending_price), "keyword": keyword, "name": name, "max_price": max_price, @@ -360,6 +371,8 @@ def task_detail_page(request: Request, task_id: int): def edit_task_form( request: Request, task_id: int, + target_variant: str = Form(""), + notify_pending_price: Optional[int] = Form(None), keyword: str = Form(...), name: str = Form(""), max_price: float = Form(0), @@ -376,6 +389,8 @@ def edit_task_form( task_service.update_task( task_id, { + "target_variant": target_variant, + "notify_pending_price": bool(notify_pending_price), "keyword": keyword, "name": name, "max_price": max_price, @@ -438,6 +453,10 @@ def listings_page( query = _apply_listing_filters(query, task_id=task_id_int, show=show, sort=sort) query = query.offset(offset) listings = query.limit(PAGE_SIZE).all() + from goodprice.services.offer_pipeline import present_listings + + listings = present_listings(listings, request.app.state.session_factory, + request.app.state.settings_service.get().vision_enabled) notify_counts = _notify_counts(session, listings) more = len(listings) >= PAGE_SIZE partial_url = f"/listings?partial=1&task_id={task_id or ''}&sort={sort}&show={show}" @@ -502,6 +521,10 @@ def listings_more( task_id_int = _optional_int(task_id) query = _apply_listing_filters(query, task_id=task_id_int, show=show, sort=sort) listings = query.offset(offset).limit(PAGE_SIZE).all() + from goodprice.services.offer_pipeline import present_listings + + listings = present_listings(listings, request.app.state.session_factory, + request.app.state.settings_service.get().vision_enabled) notify_counts = _notify_counts(session, listings) return templates.TemplateResponse( request, "listings_cards.html", {"listings": listings, "notify_counts": notify_counts} @@ -510,6 +533,11 @@ def listings_more( @router.get("/listings/{listing_id}", response_class=HTMLResponse) def listing_detail_page(request: Request, listing_id: int): + from datetime import datetime as dt + from datetime import timezone + + from goodprice.services.offer_pipeline import OfferPipeline, present_listings + with request.app.state.session_factory() as session: from goodprice.models import Listing, Notification, PriceSnapshot @@ -540,13 +568,15 @@ def listing_detail_page(request: Request, listing_id: int): request, "listings_detail.html", { - "listing": listing, + "listing": present_listings([listing], request.app.state.session_factory, + request.app.state.settings_service.get().vision_enabled)[0], "snapshots": snapshots, "notifications": notifications, "notify_count": notify_count, "first_price": first_price, "drop_pct": drop_pct, "drop_pct_threshold": DROP_PCT_DISPLAY_THRESHOLD, + "offers": OfferPipeline(request.app.state.session_factory).snapshot(listing_id, dt.now(timezone.utc)), "active": "listings", }, ) @@ -981,6 +1011,10 @@ def api_list_listings( task_id_int = _optional_int(task_id) query = _apply_listing_filters(query, task_id=task_id_int, show=show, sort=sort) rows = query.offset(offset).limit(PAGE_SIZE).all() + from goodprice.services.offer_pipeline import present_listings + + rows = present_listings(rows, request.app.state.session_factory, + request.app.state.settings_service.get().vision_enabled) return [ { "id": row.id, @@ -988,6 +1022,9 @@ def api_list_listings( "external_id": row.external_id, "title": row.title, "price": row.price, + "effective_price_cents": row.effective_price_cents, + "price_confirmed": row.price_confirmed, + "value_score": row.value_score, "url": row.url, "image_urls": row.image_urls, "condition_score": row.condition_score, diff --git a/goodprice/web/templates/listings_cards.html b/goodprice/web/templates/listings_cards.html index db5f0f5..a9a31ca 100644 --- a/goodprice/web/templates/listings_cards.html +++ b/goodprice/web/templates/listings_cards.html @@ -7,9 +7,9 @@ {% endif %} {{ item.title }}
- ¥{{ item.price }} + 页面标价 ¥{{ item.price }} {% if item.status == 'gone' %}已下架{% endif %} - {% if item.best_of_batch %}本批最优{% endif %} + {% if item.price_confirmed and item.best_of_batch %}本批最优{% endif %} {% if item.condition_score %} 品相 {{ item.condition_score }}/10 {% endif %} diff --git a/goodprice/web/templates/listings_detail.html b/goodprice/web/templates/listings_detail.html index 48766df..b240043 100644 --- a/goodprice/web/templates/listings_detail.html +++ b/goodprice/web/templates/listings_detail.html @@ -9,9 +9,9 @@

{{ listing.title }}

- ¥{{ listing.price }} + 页面标价 ¥{{ listing.price }} {% if listing.status == 'gone' %}已下架{% endif %} - {% if listing.best_of_batch %}本批最优{% endif %} + {% if listing.price_confirmed and listing.best_of_batch %}本批最优{% endif %} {% if notify_count > 1 %}已重推{% endif %} {% if listing.blocked %}已拉黑{% endif %} 满足分 {{ listing.satisfaction|round(1) }} @@ -33,10 +33,29 @@

{{ listing.title }}

点击图片可放大查看
{% endif %} +

规格与价格证据(实验性)

+ {% for offer in offers %} +
+ {{ offer.name or offer.key }} {{ offer.attributes }} +

{{ offer.reason }} · {{ offer.availability }}

+ {% if offer.effective_price_cents is not none %} +

当前{{ '人工报价' if offer.source == 'manual' else '页面确价' }}:¥{{ offer.effective_price_cents / 100 }}

+ {% else %}

价格待确认,不参与正常性价比排名。

{% endif %} +
+ + + + +
+

人工报价优先于页面价;页面证据变化或报价到期后需复核,不会自动联系卖家。

+ {% if offer.quote_id %}
{% endif %} +
+ {% else %}

尚无已核验的规格价格,旧页面标价不代表实际成交价。

{% endfor %} +

描述原文

{{ listing.description or '无描述' }}
-

价格走势

+

页面标价走势(不是目标规格成交价)

首见价 ¥{{ first_price }} · 当前价 ¥{{ listing.price }}{% if drop_pct > 0 %} · 降幅 {{ '%.1f'|format(drop_pct * 100) }}%{% endif %}
diff --git a/goodprice/web/templates/tasks.html b/goodprice/web/templates/tasks.html index f9648ab..ea4012b 100644 --- a/goodprice/web/templates/tasks.html +++ b/goodprice/web/templates/tasks.html @@ -1,6 +1,7 @@ {% extends "base.html" %} {% block title %}监控任务 - 闲鱼盯价助手{% endblock %} {% block content %} +

创建后可在任务详情设置目标型号/容量与待询价提醒。页面标价不代表具体规格的实际售价。

{% if just_ran %}
已开始执行任务 #{{ just_ran }},页面将自动刷新显示进度;完成后可查看「上次运行」与「命中列表」。 diff --git a/goodprice/web/templates/tasks_detail.html b/goodprice/web/templates/tasks_detail.html index 90b5f18..e7d9041 100644 --- a/goodprice/web/templates/tasks_detail.html +++ b/goodprice/web/templates/tasks_detail.html @@ -52,6 +52,9 @@

任务参数

+ + +
diff --git a/tests/test_api.py b/tests/test_api.py index e978f8d..a44d00f 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -211,12 +211,21 @@ def test_listings_filter_by_task(base_settings, session_factory): def test_listings_sort(base_settings, session_factory): + from test_crawl_service import FakeAdapter, _item, _service + + from goodprice.services.task_service import TaskService + client = _client(base_settings, session_factory) + task = TaskService(session_factory).create_task({"keyword": "sort"}) + a, b = _item("1", 100), _item("2", 10) + a.title, b.title = "a", "b" + crawl, _, _ = _service(session_factory, base_settings, adapter=FakeAdapter([a, b])) + crawl.run_task(task.id) with session_factory() as session: from goodprice.models import Listing - session.add(Listing(platform="xianyu", external_id="1", title="a", price=100, url="u", satisfaction=20)) - session.add(Listing(platform="xianyu", external_id="2", title="b", price=10, url="v", satisfaction=90)) + session.query(Listing).filter_by(title="a").one().satisfaction = 20 + session.query(Listing).filter_by(title="b").one().satisfaction = 90 session.commit() assert [d["title"] for d in client.get("/api/listings?sort=satisfaction").json()] == ["b", "a"] assert [d["title"] for d in client.get("/api/listings?sort=price_asc").json()] == ["b", "a"] @@ -339,7 +348,7 @@ def test_listing_detail_page_shows_analysis_blocks(base_settings, session_factor assert "需求匹配" in page.text and "符合买家要求" in page.text assert "品相" in page.text and "轻微使用痕迹" in page.text assert "性价比" in page.text and "8/10" in page.text - assert "本批最优" in page.text + assert "本批最优" not in page.text # 旧分析缺少规格/确价证据,不继续冒充当前推荐。 assert "兵哥哥" in page.text and "风险低" in page.text assert "规格" in page.text and "最低价" in page.text and "580.0" in page.text assert "首见价" in page.text and "降幅" in page.text diff --git a/tests/test_crawl_service.py b/tests/test_crawl_service.py index f649f7f..3cb4453 100644 --- a/tests/test_crawl_service.py +++ b/tests/test_crawl_service.py @@ -21,9 +21,16 @@ def search(self, keyword): def fetch_detail(self, url): self.fetch_calls.append(url) + from datetime import datetime, timezone + from goodprice.crawler.base import ListingDetail + from goodprice.offers.types import OfferCandidate, PriceClaim - return ListingDetail(description="屏幕完好 电池健康", image_urls=["https://x/d.jpg"]) + item = next((i for i in self.items if i.url == url), None) + offers = [] if item is None else [OfferCandidate("default", claims=(PriceClaim( + "dom", "exact", f"{item.price:g}", datetime.now(timezone.utc), "fixture:price", + amount_cents=round(item.price * 100)),))] + return ListingDetail(description="屏幕完好 电池健康", image_urls=["https://img.alicdn.com/bao/uploaded/d.jpg"], offers=offers) class FakeLLM: @@ -131,6 +138,7 @@ def fetch_detail(self, url): from goodprice.crawler.base import ListingDetail return ListingDetail( + offers=super().fetch_detail(url).offers, description="屏幕完好", image_urls=["https://img.alicdn.com/bao/uploaded/d.jpg"], seller_uid="2672367114", @@ -266,7 +274,7 @@ def test_gone_after_three_misses_and_reappear_reevaluates(session_factory, base_ crawl_empty.run_task(task.id) with session_factory() as session: listing = session.query(Listing).one() - assert listing.status == "gone" + assert listing.status == "unseen" # 有限搜索未见,不是详情证实下架。 assert listing.missed_count == 3 improved = FakeVision( @@ -355,8 +363,7 @@ def fetch_detail(self, url): {"name": "最低价", "price": 850.0}, {"name": "最高价", "price": 1299.0}, ] - assert "最低价 850" in notifier.messages[0].content - assert "最高价 1299" in notifier.messages[0].content + assert not notifier.messages # 区间不是可购买 SKU 的确价。 def test_batch_row_carries_requirement(session_factory, base_settings): @@ -477,6 +484,7 @@ def fetch_detail(self, url): from goodprice.crawler.base import ListingDetail return ListingDetail( + offers=super().fetch_detail(url).offers, description="无图", image_urls=["https://img.alicdn.com/imgextra/xx-2-tps-2-2.png"], ) @@ -569,20 +577,20 @@ def test_price_filter(session_factory, base_settings): task = TaskService(session_factory).create_task({"keyword": "k", "max_price": "50"}) crawl, notifier, _ = _service(session_factory, base_settings, adapter=FakeAdapter([_item(price=100.0)])) stats = crawl.run_task(task.id) - assert stats["new"] == 0 + assert stats["new"] == 1 # 已抓取证据保留,预算不匹配不推送。 assert stats["notified"] == 0 with session_factory() as session: - assert session.query(Listing).count() == 0 + assert session.query(Listing).count() == 1 def test_min_price_filter(session_factory, base_settings): task = TaskService(session_factory).create_task({"keyword": "k", "min_price": "50"}) crawl, notifier, _ = _service(session_factory, base_settings, adapter=FakeAdapter([_item(price=30.0)])) stats = crawl.run_task(task.id) - assert stats["new"] == 0 + assert stats["new"] == 1 assert stats["notified"] == 0 with session_factory() as session: - assert session.query(Listing).count() == 0 + assert session.query(Listing).count() == 1 def test_exclude_words_filter(session_factory, base_settings): @@ -678,7 +686,7 @@ def fetch_detail(self, url): crawl, notifier, _ = _service(session_factory, base_settings, adapter=BrokenAdapter([_item()])) stats = crawl.run_task(task.id) - assert stats["notified"] == 1 + assert stats["notified"] == 0 # 失败标价不冒充已核验价格。 def test_backfill_fills_missing_analysis_without_renotify(session_factory, base_settings): diff --git a/tests/test_judge.py b/tests/test_judge.py index b6ce08d..dc182ba 100644 --- a/tests/test_judge.py +++ b/tests/test_judge.py @@ -256,8 +256,10 @@ def search(self, keyword): def fetch_detail(self, url): from goodprice.crawler.base import ListingDetail + from goodprice.crawler.offers import parse_offer_candidates - return ListingDetail(description="屏幕完好 电池健康", image_urls=["https://x/d.jpg"]) + return ListingDetail(description="屏幕完好 电池健康", image_urls=["https://x/d.jpg"], + offers=parse_offer_candidates('
100
')) class FakeVision: diff --git a/tests/test_offer_api.py b/tests/test_offer_api.py new file mode 100644 index 0000000..77c9ab9 --- /dev/null +++ b/tests/test_offer_api.py @@ -0,0 +1,55 @@ +from datetime import datetime, timezone + +from fastapi.testclient import TestClient + +from goodprice.crawler.offers import parse_offer_candidates +from goodprice.main import build_app +from goodprice.models import Listing, ManualQuote +from goodprice.services.offer_pipeline import OfferPipeline +from goodprice.services.task_service import TaskService + + +def setup_client(base_settings, session_factory): + task = TaskService(session_factory).create_task({"keyword": "x"}) + with session_factory() as session: + listing = Listing(platform="xianyu", external_id="e", title="t", price=1, + url="https://invalid.test", task_id=task.id) + session.add(listing) + session.commit() + listing_id = listing.id + OfferPipeline(session_factory).evaluate_listing(listing_id, task.id, + parse_offer_candidates('
1500
'), + datetime.now(timezone.utc)) + return TestClient(build_app(base_settings, session_factory, with_scheduler=False)), listing_id + + +def test_offers_api_and_manual_quote_flow(base_settings, session_factory): + client, listing_id = setup_client(base_settings, session_factory) + response = client.get(f"/api/listings/{listing_id}/offers") + assert response.status_code == 200 + offer = response.json()[0] + assert offer["effective_price_cents"] == 150000 + saved = client.post(f"/offers/{offer['id']}/quotes", json={ + "amount": "1400", "fingerprint": offer["fingerprint"], "valid_days": 7}) + assert saved.status_code == 200 + quote_id = saved.json()["id"] + new = client.get(f"/api/listings/{listing_id}/offers").json()[0] + assert new["effective_price_cents"] == 140000 + assert new["source"] == "manual" + assert client.post(f"/quotes/{quote_id}/revoke", json={}).status_code == 200 + assert client.get(f"/api/listings/{listing_id}/offers").json()[0]["effective_price_cents"] == 150000 + page = client.get(f"/listings/{listing_id}") + assert "页面标价" in page.text + assert "确认报价" in page.text + + +def test_bad_quote_does_not_commit_and_missing_offer_is_404(base_settings, session_factory): + client, listing_id = setup_client(base_settings, session_factory) + offer = client.get(f"/api/listings/{listing_id}/offers").json()[0] + for payload in ({"amount": "NaN"}, {"amount": "1400", "fingerprint": "stale"}, + {"amount": "1400", "fingerprint": offer["fingerprint"], "valid_days": 0}): + assert client.post(f"/offers/{offer['id']}/quotes", json=payload).status_code == 422 + with session_factory() as session: + assert session.query(ManualQuote).count() == 0 + assert client.post("/offers/999/quotes", json={"amount": "100", "fingerprint": "x"}).status_code == 404 + assert client.get("/api/listings/999/offers").status_code == 404 diff --git a/tests/test_offer_identity.py b/tests/test_offer_identity.py index 4c04c67..664a8a0 100644 --- a/tests/test_offer_identity.py +++ b/tests/test_offer_identity.py @@ -22,3 +22,7 @@ def test_unknown_or_unproved_negative_is_not_matched(): def test_empty_target_does_not_invent_restrictions(): assert match_variant("", {"容量": "256GB"}, "") == "matched" + + +def test_model_and_capacity_can_be_in_one_target_phrase(): + assert match_variant("iPhone 15 256GB", {"型号": "iPhone 15", "容量": "256GB"}, "") == "matched" diff --git a/tests/test_offer_parser.py b/tests/test_offer_parser.py new file mode 100644 index 0000000..d4c1f2f --- /dev/null +++ b/tests/test_offer_parser.py @@ -0,0 +1,46 @@ +from goodprice.crawler.offers import parse_offer_candidates + + +def test_reads_actual_sku_combinations_not_price_range(): + html = '''
+
+
+
+
''' + offers = parse_offer_candidates(html) + assert [o.key for o in offers] == ["sku:part", "sku:128", "sku:256"] + assert offers[2].claims[0].amount_cents == 150000 + + +def test_only_range_is_not_exact_sku(): + offers = parse_offer_candidates('
850 - 1299
') + assert len(offers) == 1 + assert offers[0].claims[0].kind == "range_or_from" + assert offers[0].claims[0].amount_cents is None + + +def test_inquiry_and_unknown_do_not_invent_prices(): + inquiry = parse_offer_candidates('''

所有标价非售价,私聊咨询价格

+ 1
''')[0] + assert any(c.kind == "inquiry" for c in inquiry.claims) + assert parse_offer_candidates("推荐商品99元")[0].claims[0].kind == "unknown" + + +def test_collision_and_truncation_are_not_claimed_complete(): + offers = parse_offer_candidates('
' + + ''.join(f'
' for i in range(21)) + '
') + assert len(offers) == 20 + assert all(any(c.kind == "unknown" and "不完整" in c.text for c in o.claims) for o in offers) + duplicate = parse_offer_candidates('''
+
''') + assert len(duplicate) == 1 + assert duplicate[0].claims[-1].kind == "conflict" + + +def test_jsonld_product_used_and_global_disclaimer_not_ignored(): + html = '''
''' + offer = parse_offer_candidates(html)[0] + assert any(c.kind == "inquiry" for c in offer.claims) diff --git a/tests/test_offer_pipeline.py b/tests/test_offer_pipeline.py new file mode 100644 index 0000000..e7a5fc1 --- /dev/null +++ b/tests/test_offer_pipeline.py @@ -0,0 +1,137 @@ +from datetime import datetime, timezone + +from goodprice.crawler.offers import parse_offer_candidates +from goodprice.models import Listing, OfferEvaluationRecord +from goodprice.services.offer_pipeline import OfferPipeline +from goodprice.services.task_service import TaskService + + +def test_target_price_not_search_teaser_controls_budget(session_factory): + task = TaskService(session_factory).create_task({"keyword": "iphone", "target_variant": "256GB", + "min_price": 1000, "max_price": 1600}) + with session_factory() as session: + listing = Listing(platform="xianyu", external_id="e", title="iphone", price=99, + url="u", task_id=task.id) + session.add(listing) + session.commit() + listing_id = listing.id + candidates = parse_offer_candidates('''
+
+
''') + pipeline = OfferPipeline(session_factory) + ids = pipeline.evaluate_listing(listing_id, task.id, candidates, datetime.now(timezone.utc)) + with session_factory() as session: + rows = [session.get(OfferEvaluationRecord, i) for i in ids] + eligible = [row.payload for row in rows if row.payload["eligible"]] + assert len(eligible) == 1 + assert eligible[0]["amount_cents"] == 150000 + assert session.get(Listing, listing_id).price == 99 + TaskService(session_factory).update_task(task.id, {"max_price": 1400}) + ids = pipeline.evaluate_listing(listing_id, task.id, candidates, datetime.now(timezone.utc)) + with session_factory() as session: + assert not any(session.get(OfferEvaluationRecord, i).payload["eligible"] for i in ids) + + +def test_condition_threshold_and_unknown_prices_never_pass(session_factory): + task = TaskService(session_factory).create_task({"keyword": "x", "min_condition_score": 7}) + with session_factory() as session: + listing = Listing(platform="xianyu", external_id="e", title="x", price=1, + url="u", task_id=task.id) + session.add(listing) + session.commit() + listing_id = listing.id + candidates = parse_offer_candidates('
100
') + ids = OfferPipeline(session_factory).evaluate_listing(listing_id, task.id, candidates, + datetime.now(timezone.utc)) + with session_factory() as session: + payload = session.get(OfferEvaluationRecord, ids[0]).payload + assert payload["eligible"] is False + assert payload["reason"] == "品相未核验或低于门槛" + + +def test_live_crawl_does_not_filter_teaser_before_detail(session_factory, base_settings): + from test_crawl_service import FakeAdapter, _item, _service + + from goodprice.crawler.parser import parse_detail_html + + class Adapter(FakeAdapter): + def fetch_detail(self, url): + self.fetch_calls.append(url) + return parse_detail_html('''
+
''') + + task = TaskService(session_factory).create_task({"keyword": "iphone", "target_variant": "256GB", + "min_price": 1000, "max_price": 1600}) + adapter = Adapter([_item(price=99)]) + crawl, notifier, _ = _service(session_factory, base_settings, adapter=adapter) + stats = crawl.run_task(task.id) + assert len(adapter.fetch_calls) == 1 + assert stats["notified"] == 1 + assert "1500" in notifier.messages[0].content + assert "99 元" not in notifier.messages[0].content + assert crawl.vision.calls[0]["price"] == 1500 + + +def test_live_crawl_inquiry_does_not_call_value_model_or_send(session_factory, base_settings): + from test_crawl_service import FakeAdapter, FakeLLM, _item, _service + + from goodprice.crawler.parser import parse_detail_html + + class Adapter(FakeAdapter): + def fetch_detail(self, url): + return parse_detail_html('''
1 +

标价非售价,咨询价格

''') + + task = TaskService(session_factory).create_task({"keyword": "iphone"}) + llm = FakeLLM() + crawl, notifier, _ = _service(session_factory, base_settings, adapter=Adapter([_item(price=1)]), llm=llm) + assert crawl.run_task(task.id)["notified"] == 0 + assert not llm.value_calls + assert not notifier.messages + + +def test_pending_price_notifications_require_opt_in_and_are_deduplicated(session_factory, base_settings): + from test_crawl_service import FakeAdapter, _item, _service + + from goodprice.crawler.parser import parse_detail_html + + class Adapter(FakeAdapter): + def fetch_detail(self, url): + return parse_detail_html('

价格私聊

') + + task = TaskService(session_factory).create_task({"keyword": "iphone", "notify_pending_price": True}) + crawl, notifier, _ = _service(session_factory, base_settings, adapter=Adapter([_item(price=1)])) + crawl.run_task(task.id) + crawl.run_task(task.id) + assert len(notifier.messages) == 1 + assert "待询价" in notifier.messages[0].title + assert "本批最优" not in notifier.messages[0].content + + +def test_reanalysis_does_not_rank_unknown_price(session_factory, base_settings): + from test_crawl_service import FakeAdapter, FakeLLM, _item, _service + + task = TaskService(session_factory).create_task({"keyword": "x", "fetch_detail": False}) + llm = FakeLLM() + crawl, _, _ = _service(session_factory, base_settings, adapter=FakeAdapter([_item()]), llm=llm) + crawl.run_task(task.id) + with session_factory() as session: + listing_id = session.query(Listing).one().id + crawl.reanalyze_listing(listing_id) + assert not llm.value_calls + + +def test_disabled_vision_cannot_reuse_old_score_to_pass_threshold(session_factory, base_settings): + from test_crawl_service import FakeAdapter, FakeVision, _item, _service + + task = TaskService(session_factory).create_task({"keyword": "x", "min_condition_score": 7}) + with session_factory() as session: + listing = Listing(platform="xianyu", external_id="1001", title="t", price=100, + url="https://x/1001", task_id=task.id, condition_score=10) + session.add(listing) + session.commit() + crawl, notifier, _ = _service(session_factory, base_settings, adapter=FakeAdapter([_item()]), + vision=FakeVision(enabled=False)) + assert crawl.run_task(task.id)["notified"] == 0 + assert not notifier.messages diff --git a/tests/test_offer_pricing.py b/tests/test_offer_pricing.py index 522b4b7..00b95ac 100644 --- a/tests/test_offer_pricing.py +++ b/tests/test_offer_pricing.py @@ -81,3 +81,11 @@ def test_same_source_and_same_offer_required_for_drop(): current = decide_price(candidate(cents=130000), None, NOW, 1200) assert is_comparable_drop(previous, current, same_offer=True) assert not is_comparable_drop(previous, current, same_offer=False) + + +def test_incomplete_evidence_never_becomes_exact(): + offer = candidate() + incomplete = PriceClaim("dom", "unknown", "规格获取不完整", NOW, "truncated") + result = decide_price(replace(offer, claims=(*offer.claims, incomplete)), None, NOW, 1200) + assert result.state == "unknown" + assert result.amount_cents is None diff --git a/tests/test_offer_regressions.py b/tests/test_offer_regressions.py new file mode 100644 index 0000000..be6c436 --- /dev/null +++ b/tests/test_offer_regressions.py @@ -0,0 +1,158 @@ +from copy import deepcopy +from dataclasses import replace +from datetime import datetime, timedelta, timezone + +from fastapi.testclient import TestClient + +from goodprice.crawler.offers import parse_offer_candidates +from goodprice.main import build_app +from goodprice.models import Listing, ListingOffer, Notification +from goodprice.offers.pricing import decide_price, offer_fingerprint +from goodprice.offers.service import OfferService +from goodprice.offers.types import QuoteInput +from goodprice.services.offer_pipeline import OfferPipeline +from goodprice.services.task_service import TaskService + + +def test_manual_price_cannot_resolve_ambiguous_identity(): + offer = parse_offer_candidates('''
+
+
''')[0] + now = datetime.now(timezone.utc) + quote = QuoteInput(offer.key, 10000, now, now + timedelta(days=7), offer_fingerprint(offer)) + assert decide_price(offer, quote, now, 1200).state != "exact" + + +def test_description_and_media_are_analysis_dependencies(session_factory, base_settings): + from test_crawl_service import FakeAdapter, FakeLLM, _item, _service + + class Adapter(FakeAdapter): + description = "屏幕完好" + + def fetch_detail(self, url): + return replace(super().fetch_detail(url), description=self.description) + + task = TaskService(session_factory).create_task({"keyword": "x", "condition_requirement": "屏幕完好"}) + adapter, llm = Adapter([_item()]), FakeLLM() + crawl, _, _ = _service(session_factory, base_settings, adapter=adapter, llm=llm) + crawl.run_task(task.id) + adapter.description = "屏幕碎裂,无法显示" + llm.verdict = {"matched": False, "reason": "碎屏"} + crawl.run_task(task.id) + assert len(llm.calls) == 2 + with session_factory() as session: + assert session.query(Listing).one().requirement_match is False + + +def test_removed_images_clear_old_condition_score(session_factory, base_settings): + from test_crawl_service import FakeAdapter, _item, _service + + class Adapter(FakeAdapter): + empty_images = False + + def fetch_detail(self, url): + detail = super().fetch_detail(url) + return replace(detail, image_urls=[]) if self.empty_images else detail + + task = TaskService(session_factory).create_task({"keyword": "x", "min_condition_score": 7}) + adapter = Adapter([_item()]) + crawl, _, _ = _service(session_factory, base_settings, adapter=adapter) + crawl.run_task(task.id) + adapter.empty_images = True + crawl.run_task(task.id) + with session_factory() as session: + assert session.query(Listing).one().condition_score is None + + +def test_expired_snapshot_and_list_do_not_keep_recommendations(session_factory, base_settings): + from test_crawl_service import FakeAdapter, _item, _service + + from goodprice.models import OfferObservation + + task = TaskService(session_factory).create_task({"keyword": "x"}) + crawl, _, _ = _service(session_factory, base_settings, adapter=FakeAdapter([_item()])) + crawl.run_task(task.id) + with session_factory() as session: + listing_id = session.query(Listing).one().id + for observation in session.query(OfferObservation): + payload = deepcopy(observation.payload) + for claim in payload["claims"]: + claim["observed_at"] = (datetime.now(timezone.utc) - timedelta(hours=1)).isoformat() + observation.payload = payload + session.commit() + snapshot = OfferPipeline(session_factory).snapshot(listing_id, datetime.now(timezone.utc))[0] + assert snapshot["price_state"] == "unknown" + assert not snapshot["evaluation"]["eligible"] + client = TestClient(build_app(base_settings, session_factory, with_scheduler=False)) + assert "本批最优" not in client.get("/listings").text + data = client.get("/api/listings").json()[0] + assert data.get("value_score") is None + + +def test_failed_channel_not_hidden_by_successful_channel(session_factory, base_settings): + from test_crawl_service import FakeAdapter, FakeNotifier, _item, _service + + class Flaky(FakeNotifier): + attempts = 0 + + def send(self, message): + self.attempts += 1 + if self.attempts == 1: + raise RuntimeError("temporary") + super().send(message) + + task = TaskService(session_factory).create_task({"keyword": "x"}) + crawl, success, _ = _service(session_factory, base_settings, adapter=FakeAdapter([_item()])) + flaky = Flaky("flaky") + crawl.notifiers = [("log", success), ("flaky", flaky)] + crawl.run_task(task.id) + crawl.run_task(task.id) + assert flaky.attempts == 2 + assert len(success.messages) == 1 + + +def test_transient_fetch_failure_does_not_overwrite_fresh_quote_evidence(session_factory, base_settings): + from test_crawl_service import FakeAdapter, _item, _service + + class Adapter(FakeAdapter): + fail = False + + def fetch_detail(self, url): + if self.fail: + raise RuntimeError("timeout") + return super().fetch_detail(url) + + task = TaskService(session_factory).create_task({"keyword": "x"}) + adapter = Adapter([_item()]) + crawl, _, _ = _service(session_factory, base_settings, adapter=adapter) + crawl.run_task(task.id) + with session_factory() as session: + listing_id = session.query(Listing).one().id + snapshot = OfferPipeline(session_factory).snapshot(listing_id, datetime.now(timezone.utc))[0] + now = datetime.now(timezone.utc) + OfferService(session_factory).confirm_quote(snapshot["id"], QuoteInput( + snapshot["key"], 9000, now, now + timedelta(days=7), snapshot["fingerprint"])) + adapter.fail = True + crawl.run_task(task.id) + snapshot = OfferPipeline(session_factory).snapshot(listing_id, datetime.now(timezone.utc))[0] + assert snapshot["source"] == "manual" + assert snapshot["effective_price_cents"] == 9000 + + +def test_legacy_successful_notice_not_resent_after_migration(session_factory, base_settings): + from test_crawl_service import FakeAdapter, _item, _service + + task = TaskService(session_factory).create_task({"keyword": "x"}) + with session_factory() as session: + listing = Listing(platform="xianyu", external_id="1001", title="t", price=100, + url="https://x/1001", task_id=task.id, notified_at=datetime.now(), + last_notified_satisfaction=99) + session.add(listing) + session.flush() + session.add(Notification(listing_id=listing.id, task_id=task.id, channel="log", status="sent")) + session.commit() + crawl, notifier, _ = _service(session_factory, base_settings, adapter=FakeAdapter([_item()])) + crawl.run_task(task.id) + assert not notifier.messages + with session_factory() as session: + assert session.query(ListingOffer).count() == 1 From 7e871de645a6f17c760e5e72db068cd75a136f89 Mon Sep 17 00:00:00 2001 From: Comui520 Date: Mon, 26 Oct 2026 12:01:27 +0800 Subject: [PATCH 08/17] feat: configure media evidence and input budgets --- .env.example | 18 ++++++++ goodprice/config.py | 30 +++++++++++- goodprice/crawler/base.py | 2 + goodprice/crawler/parser.py | 2 + goodprice/media/__init__.py | 1 + goodprice/media/parser.py | 41 +++++++++++++++++ goodprice/media/selection.py | 35 ++++++++++++++ goodprice/media/types.py | 46 +++++++++++++++++++ goodprice/migrations.py | 3 +- goodprice/models.py | 18 ++++++++ goodprice/services/settings_service.py | 15 ++++-- tests/test_media_parser.py | 27 +++++++++++ tests/test_media_selection.py | 63 ++++++++++++++++++++++++++ 13 files changed, 295 insertions(+), 6 deletions(-) create mode 100644 goodprice/media/__init__.py create mode 100644 goodprice/media/parser.py create mode 100644 goodprice/media/selection.py create mode 100644 goodprice/media/types.py create mode 100644 tests/test_media_parser.py create mode 100644 tests/test_media_selection.py diff --git a/.env.example b/.env.example index 99788df..77449c9 100644 --- a/.env.example +++ b/.env.example @@ -72,3 +72,21 @@ ENABLE_NOVNC=0 NOVNC_PASSWORD_FILE=/app/data/.novnc-password # noVNC 对外端口(compose 端口映射与登录页链接共用;需与 BIND_ADDRESS 一起对外可达) NOVNC_PORT=16080 +# 媒体预算:提高数量会增加模型用量、延迟、内存和磁盘;不自动拆分付费请求。 +VIDEO_ANALYSIS_ENABLED=false +MEDIA_IMAGE_COUNT=8 +MEDIA_VIDEO_COUNT=1 +MEDIA_FRAMES_PER_VIDEO=6 +MEDIA_IMAGE_QUOTA=4 +MEDIA_FRAME_QUOTA=4 +MEDIA_TOTAL_LIMIT=8 +# 未知模型上限保守8;只有确认提供商支持后才提高,最高16。 +MEDIA_MODEL_LIMIT=8 +MEDIA_VIDEO_MAX_MIB=100 +MEDIA_VIDEO_MAX_SECONDS=180 +MEDIA_CACHE_MIB=1024 +MEDIA_RETENTION_DAYS=7 +MEDIA_DOWNLOAD_TIMEOUT=60 +MEDIA_DECODE_TIMEOUT=30 +MEDIA_REQUEST_MAX_MIB=16 +MEDIA_FFMPEG_THREADS=1 diff --git a/goodprice/config.py b/goodprice/config.py index cb195c1..1bb6ac4 100644 --- a/goodprice/config.py +++ b/goodprice/config.py @@ -1,7 +1,7 @@ from functools import lru_cache from pathlib import Path -from pydantic import field_validator +from pydantic import Field, field_validator, model_validator from pydantic_settings import BaseSettings, SettingsConfigDict PROJECT_ROOT = Path(__file__).resolve().parent.parent @@ -74,6 +74,34 @@ class Settings(BaseSettings): jev_auto_threshold: float = 0.85 jev_backend: str = "adapter" jev_api_key: str = "" + video_analysis_enabled: bool = False + media_image_count: int = Field(8, ge=1, le=32) + media_video_count: int = Field(1, ge=1, le=3) + media_frames_per_video: int = Field(6, ge=1, le=24) + media_image_quota: int = Field(4, ge=0, le=16) + media_frame_quota: int = Field(4, ge=0, le=16) + media_total_limit: int = Field(8, ge=1, le=16) + media_model_limit: int = Field(8, ge=1, le=16) + media_video_max_mib: int = Field(100, ge=10, le=200) + media_video_max_seconds: int = Field(180, ge=15, le=300) + media_cache_mib: int = Field(1024, ge=256, le=10240) + media_retention_days: int = Field(7, ge=1, le=30) + media_download_timeout: int = Field(60, ge=10, le=120) + media_decode_timeout: int = Field(30, ge=10, le=120) + media_request_max_mib: int = Field(16, ge=1, le=32) + media_ffmpeg_threads: int = Field(1, ge=1, le=4) + + @model_validator(mode="after") + def _media_budgets(self): + if self.media_total_limit > self.media_model_limit: + raise ValueError("视觉输入总上限不能超过已确认的模型上限(未知时为8)") + # Serial assets share the same reservation; retained outputs from earlier + # assets remain charged. JPEG outputs are capped at 2 MiB per image/frame. + required = self.media_video_max_mib + 2 * ( + self.media_image_count + self.media_video_count * self.media_frames_per_video) + if required > self.media_cache_mib: + raise ValueError("媒体缓存不足以容纳一次处理的下载和输出,请减少视频/采样预算") + return self @field_validator("jev_backend") @classmethod diff --git a/goodprice/crawler/base.py b/goodprice/crawler/base.py index e45803b..9c92cd2 100644 --- a/goodprice/crawler/base.py +++ b/goodprice/crawler/base.py @@ -2,6 +2,7 @@ from datetime import datetime from typing import Optional +from goodprice.media.types import MediaCandidate from goodprice.offers.types import OfferCandidate @@ -23,6 +24,7 @@ class ListingDetail: image_urls: list[str] = field(default_factory=list) variants: list[dict] = field(default_factory=list) offers: list[OfferCandidate] = field(default_factory=list) + media: list[MediaCandidate] = field(default_factory=list) seller_uid: Optional[str] = None seller_name: Optional[str] = None credit_label: Optional[str] = None diff --git a/goodprice/crawler/parser.py b/goodprice/crawler/parser.py index 71b8bd8..6649607 100644 --- a/goodprice/crawler/parser.py +++ b/goodprice/crawler/parser.py @@ -8,6 +8,7 @@ from goodprice.crawler.base import ListingData, ListingDetail, SellerData from goodprice.crawler.constants import GOOFISH_ORIGIN from goodprice.crawler.offers import parse_offer_candidates +from goodprice.media.parser import extract_media_candidates BASE_URL = GOOFISH_ORIGIN _PRICE_RE = re.compile(r"(\d+(?:\.\d+)?)") @@ -114,6 +115,7 @@ def parse_detail_html(html: str) -> ListingDetail: credit_el = soup.select_one(sel.DETAIL_CREDIT_LABEL) credit_label = credit_el.get_text(strip=True) if credit_el else None return ListingDetail( + media=extract_media_candidates(html), offers=parse_offer_candidates(html), description=desc[:2000], image_urls=images[:8], diff --git a/goodprice/media/__init__.py b/goodprice/media/__init__.py new file mode 100644 index 0000000..0fa0c87 --- /dev/null +++ b/goodprice/media/__init__.py @@ -0,0 +1 @@ +"""Bounded media evidence; importing this package performs no I/O.""" diff --git a/goodprice/media/parser.py b/goodprice/media/parser.py new file mode 100644 index 0000000..1701383 --- /dev/null +++ b/goodprice/media/parser.py @@ -0,0 +1,41 @@ +import hashlib +from urllib.parse import urljoin, urlsplit + +from bs4 import BeautifulSoup + +from goodprice.crawler.constants import GOOFISH_ORIGIN +from goodprice.media.types import MediaCandidate + + +def extract_media_candidates(html: str) -> list[MediaCandidate]: + soup = BeautifulSoup(html, "html.parser") + root = soup.select_one("[data-item-detail], main, [class*='detail-container']") + if root is None: + return [] + container = root.select_one("[data-item-media], [class*='item-slider'], [class*='main-media']") + nodes = (container.select("img, video") if container else + root.select("img.ant-image-img, [data-item-video]")) + result = [] + seen = set() + for node in nodes: + if node.find_parent(["aside", "nav"]) or node.find_parent(attrs={"data-recommend": True}): + continue + kind = "video" if node.name == "video" else "image" + source = node.select_one("source[src]") if kind == "video" else None + raw = node.get("src") or (source.get("src") if source else "") or node.get("data-src") + if not raw: + continue + url = urljoin(GOOFISH_ORIGIN, raw) + sku = node.get("data-sku-id") + scope = f"sku:{sku}" if sku else None + identity = (kind, url, scope) + if identity in seen: + continue + seen.add(identity) + parsed = urlsplit(url) + supported = parsed.scheme in {"http", "https"} and not parsed.path.lower().endswith((".m3u8", ".mpd")) and not node.get("data-drm") + result.append(MediaCandidate( + hashlib.sha256(repr(identity).encode()).hexdigest()[:32], kind, url, scope, + f"main-media:{len(result)}", node.get("poster"), + "pending" if supported else "unsupported")) + return result diff --git a/goodprice/media/selection.py b/goodprice/media/selection.py new file mode 100644 index 0000000..d812172 --- /dev/null +++ b/goodprice/media/selection.py @@ -0,0 +1,35 @@ +from goodprice.media.types import MediaEvidence, MediaSelection + + +def select_media(images: list[MediaEvidence], frames: list[MediaEvidence], + image_quota: int, frame_quota: int, total_limit: int) -> MediaSelection: + if not (0 <= image_quota <= 16 and 0 <= frame_quota <= 16 and 1 <= total_limit <= 16): + raise ValueError("Invalid media input budget") + groups = [images, frames] + quotas = [image_quota, frame_quota] + positions = [0, 0] + selected = [] + # First distribute the requested shares fairly, then lend unused shares. + for borrowing in (False, True): + while len(selected) < total_limit: + progressed = False + for i in (0, 1): + if (quotas[i] and positions[i] < len(groups[i]) + and (borrowing or positions[i] < quotas[i])): + selected.append(groups[i][positions[i]]) + positions[i] += 1 + progressed = True + if len(selected) == total_limit: + break + if not progressed: + break + ids = {x.id for x in selected} + omitted = tuple(x.id for x in images + frames if x.id not in ids) + limits = [] + if frames and not frame_quota: + limits.append("frames_disabled") + if images and not image_quota: + limits.append("images_disabled") + if omitted: + limits.append("partial_media_coverage") + return MediaSelection(tuple(selected), omitted, tuple(limits)) diff --git a/goodprice/media/types.py b/goodprice/media/types.py new file mode 100644 index 0000000..3411b5f --- /dev/null +++ b/goodprice/media/types.py @@ -0,0 +1,46 @@ +from dataclasses import dataclass +from pathlib import Path +from typing import Literal + + +@dataclass(frozen=True) +class MediaCandidate: + key: str + kind: Literal["image", "video"] + url: str + offer_key: str | None = None + source_locator: str = "" + poster_url: str | None = None + status: str = "pending" + + +@dataclass(frozen=True) +class MediaEvidence: + id: str + kind: Literal["image", "frame"] + content_hash: str + local_path: Path + offer_key: str | None = None + video_id: str | None = None + timestamp_ms: int | None = None + + +@dataclass(frozen=True) +class MediaSelection: + items: tuple[MediaEvidence, ...] + omitted_ids: tuple[str, ...] = () + limitations: tuple[str, ...] = () + + +@dataclass(frozen=True) +class VideoInfo: + duration_seconds: float + width: int + height: int + + +@dataclass(frozen=True) +class FrameFile: + path: Path + timestamp_ms: int + content_hash: str diff --git a/goodprice/migrations.py b/goodprice/migrations.py index 71f542b..ebaae69 100644 --- a/goodprice/migrations.py +++ b/goodprice/migrations.py @@ -4,6 +4,7 @@ from goodprice.models import ( ListingOffer, ManualQuote, + MediaAsset, OfferEvaluationRecord, OfferObservation, SchemaVersion, @@ -14,7 +15,7 @@ def migrate_offer_schema(session) -> None: connection = session.connection() tables = set(inspect(connection).get_table_names()) legacy = "schema_versions" not in tables and "watch_tasks" in tables - for model in (SchemaVersion, ListingOffer, OfferObservation, ManualQuote, OfferEvaluationRecord): + for model in (SchemaVersion, ListingOffer, OfferObservation, ManualQuote, OfferEvaluationRecord, MediaAsset): model.__table__.create(connection, checkfirst=True) if session.get(SchemaVersion, 1) is None: session.add(SchemaVersion(id=1, version=1, legacy_install=legacy)) diff --git a/goodprice/models.py b/goodprice/models.py index 49c8554..0d2e1f3 100644 --- a/goodprice/models.py +++ b/goodprice/models.py @@ -208,3 +208,21 @@ class Seller(Base): positive_rate: Mapped[Optional[float]] = mapped_column(Float, nullable=True) last_fetched_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True) blocked: Mapped[bool] = mapped_column(default=False) + + +class MediaAsset(Base): + __tablename__ = "media_assets" + id: Mapped[int] = mapped_column(primary_key=True) + listing_id: Mapped[int] = mapped_column(ForeignKey("listings.id"), index=True) + key: Mapped[str] = mapped_column(String(64)) + kind: Mapped[str] = mapped_column(String(10)) + offer_key: Mapped[Optional[str]] = mapped_column(String(200), nullable=True) + source_locator: Mapped[str] = mapped_column(Text, default="") + source_url: Mapped[str] = mapped_column(Text, default="") + parent_video_key: Mapped[Optional[str]] = mapped_column(String(64), nullable=True) + timestamp_ms: Mapped[Optional[int]] = mapped_column(Integer, nullable=True) + content_hash: Mapped[Optional[str]] = mapped_column(String(64), nullable=True) + relative_path: Mapped[Optional[str]] = mapped_column(Text, nullable=True) + status: Mapped[str] = mapped_column(String(30), default="pending") + failure: Mapped[Optional[str]] = mapped_column(Text, nullable=True) + observed_at: Mapped[datetime] = mapped_column(DateTime, default=_now) diff --git a/goodprice/services/settings_service.py b/goodprice/services/settings_service.py index 028f040..7527c6f 100644 --- a/goodprice/services/settings_service.py +++ b/goodprice/services/settings_service.py @@ -1,7 +1,7 @@ from typing import Optional from goodprice.config import Settings -from goodprice.models import AppSetting +from goodprice.models import AppSetting, SchemaVersion # 设置字段只在 goodprice.config.Settings 定义一次。运行时值与数据库覆盖都经同一个 # pydantic 模型校验与类型转换(int/float/bool 由字段类型自动完成),不再维护第二份 @@ -17,8 +17,10 @@ def __init__(self, session_factory, base: Optional[Settings] = None): def _overrides(self, session) -> dict[str, str]: return {row.key: row.value for row in session.query(AppSetting).all()} - def _merge(self, overrides: dict[str, str]) -> RuntimeSettings: + def _merge(self, overrides: dict[str, str], legacy: bool = False) -> RuntimeSettings: values = self._base.model_dump() + if legacy and "media_total_limit" not in self._base.model_fields_set: + values["media_total_limit"] = 4 for key, value in overrides.items(): if key in values and value != "": values[key] = value @@ -26,10 +28,15 @@ def _merge(self, overrides: dict[str, str]) -> RuntimeSettings: def get(self) -> RuntimeSettings: with self._session_factory() as session: - return self._merge(self._overrides(session)) + marker = session.get(SchemaVersion, 1) + return self._merge(self._overrides(session), bool(marker and marker.legacy_install)) def set_many(self, values: dict[str, str]) -> RuntimeSettings: with self._session_factory() as session: + overrides = self._overrides(session) + overrides.update(values) + marker = session.get(SchemaVersion, 1) + validated = self._merge(overrides, bool(marker and marker.legacy_install)) for key, value in values.items(): row = session.get(AppSetting, key) if value == "": @@ -40,4 +47,4 @@ def set_many(self, values: dict[str, str]) -> RuntimeSettings: else: session.add(AppSetting(key=key, value=value)) session.commit() - return self._merge(self._overrides(session)) + return validated diff --git a/tests/test_media_parser.py b/tests/test_media_parser.py new file mode 100644 index 0000000..a483552 --- /dev/null +++ b/tests/test_media_parser.py @@ -0,0 +1,27 @@ +from goodprice.crawler.parser import parse_detail_html +from goodprice.media.parser import extract_media_candidates + + +def test_only_main_media_with_scope_and_stable_order(): + html = '''
+ +
+
+
''' + items = extract_media_candidates(html) + assert [x.kind for x in items] == ["image", "video"] + assert items[0].offer_key is None + assert items[1].offer_key == "sku:s1" + assert parse_detail_html(html).media == items + + +def test_unsupported_video_kept_as_unavailable_not_image(): + items = extract_media_candidates('
') + assert len(items) == 2 + assert all(x.status == "unsupported" for x in items) + + +def test_known_detail_images_without_media_container_exclude_recommendations(): + items = extract_media_candidates('''
+
''') + assert len(items) == 1 diff --git a/tests/test_media_selection.py b/tests/test_media_selection.py new file mode 100644 index 0000000..c64e2b4 --- /dev/null +++ b/tests/test_media_selection.py @@ -0,0 +1,63 @@ +from pathlib import Path + +import pytest +from pydantic import ValidationError + +from goodprice.config import Settings +from goodprice.media.selection import select_media +from goodprice.media.types import MediaEvidence + + +def evidence(kind, count): + return [MediaEvidence(f"{kind}{i}", kind, f"hash{i}", Path(f"{i}.jpg")) for i in range(count)] + + +def test_default_mixed_selection_is_four_images_four_frames(): + result = select_media(evidence("image", 8), evidence("frame", 6), 4, 4, 8) + assert [item.kind for item in result.items] == ["image", "frame"] * 4 + assert len(result.omitted_ids) == 6 + + +def test_single_type_borrows_unused_quota(): + result = select_media(evidence("image", 10), [], 4, 4, 8) + assert len(result.items) == 8 + assert len(select_media(evidence("image", 1), evidence("frame", 10), 4, 4, 8).items) == 8 + + +def test_zero_quota_reports_omitted_type(): + result = select_media(evidence("image", 8), evidence("frame", 4), 4, 0, 8) + assert all(item.kind == "image" for item in result.items) + assert "frames_disabled" in result.limitations + assert set(result.omitted_ids) >= {"frame0", "frame1", "frame2", "frame3"} + + +@pytest.mark.parametrize("values", [dict(media_total_limit=9), dict(media_total_limit=17), + dict(media_video_max_mib=201), dict(media_image_quota=-1), + dict(media_cache_mib=256, media_video_max_mib=200, media_frames_per_video=24, media_video_count=3)]) +def test_invalid_or_over_model_limit_rejected(values): + with pytest.raises((ValueError, ValidationError)): + Settings(_env_file=None, **values) + + +def test_explicit_model_limit_allows_sixteen(): + assert Settings(_env_file=None, media_model_limit=16, media_total_limit=16).media_total_limit == 16 + + +def test_old_install_keeps_four_and_explicit_value_survives(session_factory, base_settings): + from goodprice.models import SchemaVersion + from goodprice.services.settings_service import SettingsService + with session_factory() as session: + session.add(SchemaVersion(id=1, version=1, legacy_install=True)) + session.commit() + service = SettingsService(session_factory, base_settings) + assert service.get().media_total_limit == 4 + assert service.get().video_analysis_enabled is False + assert service.set_many({"media_total_limit": "6"}).media_total_limit == 6 + + +def test_invalid_save_is_atomic(session_factory, base_settings): + from goodprice.services.settings_service import SettingsService + service = SettingsService(session_factory, base_settings) + with pytest.raises(ValidationError): + service.set_many({"media_total_limit": "17", "llm_model": "uncommitted"}) + assert service.get().llm_model != "uncommitted" From 490cdca5ec65542b438f5c45c0f77e011dc82f6f Mon Sep 17 00:00:00 2001 From: Comui520 Date: Mon, 26 Oct 2026 12:01:27 +0800 Subject: [PATCH 09/17] feat: bound media downloads and cache disk usage --- goodprice/media/budget.py | 139 +++++++++++++++++++++++++++++++++++ goodprice/media/download.py | 126 +++++++++++++++++++++++++++++++ tests/test_media_budget.py | 49 ++++++++++++ tests/test_media_download.py | 67 +++++++++++++++++ 4 files changed, 381 insertions(+) create mode 100644 goodprice/media/budget.py create mode 100644 goodprice/media/download.py create mode 100644 tests/test_media_budget.py create mode 100644 tests/test_media_download.py diff --git a/goodprice/media/budget.py b/goodprice/media/budget.py new file mode 100644 index 0000000..f080104 --- /dev/null +++ b/goodprice/media/budget.py @@ -0,0 +1,139 @@ +"""Single-process disk accounting, shared by all media workers for a root. + +Pending directories are reservations, including every intermediate output. +Only one application process may own a data directory (also true of scheduler). +""" +import shutil +import threading +import time +import uuid +from contextlib import contextmanager +from dataclasses import dataclass, field +from pathlib import Path + + +class BudgetExceeded(RuntimeError): + pass + + +@dataclass +class _State: + lock: threading.RLock = field(default_factory=threading.RLock) + reservations: dict = field(default_factory=dict) + pins: dict = field(default_factory=dict) + + +_states = {} +_states_lock = threading.Lock() + + +@dataclass +class BudgetReservation: + token: str + bytes_reserved: int + directory: Path + budget: "MediaBudget" + + def __enter__(self): + return self + + def __exit__(self, *args): + self.budget.release(self.token) + + +class MediaBudget: + def __init__(self, root: Path, max_bytes: int, retention_seconds: int, + safety_bytes: int = 64 * 1024**2): + self.root = root.resolve() + self.root.mkdir(parents=True, exist_ok=True) + self.max_bytes = max_bytes + self.retention_seconds = retention_seconds + self.safety_bytes = safety_bytes + with _states_lock: + if self.root not in _states: + # Crash leftovers are never served. No external path is followed. + for path in self.root.glob("pending-*"): + if path.is_dir() and not path.is_symlink(): + shutil.rmtree(path) + _states[self.root] = _State() + self.state = _states[self.root] + + def checked(self, path: Path) -> Path: + absolute = path.resolve() + if not absolute.is_relative_to(self.root) or path.is_symlink(): + raise ValueError("Media path outside controlled cache") + return absolute + + def _files(self): + return [p for p in self.root.rglob("*") if p.is_file() and not p.is_symlink() + and p.resolve().is_relative_to(self.root) + and not any(part.startswith("pending-") for part in p.relative_to(self.root).parts)] + + @property + def usage_bytes(self): + with self.state.lock: + return sum(p.stat().st_size for p in self._files()) + sum( + reservation.bytes_reserved for reservation in self.state.reservations.values()) + + def reserve(self, key: str, bytes_needed: int) -> BudgetReservation: + if bytes_needed < 1 or bytes_needed > self.max_bytes: + raise BudgetExceeded("单项媒体预算超过缓存容量,请降低文件或采样上限") + with self.state.lock: + files = sorted(self._files(), key=lambda p: p.stat().st_mtime) + for path in files: + if self.state.pins.get(path.resolve(), 0): + continue + if (time.time() - path.stat().st_mtime >= self.retention_seconds + or self.usage_bytes + bytes_needed > self.max_bytes): + path.unlink(missing_ok=True) + if (self.usage_bytes + bytes_needed > self.max_bytes or + shutil.disk_usage(self.root).free < bytes_needed + self.safety_bytes): + raise BudgetExceeded("媒体磁盘预算暂不可用,请稍后重试或降低预算") + token = uuid.uuid4().hex + directory = self.root / f"pending-{token}" + directory.mkdir() + reservation = BudgetReservation(token, bytes_needed, directory, self) + self.state.reservations[token] = reservation + return reservation + + def commit(self, token: str, paths: list[Path]) -> None: + with self.state.lock: + reservation = self.state.reservations[token] + paths = [self.checked(p) for p in paths] + if any(not p.is_relative_to(reservation.directory) for p in paths): + raise ValueError("Output does not belong to this reservation") + # Check ALL files, not only outputs supplied by the caller. + size = sum(p.stat().st_size for p in reservation.directory.rglob("*") if p.is_file()) + if size > reservation.bytes_reserved: + raise BudgetExceeded("媒体输出超过预留空间") + # Keep output paths stable; remove the pending prefix via move then + # return paths through committed_paths. No file can escape its root. + destination = self.root / f"asset-{token}" + relative = [p.relative_to(reservation.directory) for p in paths] + kept = set(paths) + for p in reservation.directory.rglob("*"): + if p.is_file() and p.resolve() not in kept: + p.unlink() + reservation.directory.rename(destination) + reservation.committed_paths = [destination / p for p in relative] + del self.state.reservations[token] + + def release(self, token: str) -> None: + with self.state.lock: + reservation = self.state.reservations.pop(token, None) + if reservation and reservation.directory.exists(): + self.checked(reservation.directory) + shutil.rmtree(reservation.directory) + + @contextmanager + def pin(self, paths): + with self.state.lock: + checked = [self.checked(p) for p in paths] + for p in checked: + self.state.pins[p] = self.state.pins.get(p, 0) + 1 + try: + yield + finally: + with self.state.lock: + for p in checked: + self.state.pins[p] -= 1 diff --git a/goodprice/media/download.py b/goodprice/media/download.py new file mode 100644 index 0000000..4dc3c92 --- /dev/null +++ b/goodprice/media/download.py @@ -0,0 +1,126 @@ +"""Public HTTP media only; no environment proxy, cookies or second DNS lookup.""" +import ipaddress +import socket +import time +import uuid +from urllib.parse import urljoin, urlsplit + +import httpx +from httpcore import ConnectError +from httpcore._backends.sync import SyncBackend, SyncStream + +from goodprice.media.budget import MediaBudget +from goodprice.media.types import MediaCandidate + + +class MediaDownloadError(RuntimeError): + pass + + +def resolve_public(host, port): + return list({row[4][0] for row in socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)}) + + +def public_addresses(url, resolver): + parsed = urlsplit(url) + if (parsed.scheme not in {"http", "https"} or not parsed.hostname or parsed.username + or parsed.password or parsed.port not in {None, 80, 443}): + raise MediaDownloadError("媒体地址不受支持") + host = parsed.hostname + try: + literal = ipaddress.ip_address(host) + addresses = [str(literal)] + except ValueError: + addresses = resolver(host, parsed.port or (443 if parsed.scheme == "https" else 80)) + if not addresses or any(not ipaddress.ip_address(a).is_global for a in addresses): + raise MediaDownloadError("媒体地址必须指向公网") + # Reject IPv4-mapped/private transition targets as well. + for addr in addresses: + ip = ipaddress.ip_address(addr) + if isinstance(ip, ipaddress.IPv6Address) and (ip.ipv4_mapped or ip.sixtofour or ip.teredo): + raise MediaDownloadError("媒体地址类型不受支持") + return host, parsed.port or (443 if parsed.scheme == "https" else 80), addresses[0] + + +class PinnedBackend(SyncBackend): + def __init__(self, pins): + self.pins = pins + + def connect_tcp(self, host, port, timeout=None, local_address=None, socket_options=None): + addr = self.pins.get((host, port)) + if addr is None: + raise MediaDownloadError("未经验证的媒体目标") + ip = ipaddress.ip_address(addr) + sock = socket.socket(socket.AF_INET6 if ip.version == 6 else socket.AF_INET, socket.SOCK_STREAM) + try: + sock.settimeout(timeout) + for option in socket_options or []: + sock.setsockopt(*option) + # socket.connect receives a numeric address; hostname cannot rebind. + sock.connect((str(ip), port)) + return SyncStream(sock) + except OSError as exc: + sock.close() + raise ConnectError("媒体连接失败") from exc + + +class MediaDownloader: + def __init__(self, budget: MediaBudget, transport=None, resolver=None): + self.budget = budget + self.transport = transport + self.resolver = resolver or resolve_public + + def download(self, candidate: MediaCandidate, max_bytes: int, timeout_seconds: int, + reservation=None): + own = reservation is None + reservation = reservation or self.budget.reserve(candidate.key, max_bytes) + path = reservation.directory / f"{uuid.uuid4().hex}.part" + deadline = time.monotonic() + timeout_seconds + try: + url = candidate.url + for _ in range(6): + host, port, addr = public_addresses(url, self.resolver) + remaining = deadline - time.monotonic() + if remaining <= 0: + raise MediaDownloadError("媒体下载超时") + transport = self.transport + if transport is None: + transport = httpx.HTTPTransport(trust_env=False, retries=0) + # httpcore 1.x backend preserves original Host and TLS SNI; + # only the socket destination is replaced with verified IP. + transport._pool._network_backend = PinnedBackend({(host, port): addr}) + with httpx.Client(transport=transport, trust_env=False, timeout=remaining, + follow_redirects=False) as client: + with client.stream("GET", url, headers={"Accept-Encoding": "identity"}) as response: + if response.is_redirect: + url = urljoin(url, response.headers.get("location", "")) + continue + response.raise_for_status() + if response.headers.get("content-encoding", "identity").lower() != "identity": + raise MediaDownloadError("媒体响应压缩格式不受支持") + length = response.headers.get("content-length") + if length and (not length.isdigit() or int(length) > max_bytes): + raise MediaDownloadError("媒体文件超过大小上限") + size = 0 + with path.open("xb") as output: + for chunk in response.iter_bytes(64 * 1024): + size += len(chunk) + if size > max_bytes or time.monotonic() > deadline: + raise MediaDownloadError("媒体下载超过字节或时间预算") + output.write(chunk) + if not size: + raise MediaDownloadError("媒体文件为空") + if own: + self.budget.commit(reservation.token, [path]) + return reservation.committed_paths[0] + return path + raise MediaDownloadError("媒体重定向次数超过限制") + except Exception as exc: + path.unlink(missing_ok=True) + if isinstance(exc, MediaDownloadError): + raise + # Never propagate signed URL, cookies, DNS details or response body. + raise MediaDownloadError("媒体下载失败") from None + finally: + if own: + self.budget.release(reservation.token) diff --git a/tests/test_media_budget.py b/tests/test_media_budget.py new file mode 100644 index 0000000..ad86c39 --- /dev/null +++ b/tests/test_media_budget.py @@ -0,0 +1,49 @@ +import pytest + +from goodprice.media.budget import BudgetExceeded, MediaBudget + + +def test_concurrent_reservations_include_temporary_outputs(tmp_path): + budget = MediaBudget(tmp_path, 256 * 1024**2, 86400, safety_bytes=0) + with budget.reserve("first", 200 * 1024**2): + other = MediaBudget(tmp_path, 256 * 1024**2, 86400, safety_bytes=0) + with pytest.raises(BudgetExceeded): + other.reserve("second", 200 * 1024**2) + with budget.reserve("second", 200 * 1024**2): + pass + assert budget.usage_bytes == 0 + + +def test_active_inputs_survive_cleanup_and_output_overflow_fails(tmp_path): + budget = MediaBudget(tmp_path, 32, 0, safety_bytes=0) + with budget.reserve("one", 16) as token: + path = token.directory / "image.jpg" + path.write_bytes(b"12345678") + budget.commit(token.token, [path]) + path = token.committed_paths[0] + with budget.pin([path]): + with pytest.raises(BudgetExceeded): + budget.reserve("second", 32) + assert path.exists() + with budget.reserve("oversized", 8) as token: + path = token.directory / "large.jpg" + path.write_bytes(b"x" * 9) + with pytest.raises(BudgetExceeded): + budget.commit(token.token, [path]) + assert not path.exists() + + +def test_reject_paths_outside_root(tmp_path): + budget = MediaBudget(tmp_path / "cache", 100, 1, safety_bytes=0) + outside = tmp_path / "secret" + outside.write_text("secret") + with budget.reserve("one", 30) as token: + with pytest.raises(ValueError): + budget.commit(token.token, [outside]) + assert outside.read_text() == "secret" + + +def test_impossible_reservation_does_not_wait(tmp_path): + budget = MediaBudget(tmp_path, 32, 1, safety_bytes=0) + with pytest.raises(BudgetExceeded): + budget.reserve("one", 33) diff --git a/tests/test_media_download.py b/tests/test_media_download.py new file mode 100644 index 0000000..94bdb16 --- /dev/null +++ b/tests/test_media_download.py @@ -0,0 +1,67 @@ +import httpx +import pytest + +from goodprice.media.budget import MediaBudget +from goodprice.media.download import MediaDownloader, MediaDownloadError, PinnedBackend +from goodprice.media.types import MediaCandidate + + +def downloader(tmp_path, handler, resolver=lambda host, port: ["93.184.216.34"]): + return MediaDownloader(MediaBudget(tmp_path, 1024, 86400, safety_bytes=0), + transport=httpx.MockTransport(handler), resolver=resolver) + + +@pytest.mark.parametrize("url", ["http://127.0.0.1/file", "http://[::1]/file", "file:///secret", "http://user:pass@example.com/a"]) +def test_bad_targets_rejected_before_request(tmp_path, url): + def forbidden(request): + raise AssertionError("unsafe request sent") + service = downloader(tmp_path, forbidden) + with pytest.raises(MediaDownloadError): + service.download(MediaCandidate("a", "image", url), 20, 10) + + +def test_redirect_to_metadata_rejected_and_no_credentials(tmp_path): + calls = [] + def handler(request): + calls.append(request) + assert "cookie" not in request.headers and "authorization" not in request.headers + return httpx.Response(302, headers={"location": "http://169.254.169.254/latest"}) + with pytest.raises(MediaDownloadError): + downloader(tmp_path, handler).download(MediaCandidate("a", "video", "https://cdn.example/a?token=secret"), 50, 10) + assert len(calls) == 1 + assert not list(tmp_path.rglob("*.part")) + + +@pytest.mark.parametrize("length", [None, "1", "200"]) +def test_stream_size_enforced_even_without_or_false_length(tmp_path, length): + headers = {"content-length": length} if length else {} + service = downloader(tmp_path, lambda r: httpx.Response(200, content=b"x" * 30, headers=headers)) + with pytest.raises(MediaDownloadError) as error: + service.download(MediaCandidate("a", "image", "https://cdn.example/a?token=secret"), 20, 10) + assert "secret" not in str(error.value) + assert not list(tmp_path.rglob("*.part")) + assert service.budget.usage_bytes == 0 + + +def test_valid_download_and_private_dns_rejected(tmp_path): + service = downloader(tmp_path, lambda r: httpx.Response(200, content=b"image")) + path = service.download(MediaCandidate("a", "image", "https://cdn.example/a"), 20, 10) + assert path.read_bytes() == b"image" + unsafe = downloader(tmp_path, lambda r: httpx.Response(200), lambda h, p: ["93.184.216.34", "::ffff:127.0.0.1"]) + with pytest.raises(MediaDownloadError): + unsafe.download(MediaCandidate("b", "image", "https://cdn.example/a"), 20, 10) + + +def test_connection_uses_pinned_numeric_ip_without_second_dns(monkeypatch): + calls = [] + class Socket: + def settimeout(self, value): pass + def setsockopt(self, *args): pass + def connect(self, address): calls.append(address) + def close(self): pass + monkeypatch.setattr("goodprice.media.download.socket.socket", lambda *args: Socket()) + backend = PinnedBackend({("cdn.example", 443): "93.184.216.34"}) + backend.connect_tcp("cdn.example", 443, timeout=1) + assert calls == [("93.184.216.34", 443)] + with pytest.raises(MediaDownloadError): + backend.connect_tcp("other.example", 443) From 2c5d84cd7868cf2f61936e8bd57e95f1d9c09a2f Mon Sep 17 00:00:00 2001 From: Comui520 Date: Mon, 26 Oct 2026 12:05:51 +0800 Subject: [PATCH 10/17] feat: extract bounded video frame evidence --- Dockerfile | 2 +- goodprice/media/cache.py | 19 ++++++ goodprice/media/frames.py | 102 ++++++++++++++++++++++++++++++++ goodprice/media/processing.py | 103 +++++++++++++++++++++++++++++++++ pyproject.toml | 2 + requirements.lock | 89 +++++++++++++++++++++++++++- tests/conftest.py | 1 + tests/test_media_processing.py | 62 ++++++++++++++++++++ tests/test_video_frames.py | 54 +++++++++++++++++ uv.lock | 94 ++++++++++++++++++++++++++++++ 10 files changed, 526 insertions(+), 2 deletions(-) create mode 100644 goodprice/media/cache.py create mode 100644 goodprice/media/frames.py create mode 100644 goodprice/media/processing.py create mode 100644 tests/test_media_processing.py create mode 100644 tests/test_video_frames.py diff --git a/Dockerfile b/Dockerfile index 11eae84..f163523 100644 --- a/Dockerfile +++ b/Dockerfile @@ -26,7 +26,7 @@ RUN env HTTP_PROXY="${HTTP_PROXY}" HTTPS_PROXY="${HTTPS_PROXY}" \ sh -c 'pip install --no-cache-dir --retries 10 --timeout 120 -r requirements.lock && playwright install --with-deps chromium' RUN apt-get update \ - && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends x11vnc novnc websockify \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends x11vnc novnc websockify ffmpeg \ && rm -rf /var/lib/apt/lists/* COPY pyproject.toml README.md ./ diff --git a/goodprice/media/cache.py b/goodprice/media/cache.py new file mode 100644 index 0000000..bc00b00 --- /dev/null +++ b/goodprice/media/cache.py @@ -0,0 +1,19 @@ +import hashlib +import json + + +def processing_key(content_hash: str, parameters: dict) -> str: + return hashlib.sha256(json.dumps([content_hash, parameters], sort_keys=True).encode()).hexdigest() + + +def find_cached(budget, key): + """Caller holds budget lock until cached inputs have been pinned.""" + for manifest in budget.root.glob(f"asset-*/{key}.json"): + try: + data = json.loads(manifest.read_text(encoding="utf-8")) + paths = [budget.checked(manifest.parent / item["file"]) for item in data] + if all(path.is_file() for path in paths): + return data, paths + except (OSError, ValueError, KeyError, TypeError): + continue + return None diff --git a/goodprice/media/frames.py b/goodprice/media/frames.py new file mode 100644 index 0000000..0d909d9 --- /dev/null +++ b/goodprice/media/frames.py @@ -0,0 +1,102 @@ +import hashlib +import json +import math +import shutil +import subprocess +import time +from pathlib import Path + +from goodprice.media.types import FrameFile, VideoInfo + +FORMATS = "mov,mp4,m4a,3gp,3g2,mj2,matroska,webm,avi" +MAX_PIXELS = 24_000_000 +MAX_OUTPUT_BYTES = 2 * 1024**2 + + +class MediaDecodeError(ValueError): + pass + + +def _local(path): + if not isinstance(path, Path) or not path.is_file() or path.is_symlink(): + raise MediaDecodeError("仅处理已下载的本地媒体文件") + return str(path.resolve()) + + +def _run(tool, args, timeout): + binary = shutil.which(tool) + if not binary: + raise MediaDecodeError("FFmpeg/ffprobe 未安装,视频未分析") + try: + result = subprocess.run([binary, *args], stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, timeout=max(.01, timeout), + check=True, text=True, encoding="utf-8") + except subprocess.TimeoutExpired: + raise MediaDecodeError("媒体解码超时") from None + except (subprocess.CalledProcessError, OSError): + raise MediaDecodeError("媒体文件解码失败") from None + return result.stdout + + +def probe_video(path: Path, timeout_seconds: int) -> VideoInfo: + path = _local(path) + raw = _run("ffprobe", ["-v", "error", "-protocol_whitelist", "file", "-format_whitelist", FORMATS, + "-select_streams", "v:0", "-show_entries", "stream=width,height:format=duration", + "-of", "json", path], timeout_seconds) + try: + data = json.loads(raw) + stream = data["streams"][0] + info = VideoInfo(float(data["format"]["duration"]), int(stream["width"]), int(stream["height"])) + if (not math.isfinite(info.duration_seconds) or info.duration_seconds <= 0 + or min(info.width, info.height) <= 0 or info.width * info.height > MAX_PIXELS): + raise ValueError() + return info + except (ValueError, TypeError, KeyError, IndexError): + raise MediaDecodeError("媒体时长或分辨率无效/超限") from None + + +def extract_frames(path: Path, count: int, output_dir: Path, timeout_seconds: int, + threads: int, max_seconds: int = 180) -> list[FrameFile]: + if not 1 <= count <= 24 or not 1 <= threads <= 4: + raise MediaDecodeError("抽帧数量或线程配置无效") + started = time.monotonic() + info = probe_video(path, timeout_seconds) + if info.duration_seconds > max_seconds: + raise MediaDecodeError("视频时长超过上限,未分析") + output_dir.mkdir(parents=True, exist_ok=True) + result, seen = [], set() + # Last point precedes the end by 250ms to avoid an empty seek at EOF. + end = max(0, info.duration_seconds - .25) + timestamps = [round(1000 * end * i / max(1, count - 1)) for i in range(count)] + try: + for i, timestamp in enumerate(timestamps): + output = output_dir / f"frame-{i}.jpg" + remaining = timeout_seconds - (time.monotonic() - started) + if remaining <= 0: + raise MediaDecodeError("媒体解码超时") + _run("ffmpeg", ["-v", "error", "-nostdin", "-y", "-protocol_whitelist", "file", + "-format_whitelist", FORMATS, "-threads", str(threads), + "-ss", str(timestamp / 1000), "-i", _local(path), + "-map", "0:v:0", "-an", "-sn", "-dn", "-frames:v", "1", + "-vf", "scale=1280:1280:force_original_aspect_ratio=decrease", + "-filter_threads", str(threads), "-threads", str(threads), + "-fs", str(MAX_OUTPUT_BYTES), str(output)], remaining) + if not output.is_file() or output.stat().st_size > MAX_OUTPUT_BYTES: + raise MediaDecodeError("视频帧输出无效或超过预算") + from PIL import Image, ImageFilter, ImageStat + with Image.open(output) as image: + image.verify() + with Image.open(output) as image: + # Flat or very blurred frames add no reliable visual evidence. + variance = ImageStat.Stat(image.convert("L").filter(ImageFilter.FIND_EDGES)).var[0] + digest = hashlib.sha256(output.read_bytes()).hexdigest() + if digest in seen or variance < 8: + output.unlink() + continue + seen.add(digest) + result.append(FrameFile(output, timestamp, digest)) + return result + except Exception: + for output in output_dir.glob("frame-*.jpg"): + output.unlink(missing_ok=True) + raise diff --git a/goodprice/media/processing.py b/goodprice/media/processing.py new file mode 100644 index 0000000..e766d90 --- /dev/null +++ b/goodprice/media/processing.py @@ -0,0 +1,103 @@ +import hashlib +import json +import warnings +from contextlib import ExitStack, contextmanager +from pathlib import Path + +from PIL import Image, UnidentifiedImageError + +from goodprice.media.cache import find_cached, processing_key +from goodprice.media.download import MediaDownloader, MediaDownloadError +from goodprice.media.frames import MAX_OUTPUT_BYTES, MAX_PIXELS, extract_frames +from goodprice.media.types import MediaEvidence + + +def prepare_image(path: Path, max_edge: int = 1280) -> Path: + output = path.with_suffix(".jpg") + try: + with warnings.catch_warnings(): + warnings.simplefilter("error", Image.DecompressionBombWarning) + with Image.open(path) as image: + if image.format not in {"JPEG", "PNG", "WEBP"} or image.width * image.height > MAX_PIXELS: + raise ValueError("图片格式或解码像素超过上限") + image.load() + image.thumbnail((max_edge, max_edge)) + image.convert("RGB").save(output, "JPEG", quality=85) + if output.stat().st_size > MAX_OUTPUT_BYTES: + raise ValueError("图片输出超过预算") + return output + except (OSError, UnidentifiedImageError, Image.DecompressionBombWarning, Image.DecompressionBombError): + output.unlink(missing_ok=True) + raise ValueError("图片解码失败或像素超限") from None + + +class MediaProcessor: + def __init__(self, budget, downloader=None): + self.budget = budget + self.downloader = downloader or MediaDownloader(budget) + + def process(self, candidates, settings): + with self.session(candidates, settings) as (evidence, limitations): + return evidence + + @contextmanager + def session(self, candidates, settings): + """Pins every selected file through the consumer's model request.""" + evidence, limitations = [], [] + counts = {"image": 0, "video": 0} + with ExitStack() as pins: + for candidate in candidates: + limit = settings.media_image_count if candidate.kind == "image" else settings.media_video_count + if counts[candidate.kind] >= limit: + limitations.append("retention_limit") + continue + counts[candidate.kind] += 1 + if candidate.status == "unsupported": + limitations.append("unsupported_media") + continue + if candidate.kind == "video" and not settings.video_analysis_enabled: + limitations.append("video_disabled") + continue + download_size = (settings.media_video_max_mib if candidate.kind == "video" else 10) * 1024**2 + output_size = MAX_OUTPUT_BYTES * (settings.media_frames_per_video if candidate.kind == "video" else 1) + try: + with self.budget.reserve(candidate.key, download_size + output_size + 65536) as reservation: + raw = self.downloader.download(candidate, download_size, settings.media_download_timeout, + reservation=reservation) + source_hash = hashlib.sha256(raw.read_bytes()).hexdigest() + parameters = {"version": 1, "kind": candidate.kind, "edge": 1280, + "frames": settings.media_frames_per_video, + "max_seconds": settings.media_video_max_seconds} + key = processing_key(source_hash, parameters) + with self.budget.state.lock: + cached = find_cached(self.budget, key) + if cached: + data, paths = cached + pins.enter_context(self.budget.pin(paths)) + if not cached: + if candidate.kind == "image": + outputs = [prepare_image(raw)] + data = [{"file": outputs[0].name, "timestamp": None, + "hash": hashlib.sha256(outputs[0].read_bytes()).hexdigest()}] + else: + frames = extract_frames(raw, settings.media_frames_per_video, reservation.directory / "frames", + settings.media_decode_timeout, settings.media_ffmpeg_threads, + max_seconds=settings.media_video_max_seconds) + outputs = [f.path for f in frames] + data = [{"file": str(f.path.relative_to(reservation.directory)), "timestamp": f.timestamp_ms, + "hash": f.content_hash} for f in frames] + raw.unlink(missing_ok=True) + manifest = reservation.directory / f"{key}.json" + manifest.write_text(json.dumps(data), encoding="utf-8") + with self.budget.state.lock: + self.budget.commit(reservation.token, outputs + [manifest]) + paths = reservation.committed_paths[:-1] + pins.enter_context(self.budget.pin(paths)) + for row, path in zip(data, paths, strict=True): + identity = hashlib.sha256(f"{row['hash']}:{candidate.offer_key}:{row['timestamp']}".encode()).hexdigest()[:32] + evidence.append(MediaEvidence(identity, "image" if candidate.kind == "image" else "frame", + row["hash"], path, candidate.offer_key, + candidate.key if candidate.kind == "video" else None, row["timestamp"])) + except (ValueError, RuntimeError, OSError, MediaDownloadError): + limitations.append(f"{candidate.kind}_unavailable") + yield evidence, tuple(dict.fromkeys(limitations)) diff --git a/pyproject.toml b/pyproject.toml index 759b080..184d58c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -20,6 +20,8 @@ dependencies = [ "playwright==1.62.0", "beautifulsoup4>=4.12", "httpx>=0.27", + "httpcore>=1,<2", + "pillow>=10.4", "jinja2>=3.1", "python-multipart>=0.0.9", "json-repair>=0.30", diff --git a/requirements.lock b/requirements.lock index 1f5779a..10280f3 100644 --- a/requirements.lock +++ b/requirements.lock @@ -122,7 +122,9 @@ h11==0.16.0 \ httpcore==1.0.9 \ --hash=sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55 \ --hash=sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8 - # via httpx + # via + # good-price + # httpx httpcore2==2.13.1 ; sys_platform != 'emscripten' \ --hash=sha256:e0aa977abe17e69a3b820a24542a6fa88702676d83880b8d194dcd18408e5103 \ --hash=sha256:e1e05d4f25f7d7d496bfb96748f6f4b67657b03da069b3a68c36069f3db73d0a @@ -261,6 +263,91 @@ markupsafe==3.0.3 \ --hash=sha256:f9e130248f4462aaa8e2552d547f36ddadbeaa573879158d721bbd33dfe4743a \ --hash=sha256:fed51ac40f757d41b7c48425901843666a6677e3e8eb0abcff09e4ba6e664f50 # via jinja2 +pillow==13.0.0 \ + --hash=sha256:02464da86f561f410e13b061ec37c3068a9cb557a095494157742d07b3c24a27 \ + --hash=sha256:04a63720f79e22fd54d897c233a68314c50ca0603360f54ce2bd5db8f0f43782 \ + --hash=sha256:05cd2cd39055d0b9650060e4513ef4a9a0aea870083950bd046a0d6490f93a22 \ + --hash=sha256:06903aac074d00c988e77a1fc435286dc498506d4ca8d6f69df9babddf4ecc15 \ + --hash=sha256:08b826a9bf1265c76de04a42061d61ea6b109418b7826f28ff1ecdfaa26c5502 \ + --hash=sha256:0d1a9cf7bc6917538b20f8d06b9ab2147e542f68849e96fc58eb66bdf6acc692 \ + --hash=sha256:10b5019883c98f1dd819ca48ffc6f33a91cee04948fe637c605f4345c7cafd6f \ + --hash=sha256:12b0fa15879fe5a9d9eebe66e4e2d74128a773b1a4448010c9c714f45e04b9d8 \ + --hash=sha256:12fbb851cf323cc19aac3eea1df5ad17b57e91e9fe3cfd06100dba7bb8208c13 \ + --hash=sha256:149a1c323ba366f90b48e6f1a6db6582eb50512df374588de56fc909abc0f0f3 \ + --hash=sha256:19b66ca0a4d25cc0200b381ce824c7ea090db61427315e73cfb05dbf1b35c369 \ + --hash=sha256:1c774b9c0ba0d0b356f71530677b39028878a2cae5a93e0a13779cfd17120f88 \ + --hash=sha256:20722cd5375533ad754053363a80b8fb30c79d37930a6f7228030e237ce3bd5d \ + --hash=sha256:20822140c9a2ba9c5e42df342b0a892ab2e102fc0e3db967a133bb9ce3f898bb \ + --hash=sha256:21c32061a5512dc13f87cc7303e120e4d77c2f70df89a35af0e3ce246e5ac57f \ + --hash=sha256:2310e2adfc13c6d457d7d0f2ffdd5c56a74a317650cf765887df8d515a647ac0 \ + --hash=sha256:23edf4ec4b6f4f240e8df6e4985e9ee2ced86474632433e55fb1e2548744bf40 \ + --hash=sha256:2d50500a4a3bc6469adcdb319cf05d688b08ca3bf0f22a05339f3b25b433a599 \ + --hash=sha256:2e31c4e68cffd7b1f116127626813b82fcba19edf6b940a7b0c266d6484d0055 \ + --hash=sha256:34fb3db1149c879343861831728c243c432d4a5afc8f1878ca7612ef8de2a033 \ + --hash=sha256:37290fbef0c73e2d51d594eb78fded7dc7170d1c3ad0a890269e6fd8c8c4e061 \ + --hash=sha256:405be6058e206a86fedce2f715ecc2343f94072d7925251632dffa2c05f70dca \ + --hash=sha256:40a266371ece581e5de13b6635ef78d7487824af80561b6ca27694a88c19bc0f \ + --hash=sha256:42d63be3af4f9b951b199b0a9d23365d37ab14687e6d77b74940432c8515c4ca \ + --hash=sha256:44629d4498c9cb6b2051806155f22141df0d8a929c56134b166655232dfb2f8c \ + --hash=sha256:49c12af515073e9bf511e4ef736743bf70d531a643668722279314b937742ba4 \ + --hash=sha256:4a25756244bdcce3b50580baadf320fa48297f0eeb8e114ba64139a40d4d0fc8 \ + --hash=sha256:4bd024d567eca931b668978d672dba914d4b8e2628e2c3073a2ad178de8a2620 \ + --hash=sha256:4f8db54177035a2b5f717f94475b74907e12a96dc74f6d487f3966b0a4694071 \ + --hash=sha256:51ebb529fc5e8abdd55302f6c3f8943037348ec2384329a3e55677f69ae2fdfc \ + --hash=sha256:567f20f1c75897d7663830da6ea8af719e03d712b1d9ee7f813319f9ece42e5a \ + --hash=sha256:5933fcea5a5ba64c686f55c886a115cbfd92f2900460784a1e79909fdbafd7b6 \ + --hash=sha256:5bdb38ba220b61b4ec31dd1046dea6b016d3c6850a23f43ed8b58332b2fc668e \ + --hash=sha256:5f56499c1a74d2b862beb0c33b8016f99d095858455df9d21583bbbc1c15b582 \ + --hash=sha256:67871a13cf47275201553b536c025da777f997fa562a19eb798af4ad4b97c18f \ + --hash=sha256:67c8d44ad89fa7eb88de5d6ef6199b1e03dd24974f85a67639fc8badb801fe79 \ + --hash=sha256:6a0920cb55df5a4d3f6862964908078c0bd59332bdd1bcd20d14e8c70ef27cc5 \ + --hash=sha256:6bce6a29c3898191f4c9998e4d4e8f69790caeabde46e54da4bc235ce5bde683 \ + --hash=sha256:6f38f3f885a09cafc93fd667b1173a92f379e43785d635e7983c893f0da319fb \ + --hash=sha256:73ee25cf8cf8b75d5a5aeed7cb6588f5e5dbfbb0fde51707b630bbb23897645c \ + --hash=sha256:7b999816b5850a35ada08443e6e03945f1f6488144a83afe2d37b240a552aafa \ + --hash=sha256:7c88c10dd432f3b39ccdafc9ed81d0c21b8fd634502574da76d1bfa4fe2c76d0 \ + --hash=sha256:7d5e097eabd5d4915c27ac9f6e9325bb1e01a7c94d7348b5c4451d3fc30e8521 \ + --hash=sha256:7e91e97a929232cdb828889ee6ad0e48d7f40f4ec38818f4e4a8968b2b1af22e \ + --hash=sha256:7f6299e494e46253d40c34653cac369412f44ca53b5f85d848d3f95b1ed0546c \ + --hash=sha256:816752a5ef76a80484c8c441c85593b7ea9304f6231fc5776ad1fbff7e773058 \ + --hash=sha256:83b37abd87b590acd8f47ca8a61f70893322b11834d36e07c2298dacad4d14d2 \ + --hash=sha256:8619685cad8efc4c38c66d3b36aad854523297c43d748b3b6f0cc03c90ba3463 \ + --hash=sha256:8bf295fe459343bc6093c8e5e0df60bfa189e0fcb775a22e2c2488fdeccc6c0d \ + --hash=sha256:8e459b613cf31f8980f584387b4fe47bf08e076e288d61aa8eba1f7fd4ddbb81 \ + --hash=sha256:8f36aa77f74b14a2103622b196402c71330e4e7cf4122ee45c854e9a01e0eecf \ + --hash=sha256:93ff9c9c3663ce23aaa5d0b6b6abac55359e2a0c06c7e30f3771829af816be63 \ + --hash=sha256:9a635068d9793c7f0d929f39f18b58f1bd86c236e8e7b64167b86af9232ebca0 \ + --hash=sha256:9dda0788a9932776a75d8bcc020076edc308ce2508b51f6739a45e3158b3fc42 \ + --hash=sha256:a192360d512305fe92d3053e4f7627bbad1f7708f1f4ad38cccd9336acabe9ad \ + --hash=sha256:a2a14cd73d5e6c927f644b0110ee81f59c4ef506259ec9183904747f2caacc81 \ + --hash=sha256:a625df9a2a5597fa5e5266e01aefc82604bb85d291e3547981e863560d75da6c \ + --hash=sha256:a6aaa7871aebca081eb5c45f83e845bbd9712ae99fa9286dbfdcee835e63f4b4 \ + --hash=sha256:aa694c49787040943570d61c9b8fc01271bd2a2cfdd55f0d896f8aecdce1a981 \ + --hash=sha256:adb3c2124c68e480f4632d06cb7caa3b48a51678475f483b4edb1c6dee29ea43 \ + --hash=sha256:b84be41387fc00db28c13c02e30d42b02d8d8f6a4a741a1aee6012ca7f1f6039 \ + --hash=sha256:bf97ba1b5542050e9b0c74e632323b6d8136cc99d64e193abebee22fe6013c76 \ + --hash=sha256:c3bc8ee0271387e0b1e3ad0ff893645990a656e79bf9c42f4f734a3642b11d51 \ + --hash=sha256:c50c80e3591e5cf9a8dfa6007d4f10732f02e32263850999286e56ccd650cfb2 \ + --hash=sha256:c6fd031333dedcad7ce3e03d244093f43504f209e86925ebdb2b868fa4037630 \ + --hash=sha256:c87a04e7f152ee4db5dbe15f0bfa3fe628a1aeedbe66b81c43b87f52c813f9ac \ + --hash=sha256:cdcd1ba258178f7356d74170c9a81ba4eab6a5d55748bc402a5a2a6fe6be3510 \ + --hash=sha256:d4737f50d23ddd11d6685faacbb9252887dde0bc7997127757247983d74399d6 \ + --hash=sha256:d7710eb676e9bb68965cb045be0d333b4c0d7f9dfbaa7fbf121385b1588683d3 \ + --hash=sha256:d9b0a2bee3f3e03e9f29057aedb807dbf1e1d96ee1b9d0f422d9db6c8583271d \ + --hash=sha256:e65dd6e75de67b2362a4fa3e16aa627595f65054dc4ec73d64680c84d592fa61 \ + --hash=sha256:e892354a424b1e82a78ba16785822bdd40de06f19446a3d4a28047df1a4f8932 \ + --hash=sha256:e8ca8f69b7c77fa96dad99f8638c1fc4f1c0bed4acaa1baf7a3c31bdb80a0ab1 \ + --hash=sha256:e91c2b1cbeae768c98752e67e717e283d68dae45717b41f2d91eaee648e37b7d \ + --hash=sha256:e98b4f91f99b24b1da72e10621ba181f7281922384a7403094debb4199de6c00 \ + --hash=sha256:ef5fe6b7db3f501c56000ba8fe7c7e1a35ca1d45824fbeb95a67daa85cff4dbe \ + --hash=sha256:f11a3c1358bb3c68012a7b1f41b223c3db9f7bfde90b481df028cfe1c3363313 \ + --hash=sha256:f391dc92a015c478745192fe4f2e120c2af2aba18d9fe0616341cbf537599806 \ + --hash=sha256:f7cbb7a64e7ec664775195e4f81923bc8a93d9451b963f844e59118f63147a7a \ + --hash=sha256:f9bd7d260d986f3553518c5d723544346fa48031f8aef3265d786aeb59845bc0 \ + --hash=sha256:fc481f742ccda20e9ed760b8f978dff50f2a4e9bc253ac167fb0a05b0783ef52 \ + --hash=sha256:ff3fe9eb621fbb9db4c4be33ab946f6bac5f98ab50ca8d3260503653656c961b \ + --hash=sha256:ffea6b2986a797c84b6d1d27d4f43e0b83c73fc16281a930891cd237cf0d08fe + # via good-price playwright==1.62.0 \ --hash=sha256:5108bd5b3e87169ddf269feee097da5893af7f8aea4634dfc840518d64c1f1da \ --hash=sha256:92c0d98ed04eb35af557b709875edba415b1f548bdb22ddb5bb3e1e6c835c2f1 \ diff --git a/tests/conftest.py b/tests/conftest.py index de65400..bbafb87 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -3,6 +3,7 @@ import pytest +import goodprice.models # noqa: F401 -- register tables before fixture creation from goodprice.config import Settings from goodprice.db import Base, make_session_factory diff --git a/tests/test_media_processing.py b/tests/test_media_processing.py new file mode 100644 index 0000000..cd038d8 --- /dev/null +++ b/tests/test_media_processing.py @@ -0,0 +1,62 @@ +from dataclasses import replace + +import httpx +import pytest + +from goodprice.config import Settings +from goodprice.media.budget import MediaBudget +from goodprice.media.download import MediaDownloader +from goodprice.media.processing import MediaProcessor, prepare_image +from goodprice.media.types import MediaCandidate + + +def image_bytes(color="red"): + from io import BytesIO + + from PIL import Image + data = BytesIO() + Image.new("RGB", (1600, 900), color).save(data, "PNG") + return data.getvalue() + + +def test_image_is_decoded_resized_and_bad_files_rejected(tmp_path): + from PIL import Image + path = tmp_path / "input.part" + path.write_bytes(image_bytes()) + output = prepare_image(path) + with Image.open(output) as image: + assert image.size == (1280, 720) + assert image.format == "JPEG" + path.write_bytes(b"not an image") + with pytest.raises(ValueError): + prepare_image(path) + + +def test_signed_url_changes_same_content_only_decode_once(tmp_path, monkeypatch): + import goodprice.media.processing as module + calls = [] + original = module.prepare_image + def tracked(*args, **kwargs): + calls.append(1) + return original(*args, **kwargs) + monkeypatch.setattr(module, "prepare_image", tracked) + budget = MediaBudget(tmp_path, 256 * 1024**2, 86400, safety_bytes=0) + download = MediaDownloader(budget, httpx.MockTransport(lambda r: httpx.Response(200, content=image_bytes())), lambda h,p: ["93.184.216.34"]) + processor = MediaProcessor(budget, download) + settings = Settings(_env_file=None) + candidate = MediaCandidate("one", "image", "https://cdn.example/a?sig=one") + first = processor.process([candidate], settings) + second = processor.process([replace(candidate, url="https://cdn.example/a?sig=two")], settings) + assert len(first) == len(second) == 1 + assert first[0].content_hash == second[0].content_hash + assert len(calls) == 1 + assert budget.usage_bytes < 2 * 1024**2 + first[0].local_path.unlink() + assert processor.process([candidate], settings)[0].local_path.is_file() + + +def test_video_disabled_and_failure_never_make_fake_frames(tmp_path): + budget = MediaBudget(tmp_path, 256 * 1024**2, 86400, safety_bytes=0) + def forbidden(request): raise AssertionError("disabled video downloaded") + processor = MediaProcessor(budget, MediaDownloader(budget, httpx.MockTransport(forbidden))) + assert processor.process([MediaCandidate("v", "video", "https://cdn.example/v.mp4")], Settings(_env_file=None)) == [] diff --git a/tests/test_video_frames.py b/tests/test_video_frames.py new file mode 100644 index 0000000..23bdcf4 --- /dev/null +++ b/tests/test_video_frames.py @@ -0,0 +1,54 @@ +import json +import subprocess + +import pytest + +from goodprice.media.frames import MediaDecodeError, extract_frames, probe_video + + +def test_missing_tools_and_remote_input_are_explicit(tmp_path, monkeypatch): + monkeypatch.setattr("goodprice.media.frames.shutil.which", lambda tool: None) + path = tmp_path / "file.mp4" + path.write_bytes(b"video") + with pytest.raises(MediaDecodeError, match="FFmpeg"): + probe_video(path, 10) + with pytest.raises(MediaDecodeError): + probe_video("https://example.com/video.mp4", 10) + + +def test_probe_uses_local_only_protocols_and_checks_dimensions(tmp_path, monkeypatch): + path = tmp_path / "a.mp4" + path.write_bytes(b"a") + monkeypatch.setattr("goodprice.media.frames.shutil.which", lambda tool: tool) + calls = [] + def fake(args, **kwargs): + calls.append((args, kwargs)) + return subprocess.CompletedProcess(args, 0, json.dumps({"streams": [{"width": 640, "height": 480}], "format": {"duration": "10"}})) + monkeypatch.setattr("goodprice.media.frames.subprocess.run", fake) + assert probe_video(path, 10).duration_seconds == 10 + assert calls[0][0][calls[0][0].index("-protocol_whitelist") + 1] == "file" + assert calls[0][1]["timeout"] == 10 + assert not calls[0][1].get("shell") + + +def test_probe_timeout_and_excessive_duration_not_extracted(tmp_path, monkeypatch): + path = tmp_path / "a.mp4" + path.write_bytes(b"a") + monkeypatch.setattr("goodprice.media.frames.shutil.which", lambda tool: tool) + def fake(*args, **kwargs): + raise subprocess.TimeoutExpired("ffprobe", 10) + monkeypatch.setattr("goodprice.media.frames.subprocess.run", fake) + with pytest.raises(MediaDecodeError, match="超时"): + probe_video(path, 10) + + +def test_real_generated_video_samples_start_middle_end(tmp_path): + import shutil + if not shutil.which("ffmpeg") or not shutil.which("ffprobe"): + pytest.skip("FFmpeg absent on host; required in Docker validation") + path = tmp_path / "test.mp4" + subprocess.run(["ffmpeg", "-v", "error", "-f", "lavfi", "-i", "testsrc2=size=160x120:rate=4", "-t", "3", "-pix_fmt", "yuv420p", str(path)], check=True, timeout=20) + frames = extract_frames(path, 3, tmp_path / "frames", 20, 1) + assert len(frames) == 3 + assert [f.timestamp_ms for f in frames] == [0, 1375, 2750] + assert all(f.path.is_file() and len(f.content_hash) == 64 for f in frames) diff --git a/uv.lock b/uv.lock index b053106..a70965f 100644 --- a/uv.lock +++ b/uv.lock @@ -232,9 +232,11 @@ dependencies = [ { name = "apscheduler" }, { name = "beautifulsoup4" }, { name = "fastapi" }, + { name = "httpcore" }, { name = "httpx" }, { name = "jinja2" }, { name = "json-repair" }, + { name = "pillow" }, { name = "playwright" }, { name = "pydantic" }, { name = "pydantic-settings" }, @@ -258,9 +260,11 @@ requires-dist = [ { name = "apscheduler", specifier = ">=3.10,<4" }, { name = "beautifulsoup4", specifier = ">=4.12" }, { name = "fastapi", specifier = ">=0.110" }, + { name = "httpcore", specifier = ">=1,<2" }, { name = "httpx", specifier = ">=0.27" }, { name = "jinja2", specifier = ">=3.1" }, { name = "json-repair", specifier = ">=0.30" }, + { name = "pillow", specifier = ">=10.4" }, { name = "playwright", specifier = "==1.62.0" }, { name = "pydantic", specifier = ">=2.6" }, { name = "pydantic-settings", specifier = ">=2.2" }, @@ -592,6 +596,96 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/63/34/ba1c580383c9eada3711951fef0795c80b829a078d72188184bcab9dd527/packaging-26.3-py3-none-any.whl", hash = "sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c", size = 129956, upload-time = "2026-08-04T18:15:27.159Z" }, ] +[[package]] +name = "pillow" +version = "13.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ce/a4/9cc30690a394a8305e8be4c9036aa16d0dfddadd09079860b582f562496d/pillow-13.0.0.tar.gz", hash = "sha256:cdcd1ba258178f7356d74170c9a81ba4eab6a5d55748bc402a5a2a6fe6be3510", size = 47058631, upload-time = "2026-10-16T22:18:09.718Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6c/d7/af6af1b8cc55e07cb0489d5b0950bf5a9bb1461eeba825700b6d8d0e26ea/pillow-13.0.0-cp311-cp311-macosx_10_10_x86_64.whl", hash = "sha256:6f38f3f885a09cafc93fd667b1173a92f379e43785d635e7983c893f0da319fb", size = 5592324, upload-time = "2026-10-16T22:15:24.672Z" }, + { url = "https://files.pythonhosted.org/packages/0b/d2/2c12cb4fbb5d95f38d1001ff3d95fafcb830077018cba40e56a8f932dfbc/pillow-13.0.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:37290fbef0c73e2d51d594eb78fded7dc7170d1c3ad0a890269e6fd8c8c4e061", size = 4856298, upload-time = "2026-10-16T22:15:27.669Z" }, + { url = "https://files.pythonhosted.org/packages/5c/ee/229b27472483a4801ea94990815e710391836afdf987b4b722b62bd75db6/pillow-13.0.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7c88c10dd432f3b39ccdafc9ed81d0c21b8fd634502574da76d1bfa4fe2c76d0", size = 6517304, upload-time = "2026-10-16T22:15:29.43Z" }, + { url = "https://files.pythonhosted.org/packages/36/46/f509b74787f40ce0627501089d630b2c741b63b244c1f40b640b3b01e181/pillow-13.0.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a625df9a2a5597fa5e5266e01aefc82604bb85d291e3547981e863560d75da6c", size = 7140667, upload-time = "2026-10-16T22:15:31.368Z" }, + { url = "https://files.pythonhosted.org/packages/aa/a4/5a398b8ec2c37145397d6461c0b551c23ec03072aa64f357aed25e582b00/pillow-13.0.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:10b5019883c98f1dd819ca48ffc6f33a91cee04948fe637c605f4345c7cafd6f", size = 6610583, upload-time = "2026-10-16T22:15:34.025Z" }, + { url = "https://files.pythonhosted.org/packages/4f/7b/f2f1109b230fd4e2d57ab2305de31adc37ea9b4ff8719436c623cb484415/pillow-13.0.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:f391dc92a015c478745192fe4f2e120c2af2aba18d9fe0616341cbf537599806", size = 7218241, upload-time = "2026-10-16T22:15:36.214Z" }, + { url = "https://files.pythonhosted.org/packages/ab/d3/069edb9083b267746baf59756282cacef904d3475c5288c7216fb5195a98/pillow-13.0.0-cp311-cp311-win32.whl", hash = "sha256:6a0920cb55df5a4d3f6862964908078c0bd59332bdd1bcd20d14e8c70ef27cc5", size = 6515278, upload-time = "2026-10-16T22:15:38.129Z" }, + { url = "https://files.pythonhosted.org/packages/7d/f6/6e2d96413e4c9dbec12ea584d9e646d602877d4cce3ea669099f959deac6/pillow-13.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:d7710eb676e9bb68965cb045be0d333b4c0d7f9dfbaa7fbf121385b1588683d3", size = 7278471, upload-time = "2026-10-16T22:15:40.286Z" }, + { url = "https://files.pythonhosted.org/packages/70/e7/baff21f8cb60496c8c250609501b88919eeedb4088d3da106f6c62f3f3c0/pillow-13.0.0-cp311-cp311-win_arm64.whl", hash = "sha256:12fbb851cf323cc19aac3eea1df5ad17b57e91e9fe3cfd06100dba7bb8208c13", size = 2698981, upload-time = "2026-10-16T22:15:42.637Z" }, + { url = "https://files.pythonhosted.org/packages/7a/b8/58d79332229482eb0d27c82973850808c19d37e5bdc14443d3417bd373f5/pillow-13.0.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:67c8d44ad89fa7eb88de5d6ef6199b1e03dd24974f85a67639fc8badb801fe79", size = 5568643, upload-time = "2026-10-16T22:15:44.474Z" }, + { url = "https://files.pythonhosted.org/packages/39/4b/f2ed53d83aa44ac19599ac0a54897f867fafb95f406980d1e24083f8cbe3/pillow-13.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc481f742ccda20e9ed760b8f978dff50f2a4e9bc253ac167fb0a05b0783ef52", size = 4852329, upload-time = "2026-10-16T22:15:46.284Z" }, + { url = "https://files.pythonhosted.org/packages/3b/bb/9e30e8ed18c20800ee327a12107d5249ad8dc444867b3adfcc21654e3c71/pillow-13.0.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8bf295fe459343bc6093c8e5e0df60bfa189e0fcb775a22e2c2488fdeccc6c0d", size = 6519309, upload-time = "2026-10-16T22:15:48.14Z" }, + { url = "https://files.pythonhosted.org/packages/e6/99/fdf65cc539e266ab93546c261bdf98411fe165c6e4e13f9dbf597097192b/pillow-13.0.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:42d63be3af4f9b951b199b0a9d23365d37ab14687e6d77b74940432c8515c4ca", size = 7144542, upload-time = "2026-10-16T22:15:50.093Z" }, + { url = "https://files.pythonhosted.org/packages/26/9b/f38b7c4d8f599537e2d1a6d546cec9beef7536ff8f5c49a3f9c78179d8ac/pillow-13.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:4bd024d567eca931b668978d672dba914d4b8e2628e2c3073a2ad178de8a2620", size = 6618042, upload-time = "2026-10-16T22:15:52.054Z" }, + { url = "https://files.pythonhosted.org/packages/a8/7c/45ee28531f89ee4582967d8a17f7a26d71ca64cd902b93c269590cd60a00/pillow-13.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:23edf4ec4b6f4f240e8df6e4985e9ee2ced86474632433e55fb1e2548744bf40", size = 7228137, upload-time = "2026-10-16T22:15:54.009Z" }, + { url = "https://files.pythonhosted.org/packages/01/bc/9c2a969cb1b70201d04fa17fd5aedf02a732870b8da86bceaaa30b0ca385/pillow-13.0.0-cp312-cp312-win32.whl", hash = "sha256:02464da86f561f410e13b061ec37c3068a9cb557a095494157742d07b3c24a27", size = 6515423, upload-time = "2026-10-16T22:15:56.25Z" }, + { url = "https://files.pythonhosted.org/packages/ff/cd/73a79ecdc822076002eb24b5e1e4a530b0bb5310fcb6d239abbec877110b/pillow-13.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:ff3fe9eb621fbb9db4c4be33ab946f6bac5f98ab50ca8d3260503653656c961b", size = 7272166, upload-time = "2026-10-16T22:15:58.597Z" }, + { url = "https://files.pythonhosted.org/packages/4a/a3/be5a1236efd085b3984e4305da8f179ae9dab80c991d2ad78f2bd5c4c843/pillow-13.0.0-cp312-cp312-win_arm64.whl", hash = "sha256:c3bc8ee0271387e0b1e3ad0ff893645990a656e79bf9c42f4f734a3642b11d51", size = 2698943, upload-time = "2026-10-16T22:16:00.637Z" }, + { url = "https://files.pythonhosted.org/packages/82/e2/9132d38df320e8099e2642d9fa538f18a7dfebf68a875a31725dc2a115aa/pillow-13.0.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:e98b4f91f99b24b1da72e10621ba181f7281922384a7403094debb4199de6c00", size = 4190620, upload-time = "2026-10-16T22:16:02.527Z" }, + { url = "https://files.pythonhosted.org/packages/cb/be/436dac0996876f65ef1916ffb711c5ef4af2ebc57414b6f16642e07f6b47/pillow-13.0.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:34fb3db1149c879343861831728c243c432d4a5afc8f1878ca7612ef8de2a033", size = 4290286, upload-time = "2026-10-16T22:16:04.861Z" }, + { url = "https://files.pythonhosted.org/packages/60/16/7c1007162e4169ea0d5bec47e2e0b2174f56c50cf5b8c02a1dc23e41037e/pillow-13.0.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:2d50500a4a3bc6469adcdb319cf05d688b08ca3bf0f22a05339f3b25b433a599", size = 3723169, upload-time = "2026-10-16T22:16:06.956Z" }, + { url = "https://files.pythonhosted.org/packages/08/bc/578ab2eaa04705a8978bd9be0fe15c3d43a8fc95097c4f7e39540e79e4fa/pillow-13.0.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ef5fe6b7db3f501c56000ba8fe7c7e1a35ca1d45824fbeb95a67daa85cff4dbe", size = 5568469, upload-time = "2026-10-16T22:16:08.664Z" }, + { url = "https://files.pythonhosted.org/packages/98/c8/ccede3dcc4f274b80fb49199eed1ef34ba05566c8b71976b9d1f111da2e3/pillow-13.0.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f11a3c1358bb3c68012a7b1f41b223c3db9f7bfde90b481df028cfe1c3363313", size = 4852187, upload-time = "2026-10-16T22:16:10.693Z" }, + { url = "https://files.pythonhosted.org/packages/fb/3c/cd0770e96d20bfd990db024bcbd43897ff3e88da1f4782cf90ea231b1d4f/pillow-13.0.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:19b66ca0a4d25cc0200b381ce824c7ea090db61427315e73cfb05dbf1b35c369", size = 6515908, upload-time = "2026-10-16T22:16:12.896Z" }, + { url = "https://files.pythonhosted.org/packages/ef/fe/f5cdaa2d1c4086e757430f368c271a58ee29cfe7378e7ae7b9d964015c25/pillow-13.0.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5f56499c1a74d2b862beb0c33b8016f99d095858455df9d21583bbbc1c15b582", size = 7142544, upload-time = "2026-10-16T22:16:14.844Z" }, + { url = "https://files.pythonhosted.org/packages/fb/b5/a54f6d44adbcd23737c8c12953f42be107e4cca64e247c0ca3a5b563061c/pillow-13.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a2a14cd73d5e6c927f644b0110ee81f59c4ef506259ec9183904747f2caacc81", size = 6613097, upload-time = "2026-10-16T22:16:16.893Z" }, + { url = "https://files.pythonhosted.org/packages/e6/b4/9e6c1207c21b2e91db0d15c16d30f3701ce5add1bd4da0e787761e66cb6e/pillow-13.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:08b826a9bf1265c76de04a42061d61ea6b109418b7826f28ff1ecdfaa26c5502", size = 7224846, upload-time = "2026-10-16T22:16:19.195Z" }, + { url = "https://files.pythonhosted.org/packages/78/9f/d3c3cb0518f3dd1eee0ef501535f53d15597a53c5c4118d178086143397a/pillow-13.0.0-cp313-cp313-win32.whl", hash = "sha256:12b0fa15879fe5a9d9eebe66e4e2d74128a773b1a4448010c9c714f45e04b9d8", size = 6516013, upload-time = "2026-10-16T22:16:21.389Z" }, + { url = "https://files.pythonhosted.org/packages/d7/ed/1b16fb66a37e1b9e41370b93b2c614c7da761a5b2a41dc333f7af4639962/pillow-13.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:9dda0788a9932776a75d8bcc020076edc308ce2508b51f6739a45e3158b3fc42", size = 7284777, upload-time = "2026-10-16T22:16:23.393Z" }, + { url = "https://files.pythonhosted.org/packages/87/7d/8d02eba9f7ecc87f8e90f731e032cc4773802f2a843763afd0c66a535ff4/pillow-13.0.0-cp313-cp313-win_arm64.whl", hash = "sha256:40a266371ece581e5de13b6635ef78d7487824af80561b6ca27694a88c19bc0f", size = 2698826, upload-time = "2026-10-16T22:16:25.382Z" }, + { url = "https://files.pythonhosted.org/packages/8e/45/fbad9c7f5e2f1745c373a434d46072bc84eddfefaa28b852425ac957fda6/pillow-13.0.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:44629d4498c9cb6b2051806155f22141df0d8a929c56134b166655232dfb2f8c", size = 4190721, upload-time = "2026-10-16T22:16:27.524Z" }, + { url = "https://files.pythonhosted.org/packages/a2/bf/2074902a259a90692c5dbf132885d21671644cbbd3f690047516243a55db/pillow-13.0.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:bf97ba1b5542050e9b0c74e632323b6d8136cc99d64e193abebee22fe6013c76", size = 4290236, upload-time = "2026-10-16T22:16:29.562Z" }, + { url = "https://files.pythonhosted.org/packages/ff/a5/3d1bb93b8ea38d15d21356c1b8e9f50971ac592aaa248f7fb71739094c0e/pillow-13.0.0-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:aa694c49787040943570d61c9b8fc01271bd2a2cfdd55f0d896f8aecdce1a981", size = 3723093, upload-time = "2026-10-16T22:16:31.249Z" }, + { url = "https://files.pythonhosted.org/packages/84/a1/be060135d85187b25fa8b2573786c0e45c62ae27527fff5257675eb97f4a/pillow-13.0.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:49c12af515073e9bf511e4ef736743bf70d531a643668722279314b937742ba4", size = 5572253, upload-time = "2026-10-16T22:16:33.103Z" }, + { url = "https://files.pythonhosted.org/packages/b5/f4/0bf6f1ac9ba7ee2ffe9914aeb410a1035a0c1a3c42f122388d7fc950bc4b/pillow-13.0.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:d9b0a2bee3f3e03e9f29057aedb807dbf1e1d96ee1b9d0f422d9db6c8583271d", size = 4852269, upload-time = "2026-10-16T22:16:35.102Z" }, + { url = "https://files.pythonhosted.org/packages/83/16/b603274617b3e6728da3a0133d4fba1ea8d7fbe0c4ea5fef9cbffa398b8b/pillow-13.0.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5933fcea5a5ba64c686f55c886a115cbfd92f2900460784a1e79909fdbafd7b6", size = 6516058, upload-time = "2026-10-16T22:16:37.228Z" }, + { url = "https://files.pythonhosted.org/packages/1d/f2/1371707ce438ba9a43f9d24089828494ec6f9396e28cd7903aa704dd6306/pillow-13.0.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e91c2b1cbeae768c98752e67e717e283d68dae45717b41f2d91eaee648e37b7d", size = 7142574, upload-time = "2026-10-16T22:16:39.172Z" }, + { url = "https://files.pythonhosted.org/packages/b9/a9/62cf68587f910bc87c5951bcb6e9430c7643a65076adf5f18b4bf350e2bd/pillow-13.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7f6299e494e46253d40c34653cac369412f44ca53b5f85d848d3f95b1ed0546c", size = 6614295, upload-time = "2026-10-16T22:16:41.117Z" }, + { url = "https://files.pythonhosted.org/packages/6c/c9/9e9b48178649730299be0bafbf5f810a5b6977538f5a92d47e4ba5f5d608/pillow-13.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f9bd7d260d986f3553518c5d723544346fa48031f8aef3265d786aeb59845bc0", size = 7224841, upload-time = "2026-10-16T22:16:43.118Z" }, + { url = "https://files.pythonhosted.org/packages/8f/48/c44e91384b1b6c072a714a63d00608acc16a6a968735e478e4be4384e5ac/pillow-13.0.0-cp314-cp314-win32.whl", hash = "sha256:2e31c4e68cffd7b1f116127626813b82fcba19edf6b940a7b0c266d6484d0055", size = 6508694, upload-time = "2026-10-16T22:16:45.883Z" }, + { url = "https://files.pythonhosted.org/packages/50/e2/7ad17747517e4c813dca78257d6bb593466f8231baf1384bc98d92745edb/pillow-13.0.0-cp314-cp314-win_amd64.whl", hash = "sha256:f7cbb7a64e7ec664775195e4f81923bc8a93d9451b963f844e59118f63147a7a", size = 7281830, upload-time = "2026-10-16T22:16:47.725Z" }, + { url = "https://files.pythonhosted.org/packages/41/9f/e5b51f2ce8ef4563215334035403ce61a244f93a2e65c24a940131618f87/pillow-13.0.0-cp314-cp314-win_arm64.whl", hash = "sha256:a6aaa7871aebca081eb5c45f83e845bbd9712ae99fa9286dbfdcee835e63f4b4", size = 2695468, upload-time = "2026-10-16T22:16:49.515Z" }, + { url = "https://files.pythonhosted.org/packages/97/73/a53502f731f6c84805a4ce87a8881e86d07a65aee64728b900d112dc703e/pillow-13.0.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:7b999816b5850a35ada08443e6e03945f1f6488144a83afe2d37b240a552aafa", size = 5578196, upload-time = "2026-10-16T22:16:51.291Z" }, + { url = "https://files.pythonhosted.org/packages/21/50/7e949959a56b5559c78ef172b70fc6025706acb289f3f648a15329e4fa6f/pillow-13.0.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:67871a13cf47275201553b536c025da777f997fa562a19eb798af4ad4b97c18f", size = 4857276, upload-time = "2026-10-16T22:16:53.399Z" }, + { url = "https://files.pythonhosted.org/packages/c9/52/43ded0ac08e77e39d17bd6d894162572201b5e8fdc4a022b14c55a1960a4/pillow-13.0.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e8ca8f69b7c77fa96dad99f8638c1fc4f1c0bed4acaa1baf7a3c31bdb80a0ab1", size = 6607008, upload-time = "2026-10-16T22:16:55.525Z" }, + { url = "https://files.pythonhosted.org/packages/0d/2e/4a57ed216d5bf877387cba4d44c02aa87abe51ad2e06404aa6b597c4da39/pillow-13.0.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6bce6a29c3898191f4c9998e4d4e8f69790caeabde46e54da4bc235ce5bde683", size = 7224281, upload-time = "2026-10-16T22:16:57.738Z" }, + { url = "https://files.pythonhosted.org/packages/5c/3d/2929b248e135752d63aaeabf53f6de797e905b7b6263f9c1e2adade20aba/pillow-13.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:816752a5ef76a80484c8c441c85593b7ea9304f6231fc5776ad1fbff7e773058", size = 6695600, upload-time = "2026-10-16T22:16:59.842Z" }, + { url = "https://files.pythonhosted.org/packages/bc/33/fb0da992cf084ed1ee6ee05e96b6191e35c351eb707ca2df874feb24ce47/pillow-13.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5bdb38ba220b61b4ec31dd1046dea6b016d3c6850a23f43ed8b58332b2fc668e", size = 7300589, upload-time = "2026-10-16T22:17:01.686Z" }, + { url = "https://files.pythonhosted.org/packages/bb/af/c30dd9ef6a3dce84ecb1b2393cac629047a32e66279056c6cb46c3ffd65c/pillow-13.0.0-cp314-cp314t-win32.whl", hash = "sha256:06903aac074d00c988e77a1fc435286dc498506d4ca8d6f69df9babddf4ecc15", size = 6517204, upload-time = "2026-10-16T22:17:03.722Z" }, + { url = "https://files.pythonhosted.org/packages/85/d5/ef3a8ae7609c8ab26570abf18194882534b480d245a9148b895a2262c004/pillow-13.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:9a635068d9793c7f0d929f39f18b58f1bd86c236e8e7b64167b86af9232ebca0", size = 7286695, upload-time = "2026-10-16T22:17:05.543Z" }, + { url = "https://files.pythonhosted.org/packages/84/3c/0861db0c69a03689ebe3e7599d35668a414deb5aaafd8255215e4e62e3d3/pillow-13.0.0-cp314-cp314t-win_arm64.whl", hash = "sha256:0d1a9cf7bc6917538b20f8d06b9ab2147e542f68849e96fc58eb66bdf6acc692", size = 2698098, upload-time = "2026-10-16T22:17:07.674Z" }, + { url = "https://files.pythonhosted.org/packages/6e/eb/e86a41962a3a31ad495a3d6a912d9002b01bba4a6fa71c28703abdca7f69/pillow-13.0.0-cp315-cp315-ios_13_0_arm64_iphoneos.whl", hash = "sha256:7d5e097eabd5d4915c27ac9f6e9325bb1e01a7c94d7348b5c4451d3fc30e8521", size = 4191213, upload-time = "2026-10-16T22:17:09.449Z" }, + { url = "https://files.pythonhosted.org/packages/dd/04/7e7445e706afec84ef06dc25b2c77f9cecff242c23730c62928ad1089d4a/pillow-13.0.0-cp315-cp315-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:51ebb529fc5e8abdd55302f6c3f8943037348ec2384329a3e55677f69ae2fdfc", size = 4290650, upload-time = "2026-10-16T22:17:11.196Z" }, + { url = "https://files.pythonhosted.org/packages/46/85/d534089068ac40c0f61f650389a296c812eab745a8a733f81feec39ae955/pillow-13.0.0-cp315-cp315-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:c87a04e7f152ee4db5dbe15f0bfa3fe628a1aeedbe66b81c43b87f52c813f9ac", size = 3723078, upload-time = "2026-10-16T22:17:12.772Z" }, + { url = "https://files.pythonhosted.org/packages/ce/2b/00b5c3b401df538b5dbe311d3899730fae5ba1e21cb8b80a128887ac6747/pillow-13.0.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:20722cd5375533ad754053363a80b8fb30c79d37930a6f7228030e237ce3bd5d", size = 5572292, upload-time = "2026-10-16T22:17:14.539Z" }, + { url = "https://files.pythonhosted.org/packages/72/f5/a6b0610f946e9b54ae888ba7ad29d6e2f8d9b11f69e212d70416b7186c09/pillow-13.0.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:04a63720f79e22fd54d897c233a68314c50ca0603360f54ce2bd5db8f0f43782", size = 4852222, upload-time = "2026-10-16T22:17:16.521Z" }, + { url = "https://files.pythonhosted.org/packages/1e/d0/c0ff25bb0427ec70f32aac995453321fc3c4494a46c7292ff02d9d260b04/pillow-13.0.0-cp315-cp315-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4f8db54177035a2b5f717f94475b74907e12a96dc74f6d487f3966b0a4694071", size = 6527349, upload-time = "2026-10-16T22:17:18.534Z" }, + { url = "https://files.pythonhosted.org/packages/85/11/e46fa0bcc0fc117f1eacc206b3d51f04d583fab328b4b740672d0438d83f/pillow-13.0.0-cp315-cp315-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8f36aa77f74b14a2103622b196402c71330e4e7cf4122ee45c854e9a01e0eecf", size = 7152873, upload-time = "2026-10-16T22:17:20.415Z" }, + { url = "https://files.pythonhosted.org/packages/7c/53/2bdeca705f3a58679fa9201233193c86a30238340ca29b3a152c3fcc719b/pillow-13.0.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:149a1c323ba366f90b48e6f1a6db6582eb50512df374588de56fc909abc0f0f3", size = 6626871, upload-time = "2026-10-16T22:17:22.361Z" }, + { url = "https://files.pythonhosted.org/packages/14/de/3829030f209722ade3753bb456b805a0e7f885cf885b483407365c82e9ca/pillow-13.0.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:e892354a424b1e82a78ba16785822bdd40de06f19446a3d4a28047df1a4f8932", size = 7236439, upload-time = "2026-10-16T22:17:24.352Z" }, + { url = "https://files.pythonhosted.org/packages/44/b1/67fc37388382bc036a74f04da5bb6fcbf90ec3ac1f45893e06176c373fd3/pillow-13.0.0-cp315-cp315-win32.whl", hash = "sha256:8e459b613cf31f8980f584387b4fe47bf08e076e288d61aa8eba1f7fd4ddbb81", size = 6508688, upload-time = "2026-10-16T22:17:26.314Z" }, + { url = "https://files.pythonhosted.org/packages/44/05/fdb6a79aee10861165526b41b7b440e36ee1ff58613b2fbefe3acbf06d4f/pillow-13.0.0-cp315-cp315-win_amd64.whl", hash = "sha256:d4737f50d23ddd11d6685faacbb9252887dde0bc7997127757247983d74399d6", size = 7281107, upload-time = "2026-10-16T22:17:28.198Z" }, + { url = "https://files.pythonhosted.org/packages/e6/9c/6c6f7dc2a566467baf121394e83de25ac287cffe34cc030726beb4be0224/pillow-13.0.0-cp315-cp315-win_arm64.whl", hash = "sha256:2310e2adfc13c6d457d7d0f2ffdd5c56a74a317650cf765887df8d515a647ac0", size = 2695513, upload-time = "2026-10-16T22:17:30.013Z" }, + { url = "https://files.pythonhosted.org/packages/23/16/e9120defdb8bff693b582d035312acc6f530cdc2bc5238a87f410efccd82/pillow-13.0.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:21c32061a5512dc13f87cc7303e120e4d77c2f70df89a35af0e3ce246e5ac57f", size = 5578129, upload-time = "2026-10-16T22:17:31.642Z" }, + { url = "https://files.pythonhosted.org/packages/47/22/76faf2bb2448f48a84cf894c1e2bb1090d7d2b1aa521ece131cb2f05d631/pillow-13.0.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:1c774b9c0ba0d0b356f71530677b39028878a2cae5a93e0a13779cfd17120f88", size = 4857428, upload-time = "2026-10-16T22:17:33.43Z" }, + { url = "https://files.pythonhosted.org/packages/b3/64/0861ea4e03c50b14af9ed9dd8b9f388ab42f445c623a9fcbab19aef7aa55/pillow-13.0.0-cp315-cp315t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:20822140c9a2ba9c5e42df342b0a892ab2e102fc0e3db967a133bb9ce3f898bb", size = 6612582, upload-time = "2026-10-16T22:17:35.415Z" }, + { url = "https://files.pythonhosted.org/packages/5c/20/cd1e9b3f60a9f3d5ebe46da36955c26004092a501f7f2c42f9fcc26611bc/pillow-13.0.0-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ffea6b2986a797c84b6d1d27d4f43e0b83c73fc16281a930891cd237cf0d08fe", size = 7230176, upload-time = "2026-10-16T22:17:37.362Z" }, + { url = "https://files.pythonhosted.org/packages/b6/ef/ffb0a2fc85690714fb7ac662956949645d57f15bf4426f520da361e7b6e3/pillow-13.0.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:05cd2cd39055d0b9650060e4513ef4a9a0aea870083950bd046a0d6490f93a22", size = 6703640, upload-time = "2026-10-16T22:17:39.313Z" }, + { url = "https://files.pythonhosted.org/packages/e0/c2/6402ee3ccb3363aeed8fdb3f9fe56703ac5377a539f0b81b53999d7e121f/pillow-13.0.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:c6fd031333dedcad7ce3e03d244093f43504f209e86925ebdb2b868fa4037630", size = 7309231, upload-time = "2026-10-16T22:17:41.021Z" }, + { url = "https://files.pythonhosted.org/packages/fc/96/d6b82e8b8c06ad143b66b12e0b31051e4bf489b4fabca1f390a14fe1ebde/pillow-13.0.0-cp315-cp315t-win32.whl", hash = "sha256:c50c80e3591e5cf9a8dfa6007d4f10732f02e32263850999286e56ccd650cfb2", size = 6517200, upload-time = "2026-10-16T22:17:42.886Z" }, + { url = "https://files.pythonhosted.org/packages/60/d4/0062e6005e9b270b165015aafd343364ac378ec5a685431ea42baad33357/pillow-13.0.0-cp315-cp315t-win_amd64.whl", hash = "sha256:adb3c2124c68e480f4632d06cb7caa3b48a51678475f483b4edb1c6dee29ea43", size = 7287488, upload-time = "2026-10-16T22:17:45.56Z" }, + { url = "https://files.pythonhosted.org/packages/e4/81/b8eb349746be7bce47dd95337c6634e84910af91b59aaa9159c034c614fa/pillow-13.0.0-cp315-cp315t-win_arm64.whl", hash = "sha256:4a25756244bdcce3b50580baadf320fa48297f0eeb8e114ba64139a40d4d0fc8", size = 2698073, upload-time = "2026-10-16T22:17:47.556Z" }, + { url = "https://files.pythonhosted.org/packages/68/c0/fb7b1caf25ed5ac5d46116f09b8b0706b9559833c65c667cecb1a53153f2/pillow-13.0.0-pp311-pypy311_pp80-macosx_10_15_x86_64.whl", hash = "sha256:405be6058e206a86fedce2f715ecc2343f94072d7925251632dffa2c05f70dca", size = 5521630, upload-time = "2026-10-16T22:17:50.008Z" }, + { url = "https://files.pythonhosted.org/packages/1a/e6/4d11d5b1137f4fc080605cce410cfaae4f303ad400bdec75d27d3083fa29/pillow-13.0.0-pp311-pypy311_pp80-macosx_11_0_arm64.whl", hash = "sha256:b84be41387fc00db28c13c02e30d42b02d8d8f6a4a741a1aee6012ca7f1f6039", size = 4806023, upload-time = "2026-10-16T22:17:51.932Z" }, + { url = "https://files.pythonhosted.org/packages/f1/45/0923202a658e8ff6aafd72c1a4395ebadcdebaa182c437d8595d3dd3db32/pillow-13.0.0-pp311-pypy311_pp80-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a192360d512305fe92d3053e4f7627bbad1f7708f1f4ad38cccd9336acabe9ad", size = 5464831, upload-time = "2026-10-16T22:17:53.742Z" }, + { url = "https://files.pythonhosted.org/packages/07/52/d3e38f0cad1dadea2a658b9a7d6ba8a04048f5ae14f011b806e5c3bd6bcb/pillow-13.0.0-pp311-pypy311_pp80-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:83b37abd87b590acd8f47ca8a61f70893322b11834d36e07c2298dacad4d14d2", size = 6043133, upload-time = "2026-10-16T22:17:55.625Z" }, + { url = "https://files.pythonhosted.org/packages/c4/f9/2ed7685980c1d64f8131a7d87a19c369424747577152622b73ddeff163b7/pillow-13.0.0-pp311-pypy311_pp80-win_amd64.whl", hash = "sha256:73ee25cf8cf8b75d5a5aeed7cb6588f5e5dbfbb0fde51707b630bbb23897645c", size = 7365208, upload-time = "2026-10-16T22:17:57.509Z" }, + { url = "https://files.pythonhosted.org/packages/60/da/f6d644e631c5b370c5d0b899f3ccadd2780aab070b29a110527cf4e1fd2c/pillow-13.0.0-pp312-pypy312_pp80-macosx_10_15_x86_64.whl", hash = "sha256:8619685cad8efc4c38c66d3b36aad854523297c43d748b3b6f0cc03c90ba3463", size = 5523188, upload-time = "2026-10-16T22:17:59.334Z" }, + { url = "https://files.pythonhosted.org/packages/e4/b0/2594f202d743a35d4fe80682bfd96f277e5ca7aecb27cf64ef5bfc1d000b/pillow-13.0.0-pp312-pypy312_pp80-macosx_11_0_arm64.whl", hash = "sha256:7e91e97a929232cdb828889ee6ad0e48d7f40f4ec38818f4e4a8968b2b1af22e", size = 4806808, upload-time = "2026-10-16T22:18:01.188Z" }, + { url = "https://files.pythonhosted.org/packages/8e/ed/d504165ef79af6fafe25c76726cf30f544cc236002f7ed054f1d51adc4ff/pillow-13.0.0-pp312-pypy312_pp80-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:93ff9c9c3663ce23aaa5d0b6b6abac55359e2a0c06c7e30f3771829af816be63", size = 5464358, upload-time = "2026-10-16T22:18:02.779Z" }, + { url = "https://files.pythonhosted.org/packages/7b/fe/a10aba5503ca13e9e009b8bd0a1a4ce5584bdce246245fd6f3a83701369c/pillow-13.0.0-pp312-pypy312_pp80-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e65dd6e75de67b2362a4fa3e16aa627595f65054dc4ec73d64680c84d592fa61", size = 6043514, upload-time = "2026-10-16T22:18:04.538Z" }, + { url = "https://files.pythonhosted.org/packages/c0/37/66b12dc476041fe523bca29c9d58fbf1ea92a4eaf73acfb9e993948f1191/pillow-13.0.0-pp312-pypy312_pp80-win_amd64.whl", hash = "sha256:567f20f1c75897d7663830da6ea8af719e03d712b1d9ee7f813319f9ece42e5a", size = 7366030, upload-time = "2026-10-16T22:18:06.638Z" }, +] + [[package]] name = "playwright" version = "1.62.0" From 1e4368ba72daa98c4b2034fe7d6e9af46577134d Mon Sep 17 00:00:00 2001 From: Comui520 Date: Mon, 26 Oct 2026 12:25:33 +0800 Subject: [PATCH 11/17] feat: analyze and present traceable mixed media evidence --- README.md | 18 ++- .../validation/2026-10-26-phase2-milestone.md | 27 ++++ goodprice/analysis/media_analysis.py | 84 +++++++++++ goodprice/main.py | 8 ++ goodprice/media/download.py | 57 +++++++- goodprice/media/processing.py | 5 +- goodprice/media/selection.py | 11 +- goodprice/services/crawl_service.py | 20 ++- goodprice/services/media_service.py | 102 ++++++++++++++ goodprice/services/offer_crawl.py | 15 +- goodprice/web/media_routes.py | 128 +++++++++++++++++ goodprice/web/routes.py | 19 ++- goodprice/web/templates/listings_detail.html | 25 +++- goodprice/web/templates/media_settings.html | 20 +++ goodprice/web/templates/settings.html | 1 + pyproject.toml | 3 + tests/test_media_analysis.py | 62 +++++++++ tests/test_media_api.py | 52 +++++++ tests/test_media_regressions.py | 131 ++++++++++++++++++ tests/test_media_selection.py | 5 + tests/test_media_service.py | 49 +++++++ 21 files changed, 823 insertions(+), 19 deletions(-) create mode 100644 docs/validation/2026-10-26-phase2-milestone.md create mode 100644 goodprice/analysis/media_analysis.py create mode 100644 goodprice/services/media_service.py create mode 100644 goodprice/web/media_routes.py create mode 100644 goodprice/web/templates/media_settings.html create mode 100644 tests/test_media_analysis.py create mode 100644 tests/test_media_api.py create mode 100644 tests/test_media_regressions.py create mode 100644 tests/test_media_service.py diff --git a/README.md b/README.md index cfaea46..3aedd2e 100644 --- a/README.md +++ b/README.md @@ -222,7 +222,23 @@ Conda 不是运行本项目的必需条件;新用户优先选择 Docker 或 uv 设置最低品相分后,未核验的图片或多规格共享样品不会绕过门槛。 当前支持可明确识别的商品范围 DOM 数据与 Schema.org Product/Offer,尚未取得真实多规格页面样本来验证所有闲鱼控件, -不能识别时显示待核验而不是猜测。视频分析、分阶段并发调度与完整交互改造仍属后续工作,不能将本开发分支当作已完成的稳定发布。 +不能识别时显示待核验而不是猜测。分阶段并发调度与完整交互改造仍属后续工作,不能将本开发分支当作已完成的稳定发布。 + +### 开发分支:图片与视频证据 + +设置页的“媒体证据与用量”可配置保留数量、每次原图/帧配额、总输入上限、超时、缓存和解码线程。 +视频默认关闭;新安装默认一次最多8张,混合媒体先分配4张原图+4帧,单一媒体可借用余量。 +旧数据库默认保持4张上限;提高预算需明确确认费用、延迟、内存、存储和提供商限制,硬件建议不会提高图片数。 +模型上限未知时按8张处理,只有确认提供商支持后才可提高(最高16);不自动拆成多个收费请求。 + +只处理详情主媒体的直接HTTP(S)资源。blob、HLS/DASH、受保护流及未知页面结构会明确跳过, +不保证所有闲鱼页面的视频组件都能识别。视频仅采样起/中/末画面,不验证音频、整段动作或动态故障。 +没有明确规格关联的图片/帧是共享证据,不能据此确认某一SKU的实物品相;任务设置最低品相分时仍需有效评分。 + +Docker镜像内置FFmpeg/ffprobe。本地uv/Conda运行须自行安装FFmpeg并加入PATH;缺工具时视频不可分析,应用和图片功能仍可用。 +视频默认最大100MiB、180秒,下载60秒、解码30秒,缓存1GiB/7天;处理中的视频和临时输出同样计入磁盘预算。 +缓存被清理后保留证据摘要与时间位置,页面显示文件不可用;媒体文件通过内部编号访问,不公开签名下载地址或本地路径。 +相同内容的签名URL变化不会单独触发重复视觉收费,内容/需求/模型/选图设置变化才使分析缓存失效。 应用现使用无导入副作用的工厂;自定义 ASGI 部署请使用 `uvicorn goodprice.main:create_app --factory`, 原有 `python -m goodprice` 启动方式不变。升级旧库前先停止服务并备份数据库,回退使用旧镜像与备份库成对恢复。 diff --git a/docs/validation/2026-10-26-phase2-milestone.md b/docs/validation/2026-10-26-phase2-milestone.md new file mode 100644 index 0000000..4297cf7 --- /dev/null +++ b/docs/validation/2026-10-26-phase2-milestone.md @@ -0,0 +1,27 @@ +# Media evidence stage validation + +This is an unreleased development branch. The production checkout, database and personal launchers were not changed. + +## Implemented + +- Configurable retained images/videos/frames, fair per-request input quotas, provider limit confirmation and old-install four-input default. +- Public-address media downloads with pinned socket destinations and original TLS hostname, per-hop redirect validation, byte/time limits and sanitized logs. +- Shared cache accounting for reservations and outputs, active-file pins, content-based decode reuse and bounded local-only FFmpeg processing. +- JPEG data URLs for chat and responses, allowed evidence IDs, shared-SKU/conflict safeguards, seller text in visual inputs/cache identity and manual forced reanalysis. +- Media settings with explicit cost confirmation, controlled image viewing independent of vision configuration, frame positions and scoreless observations/conflicts. +- Wheel package data now includes templates; installed applications can render pages outside the source checkout. + +## Verification + +- Before independent review: 337 passed, one host FFmpeg skip, coverage 88.94%. +- Review fixes and dormant stage-three foundations together: **356 passed, one skip, coverage 88.88%**. Ruff and lock check passed. Logs remain in ignored local execution scratch. +- One independent review found nine important defects, all addressed by regression tests. Duplicate content was regraded important because it wastes paid input slots, and deduplicated. +- A deliberately invalid Range response reproduces the pin leak when the response object is retained; cleanup now executes in a response-level `finally`. +- `second-eye:upgrade-phase2` built successfully. A container with networking disabled generated a synthetic video and verified actual FFmpeg probe/start-middle-end frames, image resizing, optional SDK import, application pages and media settings validation. No production volume or external model/notification credentials were used. +- Windows host lacks FFmpeg; application startup remains supported and video capability reports unavailable. The corresponding real-tool test was run in Linux Docker instead. + +## Limits and remaining work + +Actual Goofish video widgets and SKU/media associations still need sanitized real samples. Only main-media DOM adapters and direct HTTP(S) files are implemented; blob/HLS/DASH/DRM are explicitly unsupported. Live provider compatibility and real DNS/TLS timing were not exercised against external services. No macOS or ARM runtime was tested. + +The media stage does not complete the overall upgrade. Persistent execution foundations are being developed; the application still uses the old task queue until the staged coordinator, delivery lifecycle and resource controls are integrated. Full navigation/settings interaction work and final performance comparison remain required. No speedup is claimed here. diff --git a/goodprice/analysis/media_analysis.py b/goodprice/analysis/media_analysis.py new file mode 100644 index 0000000..8e779fa --- /dev/null +++ b/goodprice/analysis/media_analysis.py @@ -0,0 +1,84 @@ +import base64 +import hashlib +import json + +from goodprice.analysis.json_extract import extract_json_object +from goodprice.media.types import MediaSelection +from goodprice.offers.types import OfferCandidate + +PROMPT_VERSION = "media-evidence-1" +SYSTEM = """根据提供的商品媒体证据评估品相。卖家声明不等于观察事实。 +视频帧仅覆盖列出的采样时间,不能声称看过整个视频、音频或验证动态故障。 +不判断实际售价,不推断未证实的型号/容量。不把样品品相当成特定SKU实物品相。 +返回JSON: condition_score(0到10或null), observations/seller_claims/conflicts/unverified(数组, +每项{text:字符串,evidence_ids:本次提供的编号数组})。任何观察和分数必须有本次证据支持。 +""" + + +def validate_evidence_references(verdict: dict, allowed_ids: set[str]) -> dict: + result = {} + cited = set() + for field in ("observations", "seller_claims", "conflicts", "unverified"): + items = verdict.get(field, []) + if not isinstance(items, list) or len(items) > 30: + raise ValueError("媒体输出格式无效") + cleaned = [] + for item in items: + if not isinstance(item, dict): + raise ValueError("媒体输出缺少证据关联") + ids = item.get("evidence_ids", []) + if not isinstance(ids, list) or any(not isinstance(i, str) or i not in allowed_ids for i in ids): + raise ValueError("模型引用了未提供的证据") + if field == "observations" and not ids: + raise ValueError("观察缺少证据") + cited.update(ids) + cleaned.append({"text": str(item.get("text", ""))[:1000], "evidence_ids": ids}) + result[field] = cleaned + score = verdict.get("condition_score") + if score is not None and (type(score) is not int or not 0 <= score <= 10 or not cited): + raise ValueError("品相评分缺少有效证据或超出范围") + result["condition_score"] = score + return result + + +def analysis_key(offer, selection, client, requirement, settings, seller_text="") -> str: + values = [PROMPT_VERSION, offer.key, dict(offer.attributes), requirement, seller_text, + [(e.content_hash, e.offer_key, e.timestamp_ms) for e in selection.items], selection.limitations, + {k: v for k, v in settings.model_dump(mode="json").items() if k.startswith("media_") or k == "video_analysis_enabled"}, + client.model, client.base_url, client.api_format] + return hashlib.sha256(json.dumps(values, sort_keys=True).encode()).hexdigest() + + +def analyze_media(offer: OfferCandidate, selection: MediaSelection, client, + requirement: str, request_limit_bytes: int, seller_text: str = "") -> dict: + limitations = list(selection.limitations) + if not selection.items: + return {"condition_score": None, "observations": [], "limitations": limitations + ["no_visual_input"]} + descriptors = [{"id": e.id, "kind": e.kind, "offer_key": e.offer_key, + "video_id": e.video_id, "timestamp_ms": e.timestamp_ms} for e in selection.items] + content = [{"type": "text", "text": json.dumps({"offer_key": offer.key, + "attributes": dict(offer.attributes), "requirement": requirement, "seller_text": seller_text, + "evidence": descriptors, + "limitations": limitations}, ensure_ascii=False)}] + for evidence in selection.items: + if evidence.local_path.stat().st_size * 4 // 3 > request_limit_bytes: + raise ValueError("视觉请求超过字节预算,请降低输入设置") + url = "data:image/jpeg;base64," + base64.b64encode(evidence.local_path.read_bytes()).decode() + content.append({"type": "image_url", "image_url": {"url": url}}) + payload = client.payload(content, system=SYSTEM) + # Both HTTP API formats contain the same base64 inputs. Reserve conversion + # overhead instead of silently removing evidence to fit the request. + if len(json.dumps(payload).encode()) + 1024 > request_limit_bytes: + raise ValueError("视觉请求超过字节预算,请降低输入设置") + ids = {e.id for e in selection.items} + result = client.complete(payload, parser=lambda raw: validate_evidence_references(extract_json_object(raw), ids)) + if offer.key != "default" and any(e.offer_key != offer.key for e in selection.items): + result["condition_score"] = None + limitations.append("shared_media") + if result.get("conflicts"): + result["condition_score"] = None + limitations.append("conflicting_evidence") + result["limitations"] = limitations + result["evidence_ids"] = sorted(ids) + result["prompt_version"] = PROMPT_VERSION + return result diff --git a/goodprice/main.py b/goodprice/main.py index 0e5d92b..0a522fa 100644 --- a/goodprice/main.py +++ b/goodprice/main.py @@ -16,6 +16,7 @@ from goodprice.services.settings_service import SettingsService from goodprice.services.task_queue import DEFAULT_GAP_SECONDS, TaskQueue from goodprice.services.task_service import TaskService +from goodprice.web.media_routes import router as media_router from goodprice.web.offer_routes import router as offer_router from goodprice.web.routes import router @@ -51,8 +52,11 @@ def _make_crawl_service(session_factory, settings_service, guard): runtime = settings_service.get() from goodprice.analysis.llm import LLMClient from goodprice.crawler.xianyu import XianyuAdapter + from goodprice.media.budget import MediaBudget + from goodprice.media.processing import MediaProcessor from goodprice.notify.log import LogNotifier from goodprice.notify.registry import build_enabled_notifiers + from goodprice.services.media_service import MediaService adapter = XianyuAdapter(cookie=runtime.xianyu_cookie, proxy=runtime.proxy) seller_service = SellerService(session_factory, adapter=adapter) @@ -83,6 +87,9 @@ def _make_crawl_service(session_factory, settings_service, guard): settings_service=settings_service, guard=guard, seller_service=seller_service, + media_service=MediaService(session_factory, MediaProcessor(MediaBudget( + runtime.data_root / "media-cache", runtime.media_cache_mib * 1024**2, + runtime.media_retention_days * 86400))), ) @@ -171,6 +178,7 @@ async def lifespan(app: FastAPI): app.state.login_session = login_session app.include_router(router) app.include_router(offer_router) + app.include_router(media_router) return app diff --git a/goodprice/media/download.py b/goodprice/media/download.py index 4dc3c92..9b7c040 100644 --- a/goodprice/media/download.py +++ b/goodprice/media/download.py @@ -1,12 +1,15 @@ """Public HTTP media only; no environment proxy, cookies or second DNS lookup.""" import ipaddress +import logging import socket import time import uuid +from contextvars import ContextVar from urllib.parse import urljoin, urlsplit import httpx -from httpcore import ConnectError +from httpcore import ConnectError, ReadTimeout +from httpcore._backends.base import NetworkStream from httpcore._backends.sync import SyncBackend, SyncStream from goodprice.media.budget import MediaBudget @@ -17,6 +20,20 @@ class MediaDownloadError(RuntimeError): pass +_media_request = ContextVar("media_request", default=False) + + +class _RedactMediaRequest(logging.Filter): + def filter(self, record): + if _media_request.get(): + record.msg, record.args = "媒体 HTTP 请求(地址已隐藏)", () + return True + + +_log_filter = _RedactMediaRequest() +logging.getLogger("httpx").addFilter(_log_filter) + + def resolve_public(host, port): return list({row[4][0] for row in socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)}) @@ -42,9 +59,36 @@ def public_addresses(url, resolver): return host, parsed.port or (443 if parsed.scheme == "https" else 80), addresses[0] +class DeadlineStream(NetworkStream): + def __init__(self, stream, deadline): + self.stream, self.deadline = stream, deadline + + def _timeout(self, timeout): + remaining = self.deadline - time.monotonic() + if remaining <= 0: + raise ReadTimeout("媒体下载超时") + return min(timeout, remaining) if timeout is not None else remaining + + def read(self, max_bytes, timeout=None): + return self.stream.read(max_bytes, self._timeout(timeout)) + + def write(self, buffer, timeout=None): + return self.stream.write(buffer, self._timeout(timeout)) + + def start_tls(self, ssl_context, server_hostname=None, timeout=None): + return DeadlineStream(self.stream.start_tls(ssl_context, server_hostname, self._timeout(timeout)), self.deadline) + + def close(self): + self.stream.close() + + def get_extra_info(self, info): + return self.stream.get_extra_info(info) + + class PinnedBackend(SyncBackend): - def __init__(self, pins): + def __init__(self, pins, deadline=None): self.pins = pins + self.deadline = deadline def connect_tcp(self, host, port, timeout=None, local_address=None, socket_options=None): addr = self.pins.get((host, port)) @@ -58,7 +102,8 @@ def connect_tcp(self, host, port, timeout=None, local_address=None, socket_optio sock.setsockopt(*option) # socket.connect receives a numeric address; hostname cannot rebind. sock.connect((str(ip), port)) - return SyncStream(sock) + stream = SyncStream(sock) + return DeadlineStream(stream, self.deadline) if self.deadline else stream except OSError as exc: sock.close() raise ConnectError("媒体连接失败") from exc @@ -76,6 +121,7 @@ def download(self, candidate: MediaCandidate, max_bytes: int, timeout_seconds: i reservation = reservation or self.budget.reserve(candidate.key, max_bytes) path = reservation.directory / f"{uuid.uuid4().hex}.part" deadline = time.monotonic() + timeout_seconds + logging_token = _media_request.set(True) try: url = candidate.url for _ in range(6): @@ -88,7 +134,7 @@ def download(self, candidate: MediaCandidate, max_bytes: int, timeout_seconds: i transport = httpx.HTTPTransport(trust_env=False, retries=0) # httpcore 1.x backend preserves original Host and TLS SNI; # only the socket destination is replaced with verified IP. - transport._pool._network_backend = PinnedBackend({(host, port): addr}) + transport._pool._network_backend = PinnedBackend({(host, port): addr}, deadline) with httpx.Client(transport=transport, trust_env=False, timeout=remaining, follow_redirects=False) as client: with client.stream("GET", url, headers={"Accept-Encoding": "identity"}) as response: @@ -103,7 +149,7 @@ def download(self, candidate: MediaCandidate, max_bytes: int, timeout_seconds: i raise MediaDownloadError("媒体文件超过大小上限") size = 0 with path.open("xb") as output: - for chunk in response.iter_bytes(64 * 1024): + for chunk in response.iter_bytes(): size += len(chunk) if size > max_bytes or time.monotonic() > deadline: raise MediaDownloadError("媒体下载超过字节或时间预算") @@ -122,5 +168,6 @@ def download(self, candidate: MediaCandidate, max_bytes: int, timeout_seconds: i # Never propagate signed URL, cookies, DNS details or response body. raise MediaDownloadError("媒体下载失败") from None finally: + _media_request.reset(logging_token) if own: self.budget.release(reservation.token) diff --git a/goodprice/media/processing.py b/goodprice/media/processing.py index e766d90..65f37a6 100644 --- a/goodprice/media/processing.py +++ b/goodprice/media/processing.py @@ -64,7 +64,8 @@ def session(self, candidates, settings): with self.budget.reserve(candidate.key, download_size + output_size + 65536) as reservation: raw = self.downloader.download(candidate, download_size, settings.media_download_timeout, reservation=reservation) - source_hash = hashlib.sha256(raw.read_bytes()).hexdigest() + with raw.open("rb") as source: + source_hash = hashlib.file_digest(source, "sha256").hexdigest() parameters = {"version": 1, "kind": candidate.kind, "edge": 1280, "frames": settings.media_frames_per_video, "max_seconds": settings.media_video_max_seconds} @@ -84,7 +85,7 @@ def session(self, candidates, settings): settings.media_decode_timeout, settings.media_ffmpeg_threads, max_seconds=settings.media_video_max_seconds) outputs = [f.path for f in frames] - data = [{"file": str(f.path.relative_to(reservation.directory)), "timestamp": f.timestamp_ms, + data = [{"file": f.path.relative_to(reservation.directory).as_posix(), "timestamp": f.timestamp_ms, "hash": f.content_hash} for f in frames] raw.unlink(missing_ok=True) manifest = reservation.directory / f"{key}.json" diff --git a/goodprice/media/selection.py b/goodprice/media/selection.py index d812172..f8e7723 100644 --- a/goodprice/media/selection.py +++ b/goodprice/media/selection.py @@ -5,7 +5,16 @@ def select_media(images: list[MediaEvidence], frames: list[MediaEvidence], image_quota: int, frame_quota: int, total_limit: int) -> MediaSelection: if not (0 <= image_quota <= 16 and 0 <= frame_quota <= 16 and 1 <= total_limit <= 16): raise ValueError("Invalid media input budget") - groups = [images, frames] + seen = set() + groups = [] + for group in (images, frames): + unique = [] + for item in group: + identity = (item.kind, item.content_hash, item.offer_key) + if identity not in seen: + seen.add(identity) + unique.append(item) + groups.append(unique) quotas = [image_quota, frame_quota] positions = [0, 0] selected = [] diff --git a/goodprice/services/crawl_service.py b/goodprice/services/crawl_service.py index ddc9d03..4ca61ad 100644 --- a/goodprice/services/crawl_service.py +++ b/goodprice/services/crawl_service.py @@ -47,6 +47,7 @@ def __init__( settings_service, guard=None, seller_service=None, + media_service=None, ): self._session_factory = session_factory self.adapter = adapter @@ -56,6 +57,7 @@ def __init__( self.settings_service = settings_service self.guard = guard or TaskRunGuard() self.seller_service = seller_service + self.media_service = media_service self._judger = None def run_task(self, task_id: int) -> dict[str, Any]: @@ -132,8 +134,9 @@ def _fetch_detail(self, session, listing: Listing, force: bool = False) -> Listi except Exception as exc: logger.warning("详情抓取失败,退回标题判断: %s", exc) return - if detail.description: - listing.description = detail.description + listing.description = detail.description + if self.media_service: + self.media_service.record(listing.id, detail.media) merged = [] for url in detail.image_urls: if url not in merged and is_product_image(url): @@ -253,10 +256,21 @@ def _build_judger(self, settings): return judger def _condition_analysis(self, session, listing: Listing, task: WatchTask, - price: float | None = None) -> None: + price: float | None = None, offer=None, force=False) -> None: if not self.vision.enabled: return listing.condition_score = None + if self.media_service: + from goodprice.offers.types import OfferCandidate + + try: + verdict = self.media_service.analyze(listing.id, offer or OfferCandidate("default", {}), + self.vision, task.condition_requirement or "", self.settings_service.get(), force=force) + listing.condition_score = verdict.get("condition_score") + listing.condition_detail = verdict + except Exception: + listing.condition_detail = {"error": "媒体分析失败或超出预算,品相待核验"} + return valid = [u for u in (listing.image_urls or []) if is_product_image(u)] if not valid: listing.condition_detail = {"error": "无有效商品图"} diff --git a/goodprice/services/media_service.py b/goodprice/services/media_service.py new file mode 100644 index 0000000..820abbc --- /dev/null +++ b/goodprice/services/media_service.py @@ -0,0 +1,102 @@ +from datetime import datetime + +from goodprice.analysis.media_analysis import analysis_key, analyze_media +from goodprice.media.selection import select_media +from goodprice.media.types import MediaCandidate, MediaSelection +from goodprice.models import Listing, MediaAsset + + +class MediaService: + def __init__(self, factory, processor): + self.factory = factory + self.processor = processor + + def record(self, listing_id, candidates): + with self.factory() as session: + rows = session.query(MediaAsset).filter_by(listing_id=listing_id).all() + sources = {row.key: row for row in rows if row.kind != "frame" and row.source_url} + for row in sources.values(): + row.status = "superseded" + for candidate in candidates: + row = sources.get(candidate.key) + if row is None: + row = MediaAsset(listing_id=listing_id, key=candidate.key, kind=candidate.kind) + session.add(row) + row.source_url, row.source_locator = candidate.url, candidate.source_locator + row.offer_key, row.status = candidate.offer_key, candidate.status + row.observed_at = datetime.now() + session.commit() + + def _candidates(self, listing_id): + with self.factory() as session: + rows = session.query(MediaAsset).filter(MediaAsset.listing_id == listing_id, + MediaAsset.source_url != "", MediaAsset.status != "superseded").order_by(MediaAsset.id).all() + return [MediaCandidate(r.key, r.kind, r.source_url, r.offer_key, r.source_locator, + status="unsupported" if r.status == "unsupported" else "pending") for r in rows] + + def analyze(self, listing_id, offer, client, requirement, settings, *, force=False): + candidates = self._candidates(listing_id) + with self.factory() as session: + listing = session.get(Listing, listing_id) + seller_text = f"{listing.title}\n{listing.description or ''}" if listing else "" + with self.processor.session(candidates, settings) as (evidence, failures): + # A media explicitly bound to a different SKU is not this offer's evidence. + relevant = [e for e in evidence if e.offer_key is None or e.offer_key == offer.key] + selection = select_media([e for e in relevant if e.kind == "image"], + [e for e in relevant if e.kind == "frame"], settings.media_image_quota, + settings.media_frame_quota, settings.media_total_limit) + selection = MediaSelection(selection.items, selection.omitted_ids, + tuple(dict.fromkeys(selection.limitations + failures))) + # Price is excluded: this is a visual judgement, not a price verdict. + key = analysis_key(offer, selection, client, requirement, settings, seller_text) + with self.factory() as session: + listing = session.get(Listing, listing_id) + previous = listing.condition_detail or {} if listing else {} + for e in evidence: + row = session.query(MediaAsset).filter_by(listing_id=listing_id, + key=f"evidence:{e.id}").first() + if row is None: + row = MediaAsset(listing_id=listing_id, key=f"evidence:{e.id}", kind=e.kind) + session.add(row) + row.content_hash, row.offer_key = e.content_hash, e.offer_key + row.relative_path = e.local_path.relative_to(self.processor.budget.root).as_posix() + row.timestamp_ms, row.parent_video_key = e.timestamp_ms, e.video_id + row.status = "ready" + session.commit() + if not force and previous.get("media_cache_key") == key and selection.items: + return previous + result = analyze_media(offer, selection, client, requirement, settings.media_request_max_mib * 1024**2, seller_text) + result["media_cache_key"] = key + result["omitted_evidence_ids"] = list(selection.omitted_ids) + result["defects"] = [o["text"] for o in result.get("observations", [])] + return result + + def save_result(self, listing_id, result): + with self.factory() as session: + listing = session.get(Listing, listing_id) + if listing: + listing.condition_score = result.get("condition_score") + listing.condition_detail = result + session.commit() + + def public_assets(self, listing_id): + with self.factory() as session: + rows = session.query(MediaAsset).filter_by(listing_id=listing_id).order_by(MediaAsset.id).all() + result = [] + for row in rows: + if row.status == "superseded": + continue + available = False + if row.relative_path: + try: + path = self.processor.budget.checked(self.processor.budget.root / row.relative_path) + available = path.is_file() + except ValueError: + pass + viewable = available or (row.kind == "image" and bool(row.source_url) and row.status != "unsupported") + result.append({"id": row.id, "evidence_id": row.key.removeprefix("evidence:"), + "kind": row.kind, "offer_key": row.offer_key, "timestamp_ms": row.timestamp_ms, + "content_hash": row.content_hash, "status": row.status, + "available": available, "viewable": viewable, + "url": f"/media/{row.id}" if viewable else None}) + return result diff --git a/goodprice/services/offer_crawl.py b/goodprice/services/offer_crawl.py index 8beb813..37196dc 100644 --- a/goodprice/services/offer_crawl.py +++ b/goodprice/services/offer_crawl.py @@ -115,16 +115,21 @@ def run(self, task_id: int) -> dict: changed = True if needs_backfill: changed = False - if potential and (changed or needs_backfill): + if (potential or self.crawl.media_service) and (changed or needs_backfill or self.crawl.media_service): with self.factory() as session: listing = session.get(Listing, listing_id) - if self.crawl._requirement_pass(session, listing, task): + previous_media = (listing.condition_detail or {}).get("media_cache_key") + previous_score = listing.condition_score + if (not changed and not needs_backfill) or self.crawl._requirement_pass(session, listing, task): if len(candidates) == 1: - self.crawl._condition_analysis(session, listing, task, price=analysis_price) + self.crawl._condition_analysis(session, listing, task, price=analysis_price, offer=candidates[0]) else: listing.condition_score = None listing.condition_detail = {"error": "多规格共享媒体,品相待核验"} self.crawl._seller_check(session, listing, task) + if self.crawl.media_service and (previous_media != (listing.condition_detail or {}).get("media_cache_key") + or previous_score != listing.condition_score): + changed = True session.commit() ids = self.pipeline.evaluate_listing(listing_id, task_id, candidates, now) stats["reevaluated"] += int(not is_new and changed) @@ -269,9 +274,9 @@ def reanalyze(self, listing_id): potential = any(session.get(OfferEvaluationRecord, i).payload["state"] == "exact" for i in ids) analysis_price = next((session.get(OfferEvaluationRecord, i).payload["amount_cents"] / 100 for i in ids if session.get(OfferEvaluationRecord, i).payload["state"] == "exact"), None) - if potential and self.crawl._requirement_pass(session, listing, task): + if (potential or self.crawl.media_service) and self.crawl._requirement_pass(session, listing, task): if len(candidates) == 1: - self.crawl._condition_analysis(session, listing, task, price=analysis_price) + self.crawl._condition_analysis(session, listing, task, price=analysis_price, offer=candidates[0], force=True) else: listing.condition_score = None listing.condition_detail = {"error": "多规格共享媒体,品相待核验"} diff --git a/goodprice/web/media_routes.py b/goodprice/web/media_routes.py new file mode 100644 index 0000000..bf4127f --- /dev/null +++ b/goodprice/web/media_routes.py @@ -0,0 +1,128 @@ +import shutil + +from fastapi import APIRouter, HTTPException, Request +from fastapi.responses import FileResponse, RedirectResponse +from pydantic import ValidationError + +from goodprice.config import Settings +from goodprice.media.budget import MediaBudget +from goodprice.models import MediaAsset + +router = APIRouter() +MEDIA_FIELDS = {name for name in Settings.model_fields if name.startswith("media_")} | {"video_analysis_enabled"} +MEDIA_LABELS = dict(zip([ + "media_image_count", "media_video_count", "media_frames_per_video", "media_image_quota", + "media_frame_quota", "media_total_limit", "media_model_limit", "media_video_max_mib", + "media_video_max_seconds", "media_cache_mib", "media_retention_days", "media_download_timeout", + "media_decode_timeout", "media_request_max_mib", "media_ffmpeg_threads"], [ + "每商品原图保留数(1–32)", "每商品视频数(1–3)", "每视频采样帧数(1–24)", + "每请求原图配额(0–16)", "每请求帧配额(0–16)", "每请求总输入上限(1–16)", + "已确认模型上限(未知为8,最高16)", "视频大小上限(10–200 MiB)", + "视频时长上限(15–300秒)", "缓存容量(256–10240 MiB)", "保留天数(1–30)", + "下载超时(10–120秒)", "解码超时(10–120秒)", "请求体上限(1–32 MiB)", + "每进程解码线程(1–4)"], strict=True)) + + +def media_budget(settings): + return MediaBudget(settings.data_root / "media-cache", settings.media_cache_mib * 1024**2, + settings.media_retention_days * 86400) + + +def capabilities(settings): + return {"ffmpeg_available": bool(shutil.which("ffmpeg") and shutil.which("ffprobe")), + "video_enabled": settings.video_analysis_enabled, + "total_limit": settings.media_total_limit, "model_limit": settings.media_model_limit, + "image_quota": settings.media_image_quota, "frame_quota": settings.media_frame_quota, + "request_max_mib": settings.media_request_max_mib} + + +@router.get("/api/media/capabilities") +def media_capabilities(request: Request): + return capabilities(request.app.state.settings_service.get()) + + +class PinnedFileResponse(FileResponse): + def __init__(self, path, pin): + super().__init__(path, media_type="image/jpeg", headers={"X-Content-Type-Options": "nosniff"}) + self.pin = pin + + async def __call__(self, scope, receive, send): + try: + await super().__call__(scope, receive, send) + finally: + self.pin.__exit__(None, None, None) + + +def _pinned_response(budget, path): + with budget.state.lock: + path = budget.checked(path) + if not path.is_file() or path.suffix != ".jpg": + raise ValueError() + pin = budget.pin([path]) + pin.__enter__() + return PinnedFileResponse(path, pin) + + +@router.get("/media/{asset_id}") +def media_file(asset_id: int, request: Request): + settings = request.app.state.settings_service.get() + budget = media_budget(settings) + with request.app.state.session_factory() as session: + asset = session.get(MediaAsset, asset_id) + if asset is None or asset.kind not in {"image", "frame"} or asset.status == "superseded": + raise HTTPException(404, "证据文件不可用或已清理") + if asset.relative_path: + try: + return _pinned_response(budget, budget.root / asset.relative_path) + except ValueError: + raise HTTPException(404, "证据文件不可用或已清理") from None + # Viewing an image does not require vision to be enabled or configured. + # Source requests use the SAME pinned public downloader and disk budget. + if asset.kind == "image" and asset.source_url and asset.status != "unsupported": + from goodprice.media.processing import MediaProcessor + from goodprice.media.types import MediaCandidate + + processor = MediaProcessor(budget) + with processor.session([MediaCandidate(asset.key, "image", asset.source_url, asset.offer_key)], settings) as (evidence, _): + if evidence: + image = evidence[0] + response = _pinned_response(budget, image.local_path) + with request.app.state.session_factory() as session: + row = session.get(MediaAsset, asset_id) + if row: + row.relative_path = image.local_path.relative_to(budget.root).as_posix() + row.content_hash = image.content_hash + row.status = "ready" + session.commit() + return response + raise HTTPException(404, "证据文件不可用或已清理") + + +@router.post("/settings/media") +async def save_media(request: Request): + form = await request.form() + if set(form) - MEDIA_FIELDS - {"confirm_cost", "media_form"}: + raise HTTPException(422, "媒体分类含不可修改的字段") + values = {k: str(v) for k, v in form.items() if k in MEDIA_FIELDS} + if "media_form" in form: + values["video_analysis_enabled"] = "1" if "video_analysis_enabled" in form else "0" + service = request.app.state.settings_service + before = service.get() + try: + proposed = Settings(_env_file=None, **(before.model_dump() | values)) + increases = any(getattr(proposed, k) > getattr(before, k) for k in MEDIA_FIELDS + if k != "media_retention_days") + if increases and form.get("confirm_cost") != "1": + raise HTTPException(422, "增加媒体预算需确认费用、响应延迟、内存、存储与模型限制") + service.set_many(values) + except ValidationError: + raise HTTPException(422, "媒体设置无效:请检查范围、模型上限和缓存容量") from None + return RedirectResponse("/settings#media", status_code=303) + + +def public_assets(request, listing_id): + from goodprice.media.processing import MediaProcessor + from goodprice.services.media_service import MediaService + + return MediaService(request.app.state.session_factory, + MediaProcessor(media_budget(request.app.state.settings_service.get()))).public_assets(listing_id) diff --git a/goodprice/web/routes.py b/goodprice/web/routes.py index e0e5455..4bdcbaf 100644 --- a/goodprice/web/routes.py +++ b/goodprice/web/routes.py @@ -11,6 +11,7 @@ from sqlalchemy import and_, case, func, select, text from goodprice.constants import SCORE_MAX +from goodprice.web.media_routes import MEDIA_LABELS, capabilities, public_assets logger = logging.getLogger(__name__) @@ -577,6 +578,7 @@ def listing_detail_page(request: Request, listing_id: int): "drop_pct": drop_pct, "drop_pct_threshold": DROP_PCT_DISPLAY_THRESHOLD, "offers": OfferPipeline(request.app.state.session_factory).snapshot(listing_id, dt.now(timezone.utc)), + "media_assets": public_assets(request, listing_id), "active": "listings", }, ) @@ -687,6 +689,9 @@ def settings_page(request: Request): "settings.html", { "settings": settings, + "media_capabilities": capabilities(settings), + "media_fields": [name for name in type(settings).model_fields if name.startswith("media_")], + "media_labels": MEDIA_LABELS, "login_status": login_status, "login_message": login_message, **_login_context(login_session, settings.novnc_port), @@ -733,12 +738,21 @@ def settings_login_status(request: Request): @router.post("/settings") async def save_settings(request: Request): + from fastapi import HTTPException + from pydantic import ValidationError + from goodprice.config import SECRET_FIELDS, Settings + from goodprice.web.media_routes import MEDIA_FIELDS _, settings_service = _services(request) form = await request.form() + forbidden = MEDIA_FIELDS | {"database_url", "data_root", "app_name", "runtime_mode", "enable_novnc", "novnc_password_file", "novnc_port"} + if set(form) & forbidden: + raise HTTPException(422, "请使用媒体分类修改预算;部署参数仅可通过环境配置") values: dict[str, str] = {} for field in Settings.model_fields: + if field in forbidden: + continue if field.endswith("_enabled"): # 复选框未勾选时不会出现在表单里,统一记为“关闭”。 raw = form.get(field) @@ -750,7 +764,10 @@ async def save_settings(request: Request): for field in SECRET_FIELDS: if values.get(field) == "": values.pop(field) # 留空 = 保持原值 - runtime = settings_service.set_many(values) + try: + runtime = settings_service.set_many(values) + except ValidationError: + raise HTTPException(422, "设置值无效,未保存任何更改") from None from goodprice.services.satisfaction import backfill_satisfaction backfill_satisfaction(request.app.state.session_factory, vision_enabled=runtime.vision_enabled) diff --git a/goodprice/web/templates/listings_detail.html b/goodprice/web/templates/listings_detail.html index b240043..917f46f 100644 --- a/goodprice/web/templates/listings_detail.html +++ b/goodprice/web/templates/listings_detail.html @@ -26,13 +26,36 @@

{{ listing.title }}

- {% if listing.image_urls %} + {% if media_assets %} +

媒体证据

+

仅分析选中的画面;视频为有限采样,未验证音频或完整动态过程。未绑定规格的媒体为共享样品。

+
+ {% for asset in media_assets %} +
+ {% if asset.viewable %}商品证据 {{ asset.evidence_id }}{% endif %} +

{{ asset.kind }} {% if asset.timestamp_ms is not none %}{{ asset.timestamp_ms / 1000 }} 秒{% endif %} · {{ asset.offer_key or '共享媒体' }}

+

{{ asset.evidence_id }}

+ {% if not asset.available %}

{{ '证据文件已清理' if asset.content_hash else '未分析 / 不可用' }}

{% endif %} +
+ {% endfor %} +
+ {% if listing.condition_detail and listing.condition_detail.limitations %}

覆盖限制:{{ listing.condition_detail.limitations|join('、') }}

{% endif %} + {% elif listing.image_urls %}
{% for u in listing.image_urls %}
{% endfor %}
点击图片可放大查看
{% endif %} + {% if listing.condition_detail %} + {% for field, label in [('observations','画面观察'), ('seller_claims','卖家声明'), ('conflicts','信息冲突'), ('unverified','未验证项')] %} + {% if listing.condition_detail[field] %} +

{{ label }}

+ {% for item in listing.condition_detail[field] %}

{{ item.text }} 依据:{{ item.evidence_ids|join('、') or '无直接画面证据' }}

{% endfor %} + {% endif %} + {% endfor %} + {% endif %} +

规格与价格证据(实验性)

{% for offer in offers %}
diff --git a/goodprice/web/templates/media_settings.html b/goodprice/web/templates/media_settings.html new file mode 100644 index 0000000..a0c158b --- /dev/null +++ b/goodprice/web/templates/media_settings.html @@ -0,0 +1,20 @@ +
+

媒体证据与用量

+

每次评估只发一个视觉请求,不自动分批。增加图片或帧数通常会增加模型费用、响应延迟、内存与磁盘占用;具体金额以提供商账单为准。抽帧仅检查采样画面,不代表看过完整视频或音频。

+

当前每次最多 {{ settings.media_total_limit }} 张;原图配额 {{ settings.media_image_quota }},帧配额 {{ settings.media_frame_quota }}。混合时交替分配,单一类型借用剩余名额;配额为0即不分析该类。保存仅影响后续执行,不自动重新分析历史商品。

+ {% if not media_capabilities.ffmpeg_available %}

未检测到 FFmpeg/ffprobe,视频暂不可分析,图片与页面可正常使用。Docker 镜像内置工具;本地使用请安装 FFmpeg 并加入 PATH 后重启应用。

{% endif %} +
+ + +
+ {% for name, label in media_labels.items() %} + + {% endfor %} +
+

未知模型上限保守为8,确认提供商支持后才可提高至16。缓存包括下载、视频、保留原图和帧;磁盘不足会延后或跳过,不突破预算。视频默认100MiB/180秒,下载60秒/解码30秒,缓存1GiB/7天。

+ + + +
diff --git a/goodprice/web/templates/settings.html b/goodprice/web/templates/settings.html index 5642c0a..16c59f3 100644 --- a/goodprice/web/templates/settings.html +++ b/goodprice/web/templates/settings.html @@ -2,6 +2,7 @@ {% block title %}设置 - 闲鱼盯价助手{% endblock %} {% block content %}
+ {% include "media_settings.html" %}
diff --git a/pyproject.toml b/pyproject.toml index 184d58c..1fe934d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -37,6 +37,9 @@ good-price = "goodprice.main:main" [tool.setuptools.packages.find] include = ["goodprice*"] +[tool.setuptools.package-data] +"goodprice.web" = ["templates/*.html", "static/*"] + [tool.pytest.ini_options] testpaths = ["tests"] pythonpath = ["."] diff --git a/tests/test_media_analysis.py b/tests/test_media_analysis.py new file mode 100644 index 0000000..5e1c712 --- /dev/null +++ b/tests/test_media_analysis.py @@ -0,0 +1,62 @@ +import base64 +import json + +import httpx +import pytest + +from goodprice.analysis.llm import LLMClient +from goodprice.analysis.media_analysis import analyze_media, validate_evidence_references +from goodprice.media.types import MediaEvidence, MediaSelection +from goodprice.offers.types import OfferCandidate + + +@pytest.mark.parametrize("api_format", ["chat_completions", "responses"]) +def test_data_urls_and_evidence_ids_in_both_api_formats(tmp_path, api_format): + from test_media_processing import image_bytes + path = tmp_path / "private-host-path.jpg" + path.write_bytes(image_bytes()) + evidence = MediaEvidence("e1", "frame", "hash", path, "sku:one", "v1", 1500) + verdict = {"condition_score": 8, "observations": [{"text": "轻微划痕", "evidence_ids": ["e1"]}], + "seller_claims": [], "conflicts": [], "unverified": []} + calls = [] + def handler(request): + data = json.loads(request.content) + calls.append(data) + content = data["input"][0]["content"] if api_format == "responses" else data["messages"][1]["content"] + image = content[1]["image_url"] + url = image if isinstance(image, str) else image["url"] + assert base64.b64decode(url.split(",", 1)[1]) == path.read_bytes() + assert "e1" in content[0]["text"] and "1500" in content[0]["text"] + assert str(tmp_path) not in request.content.decode() + if api_format == "responses": + return httpx.Response(200, text="data: " + json.dumps({"type": "response.output_text.delta", "delta": json.dumps(verdict)}) + "\n\n") + return httpx.Response(200, json={"choices": [{"message": {"content": json.dumps(verdict)}}]}) + client = LLMClient("https://model.example/v1", "fake", "vision", transport=httpx.MockTransport(handler), api_format=api_format) + result = analyze_media(OfferCandidate("sku:one", {}), MediaSelection((evidence,)), client, "屏幕完好", 16 * 1024**2) + assert result["condition_score"] == 8 and len(calls) == 1 + + +def test_hallucinated_reference_and_shared_sku_score_rejected(tmp_path): + with pytest.raises(ValueError): + validate_evidence_references({"observations": [{"evidence_ids": ["invented"]}]}, {"real"}) + path = tmp_path / "image.jpg" + path.write_bytes(b"jpg") + class Client: + def payload(self, content, system): return {"messages": content} + def complete(self, payload, parser): + return parser(json.dumps({"condition_score": 9, "observations": [{"text": "样品", "evidence_ids": ["one"]}]})) + result = analyze_media(OfferCandidate("sku:one", {}), MediaSelection((MediaEvidence("one", "image", "h", path),)), Client(), "", 10000) + assert result["condition_score"] is None + assert "shared_media" in result["limitations"] + + +def test_empty_or_oversized_input_does_not_send(tmp_path): + class Client: + def payload(self, content, system): return {"messages": content} + def complete(self, *args, **kwargs): raise AssertionError("model request sent") + offer = OfferCandidate("default", {}) + assert analyze_media(offer, MediaSelection(()), Client(), "", 100)["condition_score"] is None + path = tmp_path / "image.jpg" + path.write_bytes(b"x" * 1000) + with pytest.raises(ValueError, match="预算"): + analyze_media(offer, MediaSelection((MediaEvidence("one", "image", "h", path),)), Client(), "", 100) diff --git a/tests/test_media_api.py b/tests/test_media_api.py new file mode 100644 index 0000000..47dc5fc --- /dev/null +++ b/tests/test_media_api.py @@ -0,0 +1,52 @@ +from fastapi.testclient import TestClient + +from goodprice.main import build_app +from goodprice.models import Listing, MediaAsset + + +def test_media_api_confined_to_ids_and_cache_root(session_factory, base_settings, tmp_path): + app = build_app(base_settings, session_factory, with_scheduler=False) + root = base_settings.data_root / "media-cache" + root.mkdir(parents=True, exist_ok=True) + image = root / "sample.jpg" + image.write_bytes(b"jpeg") + outside = tmp_path / "secret.jpg" + outside.write_bytes(b"secret") + with session_factory() as session: + row = Listing(platform="xianyu", external_id="one", title="x", url="x", price=1) + session.add(row) + session.flush() + valid = MediaAsset(listing_id=row.id, key="one", kind="image", relative_path="sample.jpg", status="ready") + invalid = MediaAsset(listing_id=row.id, key="two", kind="image", relative_path=str(outside), status="ready") + session.add_all([valid, invalid]) + session.commit() + client = TestClient(app) + assert client.get(f"/media/{valid.id}").content == b"jpeg" + assert client.get(f"/media/{invalid.id}").status_code == 404 + assert client.get("/media/9999").status_code == 404 + image.unlink() + assert client.get(f"/media/{valid.id}").status_code == 404 + capabilities = client.get("/api/media/capabilities") + assert capabilities.status_code == 200 + assert str(base_settings.data_root) not in capabilities.text + + +def test_increasing_inputs_requires_confirmation_and_keeps_other_settings(session_factory, base_settings): + app = build_app(base_settings, session_factory, with_scheduler=False) + client = TestClient(app) + service = app.state.settings_service + service.set_many({"media_total_limit": "4", "feishu_enabled": "1"}) + assert client.post("/settings/media", data={"media_total_limit": "6"}).status_code == 422 + assert service.get().media_total_limit == 4 + assert client.post("/settings/media", data={"media_total_limit": "6", "confirm_cost": "1"}, follow_redirects=False).status_code == 303 + assert service.get().media_total_limit == 6 + assert service.get().feishu_enabled is True + assert client.post("/settings/media", data={"database_url": "sqlite:///evil"}).status_code == 422 + assert "费用" in client.get("/settings").text + + +def test_general_settings_cannot_override_media_or_deployment(session_factory, base_settings): + app = build_app(base_settings, session_factory, with_scheduler=False) + client = TestClient(app) + for field, value in [("media_total_limit", "16"), ("data_root", "outside"), ("database_url", "sqlite:///outside")]: + assert client.post("/settings", data={field: value}).status_code == 422 diff --git a/tests/test_media_regressions.py b/tests/test_media_regressions.py new file mode 100644 index 0000000..4b2e90b --- /dev/null +++ b/tests/test_media_regressions.py @@ -0,0 +1,131 @@ +import logging +from dataclasses import replace + +import httpx +from fastapi.testclient import TestClient + +from goodprice.main import build_app +from goodprice.media.budget import MediaBudget +from goodprice.media.download import MediaDownloader +from goodprice.media.types import MediaCandidate +from goodprice.models import Listing, MediaAsset +from goodprice.services.task_service import TaskService + + +def test_signed_download_does_not_log_credentials(tmp_path, caplog): + caplog.set_level(logging.INFO) + budget = MediaBudget(tmp_path, 1024, 60, safety_bytes=0) + downloader = MediaDownloader(budget, httpx.MockTransport(lambda r: httpx.Response(200, content=b"file")), lambda h,p: ["93.184.216.34"]) + downloader.download(MediaCandidate("one", "image", "https://cdn.example/a?token=offline-secret"), 10, 10) + assert "offline-secret" not in caplog.text + + +def test_failed_range_releases_cache_pin(session_factory, base_settings): + import asyncio + + from starlette.requests import Request + + from goodprice.web.media_routes import media_file + app = build_app(base_settings, session_factory, with_scheduler=False) + root = base_settings.data_root / "media-cache" + budget = MediaBudget(root, 1024, 60, safety_bytes=0) + path = root / "a.jpg" + path.write_bytes(b"jpeg") + with session_factory() as session: + listing = Listing(platform="x", external_id="one", title="x", url="x", price=1) + session.add(listing) + session.flush() + asset = MediaAsset(listing_id=listing.id, key="one", kind="image", relative_path="a.jpg", status="ready") + session.add(asset) + session.commit() + scope = {"type": "http", "method": "GET", "headers": [(b"range", b"bytes=999999-")], "app": app} + response = media_file(asset.id, Request(scope)) + events = [] + async def send(event): events.append(event) + async def receive(): return {"type": "http.request"} + asyncio.run(response(scope, receive, send)) + assert events[0]["status"] == 416 + assert budget.state.pins.get(path.resolve(), 0) == 0 + + +class Media: + score = 8 + failure = None + calls = 0 + force = False + + def record(self, *args): pass + def analyze(self, *args, **kwargs): + self.calls += 1 + self.force = kwargs.get("force", False) + if self.failure: + raise self.failure + return {"condition_score": self.score, "media_cache_key": str(self.score)} + + +def test_provider_timeout_is_unverified_and_following_listing_processed(session_factory, base_settings): + from test_crawl_service import FakeAdapter, _item, _service + task = TaskService(session_factory).create_task({"keyword": "x"}) + crawl, _, _ = _service(session_factory, base_settings, adapter=FakeAdapter([_item("one"), _item("two")])) + media = Media() + media.failure = httpx.ReadTimeout("private provider details") + crawl.media_service = media + crawl.run_task(task.id) + with session_factory() as session: + rows = session.query(Listing).all() + assert len(rows) == 2 + assert all(row.condition_score is None for row in rows) + + +def test_same_url_new_media_score_invalidates_ranking_and_manual_reanalysis_forces(session_factory, base_settings): + from test_crawl_service import FakeAdapter, _item, _service + task = TaskService(session_factory).create_task({"keyword": "x"}) + crawl, _, _ = _service(session_factory, base_settings, adapter=FakeAdapter([_item()])) + crawl.media_service = media = Media() + crawl.run_task(task.id) + with session_factory() as session: + row = session.query(Listing).one() + old = row.satisfaction + listing_id = row.id + media.score = 2 + crawl.run_task(task.id) + with session_factory() as session: + assert session.get(Listing, listing_id).satisfaction < old + crawl.reanalyze_listing(listing_id) + assert media.force + + +def test_pending_price_still_gets_independent_media_analysis(session_factory, base_settings): + from test_crawl_service import FakeAdapter, _item, _service + class Inquiry(FakeAdapter): + def fetch_detail(self, url): + from datetime import datetime, timezone + + from goodprice.offers.types import OfferCandidate, PriceClaim + return replace(super().fetch_detail(url), offers=[OfferCandidate("default", {}, claims=( + PriceClaim("description", "inquiry", "价格请咨询", datetime.now(timezone.utc), "description"),))]) + task = TaskService(session_factory).create_task({"keyword": "x"}) + crawl, _, _ = _service(session_factory, base_settings, adapter=Inquiry([_item()])) + crawl.media_service = media = Media() + crawl.run_task(task.id) + assert media.calls == 1 + with session_factory() as session: + assert session.query(Listing).one().value_score is None + + +def test_scoreless_evidence_and_pending_image_gallery_visible(session_factory, base_settings): + app = build_app(base_settings, session_factory, with_scheduler=False) + with session_factory() as session: + row = Listing(platform="xianyu", external_id="one", title="x", price=1, url="x", + image_urls=["https://cdn.example/a?token=private"], + condition_detail={"observations": [{"text": "屏幕有划痕", "evidence_ids": ["frame1"]}], + "conflicts": [{"text": "描述与画面冲突", "evidence_ids": ["frame1"]}]}) + session.add(row) + session.flush() + asset = MediaAsset(listing_id=row.id, key="source", kind="image", source_url=row.image_urls[0], status="pending") + session.add(asset) + session.commit() + html = TestClient(app).get(f"/listings/{row.id}").text + assert f'src="/media/{asset.id}"' in html + assert "token=private" not in html + assert "屏幕有划痕" in html and "描述与画面冲突" in html and "frame1" in html diff --git a/tests/test_media_selection.py b/tests/test_media_selection.py index c64e2b4..b77435f 100644 --- a/tests/test_media_selection.py +++ b/tests/test_media_selection.py @@ -18,6 +18,11 @@ def test_default_mixed_selection_is_four_images_four_frames(): assert len(result.omitted_ids) == 6 +def test_duplicate_content_does_not_consume_paid_input_slots(): + images = [MediaEvidence(str(i), "image", "same-hash", Path(f"{i}.jpg")) for i in range(8)] + assert len(select_media(images, [], 4, 4, 8).items) == 1 + + def test_single_type_borrows_unused_quota(): result = select_media(evidence("image", 10), [], 4, 4, 8) assert len(result.items) == 8 diff --git a/tests/test_media_service.py b/tests/test_media_service.py new file mode 100644 index 0000000..c3f14e0 --- /dev/null +++ b/tests/test_media_service.py @@ -0,0 +1,49 @@ +import json + +import httpx + +from goodprice.analysis.llm import LLMClient +from goodprice.media.budget import MediaBudget +from goodprice.media.download import MediaDownloader +from goodprice.media.processing import MediaProcessor +from goodprice.models import Listing +from goodprice.offers.types import OfferCandidate +from goodprice.services.media_service import MediaService + + +def test_stored_media_cache_and_unknown_score_after_removed_media(session_factory, base_settings, tmp_path): + from test_media_processing import image_bytes + + from goodprice.media.types import MediaCandidate + budget = MediaBudget(tmp_path / "cache", 256 * 1024**2, 86400, safety_bytes=0) + processor = MediaProcessor(budget, MediaDownloader(budget, httpx.MockTransport(lambda r: httpx.Response(200, content=image_bytes())), lambda h,p: ["93.184.216.34"])) + service = MediaService(session_factory, processor) + with session_factory() as session: + listing = Listing(platform="xianyu", external_id="a", title="x", price=1, url="https://item.example/a") + session.add(listing) + session.commit() + service.record(listing.id, [MediaCandidate("one", "image", "https://cdn.example/a?sig=secret")]) + calls = [] + def handler(request): + data = json.loads(request.content) + descriptors = json.loads(data["messages"][1]["content"][0]["text"])["evidence"] + calls.append(1) + result = {"condition_score": 8, "observations": [{"text": "完好", "evidence_ids": [descriptors[0]["id"]]}]} + return httpx.Response(200, json={"choices": [{"message": {"content": json.dumps(result)}}]}) + client = LLMClient("https://model.example/v1", "fake", "vision", transport=httpx.MockTransport(handler)) + offer = OfferCandidate("default", {}) + first = service.analyze(listing.id, offer, client, "", base_settings) + service.save_result(listing.id, first) + second = service.analyze(listing.id, offer, client, "", base_settings) + assert first["condition_score"] == second["condition_score"] == 8 + assert len(calls) == 1 + service.analyze(listing.id, offer, client, "", base_settings, force=True) + assert len(calls) == 2 + with session_factory() as session: + session.get(Listing, listing.id).description = "屏幕已碎裂" + session.commit() + service.analyze(listing.id, offer, client, "", base_settings) + assert len(calls) == 3 + service.record(listing.id, []) + assert service.analyze(listing.id, offer, client, "", base_settings)["condition_score"] is None + assert "secret" not in json.dumps(service.public_assets(listing.id)) From 31e5d9fb16d5dfbc8580b879c2de865d0e53481f Mon Sep 17 00:00:00 2001 From: Comui520 Date: Mon, 26 Oct 2026 12:26:38 +0800 Subject: [PATCH 12/17] feat: persist versioned task work and cancellation state --- goodprice/execution/__init__.py | 1 + goodprice/execution/store.py | 217 +++++++++++++++++++++++++++++ goodprice/execution/types.py | 61 ++++++++ goodprice/migrations.py | 7 +- goodprice/models.py | 58 ++++++++ goodprice/services/task_service.py | 6 + tests/test_execution_store.py | 98 +++++++++++++ 7 files changed, 447 insertions(+), 1 deletion(-) create mode 100644 goodprice/execution/__init__.py create mode 100644 goodprice/execution/store.py create mode 100644 goodprice/execution/types.py create mode 100644 tests/test_execution_store.py diff --git a/goodprice/execution/__init__.py b/goodprice/execution/__init__.py new file mode 100644 index 0000000..e1d0a4d --- /dev/null +++ b/goodprice/execution/__init__.py @@ -0,0 +1 @@ +"""Persistent bounded execution; no background work on import.""" diff --git a/goodprice/execution/store.py b/goodprice/execution/store.py new file mode 100644 index 0000000..a8d923c --- /dev/null +++ b/goodprice/execution/store.py @@ -0,0 +1,217 @@ +import threading +import uuid +from datetime import datetime, timedelta, timezone + +from goodprice.execution.types import RunToken, Stage, WorkItemDTO, freeze, thaw +from goodprice.models import NotificationDelivery, NotificationEvent, TaskRun, WatchTask, WorkItem + +_locks = {} +_locks_guard = threading.Lock() +OPEN_WORK = ("pending", "running", "retry") +OPEN_RUN = ("queued", "active") + + +def naive(value): + return value.astimezone(timezone.utc).replace(tzinfo=None) if value.tzinfo else value + + +def invalidate_in_session(session, task_id): + session.query(TaskRun).filter(TaskRun.task_id == task_id, TaskRun.state.in_(OPEN_RUN)).update( + {"state": "cancelled", "followup": False}) + session.query(WorkItem).filter(WorkItem.task_id == task_id, WorkItem.state.in_(OPEN_WORK)).update({"state": "cancelled"}) + events = session.query(NotificationEvent.id).filter_by(task_id=task_id) + session.query(NotificationDelivery).filter(NotificationDelivery.event_id.in_(events), + NotificationDelivery.state.in_(("pending", "failed", "awaiting_revalidation"))).update({"state": "cancelled"}, synchronize_session=False) + + +class ExecutionStore: + def __init__(self, session_factory, global_limit=200, task_limit=60, active_limit=2, + lease_seconds=120, run_minutes=15): + self.factory = session_factory + self.global_limit, self.task_limit, self.active_limit = global_limit, task_limit, active_limit + self.lease_seconds, self.run_minutes = lease_seconds, run_minutes + with _locks_guard: + self.lock = _locks.setdefault(str(session_factory.kw["bind"].url), threading.RLock()) + + def _room(self, session, task_id): + query = session.query(WorkItem).filter(WorkItem.state.in_(OPEN_WORK)) + return query.count() < self.global_limit and query.filter_by(task_id=task_id).count() < self.task_limit + + def _submit(self, session, task, kind, target_id): + if not self._room(session, task.id): + return None + run = TaskRun(task_id=task.id, task_version=task.version, kind=kind, target_id=target_id) + previous = session.query(TaskRun).filter_by(task_id=task.id, task_version=task.version, + state="deferred", kind=kind, target_id=target_id).order_by(TaskRun.id.desc()).first() + if previous: + run.cursor = {**previous.cursor, "resume_run_id": previous.id} + session.add(run) + session.flush() + session.add(WorkItem(run_id=run.id, task_id=task.id, stage=Stage.SEARCH.value, + payload={"target_id": target_id} if target_id else {})) + return run.id + + def submit(self, task_id: int, kind: str = "crawl", target_id=None): + with self.lock, self.factory() as session: + task = session.get(WatchTask, task_id) + if task is None or not task.enabled: + return None + existing = session.query(TaskRun).filter(TaskRun.task_id == task_id, TaskRun.state.in_(OPEN_RUN)).first() + if existing and existing.task_version == task.version: + if existing.state == "active": + existing.followup = True + session.commit() + return existing.id + if existing: + invalidate_in_session(session, task_id) + result = self._submit(session, task, kind, target_id) + session.commit() + return result + + def enqueue(self, run_id, stage, fingerprint, payload): + with self.lock, self.factory() as session: + run = session.get(TaskRun, run_id) + if run is None or run.state not in OPEN_RUN or not self._room(session, run.task_id): + return False + if session.query(WorkItem).filter_by(run_id=run_id, stage=stage.value, input_fingerprint=fingerprint).first(): + return True + session.add(WorkItem(run_id=run_id, task_id=run.task_id, stage=stage.value, + input_fingerprint=fingerprint, payload=thaw(payload))) + session.commit() + return True + + def set_cursor(self, run_id, cursor): + with self.lock, self.factory() as session: + run = session.get(TaskRun, run_id) + if run is None or run.state not in OPEN_RUN: + return False + run.cursor = thaw(cursor) + session.commit() + return True + + def _valid(self, session, run, now): + task = session.get(WatchTask, run.task_id) + return (task is not None and task.enabled and task.version == run.task_version + and run.state in OPEN_RUN and (run.deadline_at is None or run.deadline_at > now)) + + def claim(self, stage: Stage, now: datetime): + now = naive(now) + with self.lock, self.factory() as session: + active = session.query(TaskRun).filter_by(state="active").count() + rows = session.query(WorkItem).join(TaskRun, TaskRun.id == WorkItem.run_id).filter( + WorkItem.stage == stage.value, WorkItem.state.in_(("pending", "retry")), + TaskRun.state.in_(OPEN_RUN)).order_by(WorkItem.attempt, WorkItem.id).all() + for item in rows: + run = session.get(TaskRun, item.run_id) + if not self._valid(session, run, now): + invalidate_in_session(session, run.task_id) + continue + if item.not_before and item.not_before > now: + continue + if run.state == "queued": + if active >= self.active_limit: + continue + run.state, run.started_at = "active", now + run.deadline_at = now + timedelta(minutes=self.run_minutes) + item.state, item.lease_id = "running", uuid.uuid4().hex + item.lease_until = min(run.deadline_at, now + timedelta(seconds=self.lease_seconds)) + item.attempt += 1 + session.commit() + return WorkItemDTO(item.id, RunToken(run.id, run.task_id, run.task_version, item.lease_id), + stage, item.input_fingerprint, freeze(item.payload), item.attempt, + run.deadline_at.replace(tzinfo=timezone.utc)) + session.commit() + return None + + def complete(self, result, now: datetime): + now = naive(now) + with self.lock, self.factory() as session: + run = session.get(TaskRun, result.token.run_id) + if run is None or not self._valid(session, run, now) or run.task_version != result.token.task_version: + return False + item = session.query(WorkItem).filter_by(run_id=run.id, lease_id=result.token.lease_id, + state="running", stage=result.stage.value, input_fingerprint=result.input_fingerprint).first() + if item is None or item.lease_until <= now: + return False + item.state, item.output, item.lease_id = "done", thaw(result.output), None + session.commit() + return True + + def finish_run(self, run_id, now): + with self.lock, self.factory() as session: + run = session.get(TaskRun, run_id) + if run is None or not self._valid(session, run, naive(now)): + return False + if session.query(WorkItem).filter(WorkItem.run_id == run_id, WorkItem.state.in_(OPEN_WORK)).count(): + return False + if any((item.output or {}).get("next") for item in session.query(WorkItem).filter_by(run_id=run_id, state="done")): + return False + run.state = "completed" + followup = run.followup + run.followup = False + session.flush() + if followup: + self._submit(session, session.get(WatchTask, run.task_id), run.kind, run.target_id) + session.commit() + return True + + def recover(self, now): + now = naive(now) + count = 0 + with self.lock, self.factory() as session: + for item in session.query(WorkItem).filter_by(state="running"): + if item.lease_until and item.lease_until <= now: + item.state, item.lease_id = "pending", None + count += 1 + for run in session.query(TaskRun).filter(TaskRun.state.in_(OPEN_RUN)): + task = session.get(WatchTask, run.task_id) + if task is None or not task.enabled or task.version != run.task_version: + invalidate_in_session(session, run.task_id) + elif run.deadline_at and run.deadline_at <= now: + run.state = "deferred" + session.query(WorkItem).filter(WorkItem.run_id == run.id, + WorkItem.state.in_(OPEN_WORK)).update({"state": "deferred", "lease_id": None}) + session.query(NotificationDelivery).filter(NotificationDelivery.state == "sending", + NotificationDelivery.lease_until <= now).update({"state": "uncertain"}) + session.commit() + return count + + def invalidate_task(self, task_id): + with self.lock, self.factory() as session: + invalidate_in_session(session, task_id) + session.commit() + + def materialize(self, now): + """Durably drain bounded fan-out; unexpanded outputs survive a restart.""" + with self.lock, self.factory() as session: + rows = session.query(WorkItem).filter_by(state="done").order_by(WorkItem.id).all() + for item in rows: + pending = list((item.output or {}).get("next", [])) + if not pending: + continue + run = session.get(TaskRun, item.run_id) + if not self._valid(session, run, naive(now)): + continue + expanded = 0 + for node in pending: + if not self._room(session, run.task_id): + break + stage = Stage(node["stage"]) + fingerprint = str(node.get("fingerprint", ""))[:64] + if not session.query(WorkItem).filter_by(run_id=run.id, stage=stage.value, input_fingerprint=fingerprint).first(): + session.add(WorkItem(run_id=run.id, task_id=run.task_id, stage=stage.value, + input_fingerprint=fingerprint, payload=thaw(node.get("payload", {})))) + session.flush() + expanded += 1 + item.output = {**item.output, "next": pending[expanded:]} + session.commit() + + def fail(self, item, now): + with self.lock, self.factory() as session: + row = session.get(WorkItem, item.id) + if row is None or row.state != "running" or row.lease_id != item.token.lease_id: + return + row.state = "retry" if row.attempt < 3 else "failed" + row.lease_id = None + row.not_before = naive(now) + timedelta(seconds=2 ** row.attempt) + session.commit() diff --git a/goodprice/execution/types.py b/goodprice/execution/types.py new file mode 100644 index 0000000..438c2a3 --- /dev/null +++ b/goodprice/execution/types.py @@ -0,0 +1,61 @@ +from dataclasses import dataclass +from datetime import datetime +from enum import Enum +from types import MappingProxyType +from typing import Mapping + + +class Stage(str, Enum): + SEARCH = "search" + DETAIL = "detail" + PARSE = "parse" + MEDIA = "media" + ANALYZE = "analyze" + AGGREGATE = "aggregate" + NOTIFY = "notify" + + +def freeze(value): + if isinstance(value, dict): + return MappingProxyType({k: freeze(v) for k, v in value.items()}) + if isinstance(value, (tuple, list)): + return tuple(freeze(v) for v in value) + return value + + +def thaw(value): + if isinstance(value, Mapping): + return {k: thaw(v) for k, v in value.items()} + if isinstance(value, (tuple, list)): + return [thaw(v) for v in value] + return value + + +@dataclass(frozen=True) +class RunToken: + run_id: int + task_id: int + task_version: int + lease_id: str + + +@dataclass(frozen=True) +class WorkItemDTO: + id: int + token: RunToken + stage: Stage + input_fingerprint: str + payload: Mapping + attempt: int + deadline_at: datetime + + +@dataclass(frozen=True) +class WorkResult: + token: RunToken + stage: Stage + input_fingerprint: str + output: Mapping + + def __post_init__(self): + object.__setattr__(self, "output", freeze(self.output)) diff --git a/goodprice/migrations.py b/goodprice/migrations.py index ebaae69..23cdff8 100644 --- a/goodprice/migrations.py +++ b/goodprice/migrations.py @@ -5,9 +5,13 @@ ListingOffer, ManualQuote, MediaAsset, + NotificationDelivery, + NotificationEvent, OfferEvaluationRecord, OfferObservation, SchemaVersion, + TaskRun, + WorkItem, ) @@ -15,7 +19,8 @@ def migrate_offer_schema(session) -> None: connection = session.connection() tables = set(inspect(connection).get_table_names()) legacy = "schema_versions" not in tables and "watch_tasks" in tables - for model in (SchemaVersion, ListingOffer, OfferObservation, ManualQuote, OfferEvaluationRecord, MediaAsset): + for model in (SchemaVersion, ListingOffer, OfferObservation, ManualQuote, OfferEvaluationRecord, MediaAsset, + TaskRun, WorkItem, NotificationEvent, NotificationDelivery): model.__table__.create(connection, checkfirst=True) if session.get(SchemaVersion, 1) is None: session.add(SchemaVersion(id=1, version=1, legacy_install=legacy)) diff --git a/goodprice/models.py b/goodprice/models.py index 0d2e1f3..09099ed 100644 --- a/goodprice/models.py +++ b/goodprice/models.py @@ -226,3 +226,61 @@ class MediaAsset(Base): status: Mapped[str] = mapped_column(String(30), default="pending") failure: Mapped[Optional[str]] = mapped_column(Text, nullable=True) observed_at: Mapped[datetime] = mapped_column(DateTime, default=_now) + + +class TaskRun(Base): + __tablename__ = "task_runs" + id: Mapped[int] = mapped_column(primary_key=True) + task_id: Mapped[int] = mapped_column(Integer, index=True) + task_version: Mapped[int] = mapped_column(Integer) + kind: Mapped[str] = mapped_column(String(30), default="crawl") + target_id: Mapped[Optional[int]] = mapped_column(Integer, nullable=True) + state: Mapped[str] = mapped_column(String(30), default="queued", index=True) + followup: Mapped[bool] = mapped_column(default=False) + cursor: Mapped[dict] = mapped_column(JSON, default=dict) + created_at: Mapped[datetime] = mapped_column(DateTime, default=_now) + started_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True) + deadline_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True) + + +class WorkItem(Base): + __tablename__ = "work_items" + __table_args__ = (UniqueConstraint("run_id", "stage", "input_fingerprint", name="uq_work_input"),) + id: Mapped[int] = mapped_column(primary_key=True) + run_id: Mapped[int] = mapped_column(ForeignKey("task_runs.id"), index=True) + task_id: Mapped[int] = mapped_column(Integer, index=True) + stage: Mapped[str] = mapped_column(String(20), index=True) + input_fingerprint: Mapped[str] = mapped_column(String(64), default="") + payload: Mapped[dict] = mapped_column(JSON, default=dict) + output: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True) + state: Mapped[str] = mapped_column(String(30), default="pending", index=True) + attempt: Mapped[int] = mapped_column(Integer, default=0) + lease_id: Mapped[Optional[str]] = mapped_column(String(64), nullable=True, index=True) + lease_until: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True) + not_before: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True) + + +class NotificationEvent(Base): + __tablename__ = "notification_events" + id: Mapped[int] = mapped_column(primary_key=True) + business_key: Mapped[str] = mapped_column(String(128), unique=True) + task_id: Mapped[int] = mapped_column(Integer, index=True) + task_version: Mapped[int] = mapped_column(Integer) + offer_id: Mapped[int] = mapped_column(Integer) + listing_id: Mapped[int] = mapped_column(Integer) + payload: Mapped[dict] = mapped_column(JSON, default=dict) + state: Mapped[str] = mapped_column(String(30), default="pending") + created_at: Mapped[datetime] = mapped_column(DateTime, default=_now) + + +class NotificationDelivery(Base): + __tablename__ = "notification_deliveries" + __table_args__ = (UniqueConstraint("event_id", "channel", name="uq_delivery_channel"),) + id: Mapped[int] = mapped_column(primary_key=True) + event_id: Mapped[int] = mapped_column(ForeignKey("notification_events.id"), index=True) + channel: Mapped[str] = mapped_column(String(50)) + state: Mapped[str] = mapped_column(String(30), default="pending", index=True) + attempt: Mapped[int] = mapped_column(Integer, default=0) + lease_id: Mapped[Optional[str]] = mapped_column(String(64), nullable=True) + lease_until: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True) + not_before: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True) diff --git a/goodprice/services/task_service.py b/goodprice/services/task_service.py index f7e302d..9133264 100644 --- a/goodprice/services/task_service.py +++ b/goodprice/services/task_service.py @@ -108,20 +108,24 @@ def create_task(self, data: dict) -> WatchTask: return task def toggle_task(self, task_id: int) -> Optional[WatchTask]: + from goodprice.execution.store import invalidate_in_session with self._session_factory() as session: task = session.get(WatchTask, task_id) if task: task.enabled = not task.enabled task.version += 1 + invalidate_in_session(session, task_id) session.commit() session.refresh(task) return task def delete_task(self, task_id: int) -> bool: + from goodprice.execution.store import invalidate_in_session with self._session_factory() as session: task = session.get(WatchTask, task_id) if not task: return False + invalidate_in_session(session, task_id) session.query(Listing).filter_by(task_id=task_id).update({"task_id": None}) session.query(Notification).filter_by(task_id=task_id).update({"task_id": None}) session.delete(task) @@ -133,6 +137,7 @@ def enabled_tasks(self) -> list[WatchTask]: return session.query(WatchTask).filter(WatchTask.enabled.is_(True)).all() def update_task(self, task_id: int, data: dict) -> Optional[WatchTask]: + from goodprice.execution.store import invalidate_in_session with self._session_factory() as session: task = session.get(WatchTask, task_id) if not task: @@ -143,6 +148,7 @@ def update_task(self, task_id: int, data: dict) -> Optional[WatchTask]: for field in TASK_FIELDS: if getattr(task, field) != values[field]: task.version += 1 + invalidate_in_session(session, task_id) break for field in TASK_FIELDS: setattr(task, field, values[field]) diff --git a/tests/test_execution_store.py b/tests/test_execution_store.py new file mode 100644 index 0000000..e832a06 --- /dev/null +++ b/tests/test_execution_store.py @@ -0,0 +1,98 @@ +from concurrent.futures import ThreadPoolExecutor +from datetime import datetime, timedelta, timezone + +from goodprice.execution.store import ExecutionStore +from goodprice.execution.types import Stage, WorkResult +from goodprice.models import TaskRun, WorkItem +from goodprice.services.task_service import TaskService + + +def task(factory): + return TaskService(factory).create_task({"keyword": "controlled fixture"}) + + +def test_submit_deduplicates_and_running_signals_merge_once(session_factory): + row = task(session_factory) + store = ExecutionStore(session_factory) + first = store.submit(row.id, "crawl") + assert first and store.submit(row.id, "crawl") == first + now = datetime.now(timezone.utc) + claimed = store.claim(Stage.SEARCH, now) + for _ in range(5): + assert store.submit(row.id, "crawl") == first + assert store.complete(WorkResult(claimed.token, Stage.SEARCH, "", {}), now) + assert store.finish_run(first, now) + with session_factory() as session: + assert session.query(TaskRun).count() == 2 + assert session.query(TaskRun).filter_by(state="queued").count() == 1 + + +def test_claim_is_exclusive_and_recovery_invalidates_old_lease(session_factory): + row = task(session_factory) + store = ExecutionStore(session_factory, lease_seconds=5) + store.submit(row.id, "crawl") + now = datetime.now(timezone.utc) + with ThreadPoolExecutor(max_workers=2) as workers: + claims = list(workers.map(lambda _: store.claim(Stage.SEARCH, now), range(2))) + claim = next(c for c in claims if c) + assert sum(c is not None for c in claims) == 1 + assert store.recover(now + timedelta(seconds=6)) == 1 + newer = store.claim(Stage.SEARCH, now + timedelta(seconds=6)) + assert newer.token.lease_id != claim.token.lease_id + assert not store.complete(WorkResult(claim.token, Stage.SEARCH, "", {"late": True}), now + timedelta(seconds=6)) + + +def test_pause_version_change_or_deadline_rejects_late_result(session_factory): + row = task(session_factory) + store = ExecutionStore(session_factory) + store.submit(row.id, "crawl") + now = datetime.now(timezone.utc) + claim = store.claim(Stage.SEARCH, now) + TaskService(session_factory).update_task(row.id, {"target_variant": "256GB"}) + assert not store.complete(WorkResult(claim.token, Stage.SEARCH, "", {"bad": True}), now) + store.invalidate_task(row.id) + store.submit(row.id, "crawl") + claim = store.claim(Stage.SEARCH, now) + assert not store.complete(WorkResult(claim.token, Stage.SEARCH, "", {}), now + timedelta(minutes=16)) + + +def test_global_and_per_task_backpressure_does_not_drop_cursor(session_factory): + row = task(session_factory) + store = ExecutionStore(session_factory, global_limit=3, task_limit=2) + run_id = store.submit(row.id, "crawl") + assert store.enqueue(run_id, Stage.DETAIL, "one", {"listing_id": 1}) + assert not store.enqueue(run_id, Stage.DETAIL, "two", {"listing_id": 2}) + assert store.set_cursor(run_id, {"remaining_ids": [2, 3]}) + with session_factory() as session: + assert session.get(TaskRun, run_id).cursor == {"remaining_ids": [2, 3]} + assert session.query(WorkItem).filter(WorkItem.state.in_(["pending", "running"])).count() == 2 + other = task(session_factory) + assert store.submit(other.id, "crawl") + third = task(session_factory) + assert store.submit(third.id, "crawl") is None + + +def test_task_edits_cancel_pending_work_in_same_transaction(session_factory): + row = task(session_factory) + store = ExecutionStore(session_factory) + run_id = store.submit(row.id) + TaskService(session_factory).toggle_task(row.id) + with session_factory() as session: + assert session.get(TaskRun, run_id).state == "cancelled" + assert session.query(WorkItem).one().state == "cancelled" + + +def test_expired_run_keeps_resume_reference_for_next_scheduled_run(session_factory): + row = task(session_factory) + store = ExecutionStore(session_factory) + old = store.submit(row.id) + now = datetime.now(timezone.utc) + store.claim(Stage.SEARCH, now) + store.set_cursor(old, {"remaining_ids": [10, 11]}) + store.recover(now + timedelta(minutes=16)) + newer = store.submit(row.id) + assert newer != old + with session_factory() as session: + assert session.get(TaskRun, old).state == "deferred" + assert session.get(TaskRun, newer).cursor["resume_run_id"] == old + assert session.get(TaskRun, newer).cursor["remaining_ids"] == [10, 11] From 6ab72a7a272cadfeb595c43fba0131469f7d3eab Mon Sep 17 00:00:00 2001 From: Comui520 Date: Mon, 26 Oct 2026 17:33:11 +0800 Subject: [PATCH 13/17] fix: preserve navigation and isolate validated resource settings --- .env.example | 12 ++ .github/workflows/ci.yml | 12 +- README.md | 14 +- .../plans/2026-10-26-upgrade-roadmap.md | 5 +- goodprice/config.py | 18 ++- goodprice/db.py | 6 + goodprice/execution/limits.py | 51 +++++++ goodprice/main.py | 12 ++ goodprice/media/budget.py | 19 +++ goodprice/media/processing.py | 109 ++++++------- goodprice/models.py | 4 + goodprice/offers/service.py | 33 +++- goodprice/resources.py | 144 ++++++++++++++++++ goodprice/services/crawl_service.py | 26 +++- goodprice/services/media_service.py | 1 + goodprice/services/offer_crawl.py | 21 ++- goodprice/services/offer_pipeline.py | 102 +++++++++---- goodprice/services/settings_service.py | 17 ++- goodprice/web/list_context.py | 64 ++++++++ goodprice/web/media_routes.py | 30 +++- goodprice/web/offer_routes.py | 6 +- goodprice/web/resource_routes.py | 44 ++++++ goodprice/web/routes.py | 120 +++++++++++---- goodprice/web/settings_sections.py | 54 +++++++ goodprice/web/static/THIRD_PARTY.md | 14 ++ goodprice/web/static/app.css | 1 + goodprice/web/static/htmx.LICENSE | 13 ++ goodprice/web/static/htmx.min.js | 1 + goodprice/web/static/navigation.js | 81 ++++++++++ goodprice/web/static/tailwind.LICENSE | 21 +++ goodprice/web/static/tailwind.input.css | 3 + goodprice/web/templates/base.html | 7 +- goodprice/web/templates/dashboard.html | 2 +- goodprice/web/templates/listings.html | 19 ++- goodprice/web/templates/listings_cards.html | 22 ++- goodprice/web/templates/listings_detail.html | 16 +- goodprice/web/templates/listings_more.html | 3 + .../web/templates/listings_more_button.html | 4 + goodprice/web/templates/media_settings.html | 2 +- goodprice/web/templates/notifications.html | 7 +- .../web/templates/resource_settings.html | 22 +++ goodprice/web/templates/settings.html | 41 ++++- goodprice/web/templates/settings_clear.html | 6 + goodprice/web/templates/tasks.html | 13 +- pyproject.toml | 1 + requirements.lock | 23 +++ tests/e2e/conftest.py | 56 +++++++ tests/e2e/test_navigation.py | 83 ++++++++++ tests/e2e/test_settings.py | 19 +++ tests/test_api.py | 2 +- tests/test_execution_limits.py | 18 +++ tests/test_list_context.py | 42 +++++ tests/test_listing_queries.py | 21 +++ tests/test_media_api.py | 22 +++ tests/test_media_concurrency.py | 34 +++++ tests/test_media_regressions.py | 12 ++ tests/test_pipeline_regressions.py | 52 +++++++ tests/test_price_filters.py | 48 ++++++ tests/test_resource_api.py | 25 +++ tests/test_resource_profiles.py | 64 ++++++++ tests/test_settings_sections.py | 43 ++++++ tests/test_task_feedback.py | 15 ++ uv.lock | 30 ++++ 63 files changed, 1648 insertions(+), 184 deletions(-) create mode 100644 goodprice/execution/limits.py create mode 100644 goodprice/resources.py create mode 100644 goodprice/web/list_context.py create mode 100644 goodprice/web/resource_routes.py create mode 100644 goodprice/web/settings_sections.py create mode 100644 goodprice/web/static/THIRD_PARTY.md create mode 100644 goodprice/web/static/app.css create mode 100644 goodprice/web/static/htmx.LICENSE create mode 100644 goodprice/web/static/htmx.min.js create mode 100644 goodprice/web/static/navigation.js create mode 100644 goodprice/web/static/tailwind.LICENSE create mode 100644 goodprice/web/static/tailwind.input.css create mode 100644 goodprice/web/templates/listings_more.html create mode 100644 goodprice/web/templates/listings_more_button.html create mode 100644 goodprice/web/templates/resource_settings.html create mode 100644 goodprice/web/templates/settings_clear.html create mode 100644 tests/e2e/conftest.py create mode 100644 tests/e2e/test_navigation.py create mode 100644 tests/e2e/test_settings.py create mode 100644 tests/test_execution_limits.py create mode 100644 tests/test_list_context.py create mode 100644 tests/test_listing_queries.py create mode 100644 tests/test_media_concurrency.py create mode 100644 tests/test_pipeline_regressions.py create mode 100644 tests/test_price_filters.py create mode 100644 tests/test_resource_api.py create mode 100644 tests/test_resource_profiles.py create mode 100644 tests/test_settings_sections.py create mode 100644 tests/test_task_feedback.py diff --git a/.env.example b/.env.example index 77449c9..7366776 100644 --- a/.env.example +++ b/.env.example @@ -90,3 +90,15 @@ MEDIA_DOWNLOAD_TIMEOUT=60 MEDIA_DECODE_TIMEOUT=30 MEDIA_REQUEST_MAX_MIB=16 MEDIA_FFMPEG_THREADS=1 +# 分阶段执行预算(设备建议不自动提高访问频率或模型RPM) +EXECUTION_ACTIVE_RUNS=2 +EXECUTION_CRAWLER_WORKERS=1 +EXECUTION_MODEL_WORKERS=1 +EXECUTION_MEDIA_WORKERS=1 +EXECUTION_PLATFORM_INTERVAL_SECONDS=10 +EXECUTION_MODEL_RPM=10 +EXECUTION_GLOBAL_ITEMS=200 +EXECUTION_TASK_ITEMS=60 +EXECUTION_RUN_DETAILS=30 +EXECUTION_RUN_HTTP_ATTEMPTS=60 +EXECUTION_RUN_MINUTES=15 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 08f38a5..95521e7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -5,6 +5,7 @@ on: push: branches: - main + - release permissions: contents: read @@ -14,8 +15,12 @@ concurrency: jobs: test: - name: Python tests - runs-on: ubuntu-latest + name: Python and browser tests (${{ matrix.os }}) + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + os: [ubuntu-latest, windows-latest] steps: - name: Check out repository uses: actions/checkout@v4 @@ -39,6 +44,9 @@ jobs: - name: Check whitespace errors run: git diff --check + - name: Install test browser + run: uv run --locked playwright install --with-deps chromium + - name: Lint (ruff) run: uv run --locked ruff check . diff --git a/README.md b/README.md index 3aedd2e..f62cbe2 100644 --- a/README.md +++ b/README.md @@ -222,7 +222,7 @@ Conda 不是运行本项目的必需条件;新用户优先选择 Docker 或 uv 设置最低品相分后,未核验的图片或多规格共享样品不会绕过门槛。 当前支持可明确识别的商品范围 DOM 数据与 Schema.org Product/Offer,尚未取得真实多规格页面样本来验证所有闲鱼控件, -不能识别时显示待核验而不是猜测。分阶段并发调度与完整交互改造仍属后续工作,不能将本开发分支当作已完成的稳定发布。 +不能识别时显示待核验而不是猜测。分阶段并发调度仍在接入中,不能将本开发分支当作已完成的稳定发布。 ### 开发分支:图片与视频证据 @@ -240,6 +240,18 @@ Docker镜像内置FFmpeg/ffprobe。本地uv/Conda运行须自行安装FFmpeg并 缓存被清理后保留证据摘要与时间位置,页面显示文件不可用;媒体文件通过内部编号访问,不公开签名下载地址或本地路径。 相同内容的签名URL变化不会单独触发重复视觉收费,内容/需求/模型/选图设置变化才使分析缓存失效。 +### 开发分支:导航与设置整理 + +列表的任务、状态、价格状态、排序、已加载范围会带入详情及操作返回链接。加载更多不再反复请求同一页, +页面返回与原生后退可恢复列表位置;选择商品或操作表单时不自动替换正在使用的列表。 +设置分为账号、模型、通知、媒体与资源,单独保存互不覆盖,凭据留空保持原值,清除需明确确认。 +页面使用随包提供的 HTMX 和预编译样式,保留现有简洁界面,普通启动不需要连接 CDN 或安装 Node.js。 +维护者修改样式类后可按 `goodprice/web/static/THIRD_PARTY.md` 重建 CSS。 + +设备建议只在本机检测 CPU、内存及容器限额,不上传机器标识或自动提高配置;低占用/均衡仅设置工作并发, +不会增加视觉输入、开启视频或提高访问频率。执行控制与通知 outbox 的基础组件仍在接入, +在完整调度验收前,资源表单中的调度参数不能视为已替换旧队列,也不据此承诺提速。 + 应用现使用无导入副作用的工厂;自定义 ASGI 部署请使用 `uvicorn goodprice.main:create_app --factory`, 原有 `python -m goodprice` 启动方式不变。升级旧库前先停止服务并备份数据库,回退使用旧镜像与备份库成对恢复。 diff --git a/docs/superpowers/plans/2026-10-26-upgrade-roadmap.md b/docs/superpowers/plans/2026-10-26-upgrade-roadmap.md index a11bfb1..52a78e6 100644 --- a/docs/superpowers/plans/2026-10-26-upgrade-roadmap.md +++ b/docs/superpowers/plans/2026-10-26-upgrade-roadmap.md @@ -72,7 +72,10 @@ ## 当前状态 -仅设计与实施计划已保存,本清单的实现项均未执行。下一步由用户审阅计划并选择执行方式。 +设计与计划已批准,采用主线程持续实施、阶段末独立审查。阶段一的规格/价格核心和阶段二媒体核心已实施并记录验证, +真实闲鱼规格与媒体控件仍待脱敏样本验证。阶段三已实施导航/设置整理、查询批量化、设备检测及执行/通知基础组件, +协调器尚未替换正式应用的旧队列;预算/限速、浏览器所有权、恢复与投递的完整集成、性能对比和最终独立审查仍是必需工作。 +不能将基础模块单测通过写成整项升级完成;不自动推送、发版或替换生产服务。 ## 自审覆盖记录 diff --git a/goodprice/config.py b/goodprice/config.py index 1bb6ac4..ca12d29 100644 --- a/goodprice/config.py +++ b/goodprice/config.py @@ -19,6 +19,7 @@ "feishu_secret", "gotify_token", "jev_api_key", + "xianyu_cookie", } ) @@ -49,8 +50,8 @@ class Settings(BaseSettings): llm_api_format: str = "chat_completions" serverchan_sendkey: str = "" proxy: str = "" - default_crawl_interval_minutes: int = 20 - default_crawl_jitter_minutes: int = 10 + default_crawl_interval_minutes: int = Field(20, ge=1) + default_crawl_jitter_minutes: int = Field(10, ge=0) vision_base_url: str = "" vision_api_key: str = "" vision_model: str = "glm-4.6v-flash" @@ -61,7 +62,7 @@ class Settings(BaseSettings): feishu_enabled: bool = True gotify_url: str = "" gotify_token: str = "" - gotify_priority: int = 5 + gotify_priority: int = Field(5, ge=0, le=10) gotify_enabled: bool = True serverchan_enabled: bool = True wecom_robot_enabled: bool = True @@ -90,6 +91,17 @@ class Settings(BaseSettings): media_decode_timeout: int = Field(30, ge=10, le=120) media_request_max_mib: int = Field(16, ge=1, le=32) media_ffmpeg_threads: int = Field(1, ge=1, le=4) + execution_active_runs: int = Field(2, ge=1, le=4) + execution_crawler_workers: int = Field(1, ge=1, le=2) + execution_model_workers: int = Field(1, ge=1, le=8) + execution_media_workers: int = Field(1, ge=1, le=2) + execution_platform_interval_seconds: int = Field(10, ge=5, le=300) + execution_model_rpm: int = Field(10, ge=1, le=60) + execution_global_items: int = Field(200, ge=1, le=200) + execution_task_items: int = Field(60, ge=1, le=60) + execution_run_details: int = Field(30, ge=1, le=100) + execution_run_http_attempts: int = Field(60, ge=1, le=200) + execution_run_minutes: int = Field(15, ge=5, le=60) @model_validator(mode="after") def _media_budgets(self): diff --git a/goodprice/db.py b/goodprice/db.py index b849dcc..3261ac0 100644 --- a/goodprice/db.py +++ b/goodprice/db.py @@ -85,6 +85,12 @@ def init_db(database_url: str) -> None: def migrate_schema(session_factory) -> None: """幂等迁移:为已有数据库补齐新列。""" columns = { + "listing_offers": [ + ("effective_price_state", "effective_price_state VARCHAR(20) DEFAULT 'unknown'"), + ("effective_price_cents", "effective_price_cents INTEGER"), + ("price_valid_until", "price_valid_until DATETIME"), + ("price_task_version", "price_task_version INTEGER"), + ], "watch_tasks": [ ("fetch_detail", "fetch_detail BOOLEAN DEFAULT 1"), ("min_price", "min_price FLOAT DEFAULT 0"), diff --git a/goodprice/execution/limits.py b/goodprice/execution/limits.py new file mode 100644 index 0000000..f7ca703 --- /dev/null +++ b/goodprice/execution/limits.py @@ -0,0 +1,51 @@ +import threading +from dataclasses import dataclass + + +@dataclass(frozen=True) +class ResourceLimits: + active_runs: int = 2 + crawler_workers: int = 1 + model_workers: int = 1 + media_workers: int = 1 + platform_interval_seconds: int = 10 + model_rpm: int = 10 + global_items: int = 200 + task_items: int = 60 + run_details: int = 30 + run_http_attempts: int = 60 + run_minutes: int = 15 + + def __post_init__(self): + bounds = {"active_runs": (1, 4), "crawler_workers": (1, 2), "model_workers": (1, 8), + "media_workers": (1, 2), "platform_interval_seconds": (5, 300), "model_rpm": (1, 60), + "global_items": (1, 200), "task_items": (1, 60), "run_details": (1, 100), + "run_http_attempts": (1, 200), "run_minutes": (5, 60)} + for field, (minimum, maximum) in bounds.items(): + if not minimum <= getattr(self, field) <= maximum: + raise ValueError(f"{field} must be {minimum}..{maximum}") + + @classmethod + def from_settings(cls, settings): + return cls(**{name: getattr(settings, f"execution_{name}") for name in cls.__dataclass_fields__}) + + +class RateGate: + def __init__(self, interval_seconds): + self.interval = interval_seconds + self._next = {} + self._lock = threading.Lock() + + def ready_at(self, key, now): + with self._lock: + return max(now, self._next.get(key, now)) + + def reserve(self, key, now): + with self._lock: + allowed = max(now, self._next.get(key, now)) + self._next[key] = allowed + self.interval + return allowed + + def postpone(self, key, until): + with self._lock: + self._next[key] = max(until, self._next.get(key, until)) diff --git a/goodprice/main.py b/goodprice/main.py index 0a522fa..084dda0 100644 --- a/goodprice/main.py +++ b/goodprice/main.py @@ -7,9 +7,12 @@ import uvicorn from fastapi import FastAPI +from fastapi.responses import JSONResponse +from fastapi.staticfiles import StaticFiles from goodprice.config import Settings, get_settings from goodprice.db import init_db, migrate_schema +from goodprice.media.budget import BudgetExceeded from goodprice.scheduler import _sync_tasks, build_scheduler from goodprice.services.crawl_service import CrawlService, TaskRunGuard from goodprice.services.seller_service import SellerService @@ -18,7 +21,9 @@ from goodprice.services.task_service import TaskService from goodprice.web.media_routes import router as media_router from goodprice.web.offer_routes import router as offer_router +from goodprice.web.resource_routes import router as resource_router from goodprice.web.routes import router +from goodprice.web.settings_sections import router as settings_sections_router LOG_DIR = Path(__file__).resolve().parent.parent / "data" / "logs" @@ -167,6 +172,11 @@ async def lifespan(app: FastAPI): task_queue.stop() app = FastAPI(title=settings.app_name, lifespan=lifespan) + @app.exception_handler(BudgetExceeded) + async def media_busy_handler(request, exc): + return JSONResponse({"detail": "媒体处理繁忙或磁盘预算不足,请稍后重试"}, + status_code=503, headers={"Retry-After": "5"}) + app.mount("/static", StaticFiles(directory=str(Path(__file__).parent / "web" / "static")), name="static") app.state.session_factory = session_factory app.state.settings_service = settings_service app.state.task_service = task_service @@ -179,6 +189,8 @@ async def lifespan(app: FastAPI): app.include_router(router) app.include_router(offer_router) app.include_router(media_router) + app.include_router(settings_sections_router) + app.include_router(resource_router) return app diff --git a/goodprice/media/budget.py b/goodprice/media/budget.py index f080104..872541c 100644 --- a/goodprice/media/budget.py +++ b/goodprice/media/budget.py @@ -21,6 +21,10 @@ class _State: lock: threading.RLock = field(default_factory=threading.RLock) reservations: dict = field(default_factory=dict) pins: dict = field(default_factory=dict) + processing: int = 0 + + def __post_init__(self): + self.condition = threading.Condition(self.lock) _states = {} @@ -125,6 +129,19 @@ def release(self, token: str) -> None: self.checked(reservation.directory) shutil.rmtree(reservation.directory) + @contextmanager + def processing_slot(self, limit, timeout_seconds=5): + with self.state.condition: + if not self.state.condition.wait_for(lambda: self.state.processing < limit, timeout_seconds): + raise BudgetExceeded("媒体处理繁忙,请稍后重试") + self.state.processing += 1 + try: + yield + finally: + with self.state.condition: + self.state.processing -= 1 + self.state.condition.notify_all() + @contextmanager def pin(self, paths): with self.state.lock: @@ -137,3 +154,5 @@ def pin(self, paths): with self.state.lock: for p in checked: self.state.pins[p] -= 1 + if not self.state.pins[p]: + del self.state.pins[p] diff --git a/goodprice/media/processing.py b/goodprice/media/processing.py index 65f37a6..c2baffe 100644 --- a/goodprice/media/processing.py +++ b/goodprice/media/processing.py @@ -46,59 +46,60 @@ def session(self, candidates, settings): evidence, limitations = [], [] counts = {"image": 0, "video": 0} with ExitStack() as pins: - for candidate in candidates: - limit = settings.media_image_count if candidate.kind == "image" else settings.media_video_count - if counts[candidate.kind] >= limit: - limitations.append("retention_limit") - continue - counts[candidate.kind] += 1 - if candidate.status == "unsupported": - limitations.append("unsupported_media") - continue - if candidate.kind == "video" and not settings.video_analysis_enabled: - limitations.append("video_disabled") - continue - download_size = (settings.media_video_max_mib if candidate.kind == "video" else 10) * 1024**2 - output_size = MAX_OUTPUT_BYTES * (settings.media_frames_per_video if candidate.kind == "video" else 1) - try: - with self.budget.reserve(candidate.key, download_size + output_size + 65536) as reservation: - raw = self.downloader.download(candidate, download_size, settings.media_download_timeout, - reservation=reservation) - with raw.open("rb") as source: - source_hash = hashlib.file_digest(source, "sha256").hexdigest() - parameters = {"version": 1, "kind": candidate.kind, "edge": 1280, - "frames": settings.media_frames_per_video, - "max_seconds": settings.media_video_max_seconds} - key = processing_key(source_hash, parameters) - with self.budget.state.lock: - cached = find_cached(self.budget, key) - if cached: - data, paths = cached - pins.enter_context(self.budget.pin(paths)) - if not cached: - if candidate.kind == "image": - outputs = [prepare_image(raw)] - data = [{"file": outputs[0].name, "timestamp": None, - "hash": hashlib.sha256(outputs[0].read_bytes()).hexdigest()}] - else: - frames = extract_frames(raw, settings.media_frames_per_video, reservation.directory / "frames", - settings.media_decode_timeout, settings.media_ffmpeg_threads, - max_seconds=settings.media_video_max_seconds) - outputs = [f.path for f in frames] - data = [{"file": f.path.relative_to(reservation.directory).as_posix(), "timestamp": f.timestamp_ms, - "hash": f.content_hash} for f in frames] - raw.unlink(missing_ok=True) - manifest = reservation.directory / f"{key}.json" - manifest.write_text(json.dumps(data), encoding="utf-8") + with self.budget.processing_slot(settings.execution_media_workers): + for candidate in candidates: + limit = settings.media_image_count if candidate.kind == "image" else settings.media_video_count + if counts[candidate.kind] >= limit: + limitations.append("retention_limit") + continue + counts[candidate.kind] += 1 + if candidate.status == "unsupported": + limitations.append("unsupported_media") + continue + if candidate.kind == "video" and not settings.video_analysis_enabled: + limitations.append("video_disabled") + continue + download_size = (settings.media_video_max_mib if candidate.kind == "video" else 10) * 1024**2 + output_size = MAX_OUTPUT_BYTES * (settings.media_frames_per_video if candidate.kind == "video" else 1) + try: + with self.budget.reserve(candidate.key, download_size + output_size + 65536) as reservation: + raw = self.downloader.download(candidate, download_size, settings.media_download_timeout, + reservation=reservation) + with raw.open("rb") as source: + source_hash = hashlib.file_digest(source, "sha256").hexdigest() + parameters = {"version": 1, "kind": candidate.kind, "edge": 1280, + "frames": settings.media_frames_per_video, + "max_seconds": settings.media_video_max_seconds} + key = processing_key(source_hash, parameters) with self.budget.state.lock: - self.budget.commit(reservation.token, outputs + [manifest]) - paths = reservation.committed_paths[:-1] - pins.enter_context(self.budget.pin(paths)) - for row, path in zip(data, paths, strict=True): - identity = hashlib.sha256(f"{row['hash']}:{candidate.offer_key}:{row['timestamp']}".encode()).hexdigest()[:32] - evidence.append(MediaEvidence(identity, "image" if candidate.kind == "image" else "frame", - row["hash"], path, candidate.offer_key, - candidate.key if candidate.kind == "video" else None, row["timestamp"])) - except (ValueError, RuntimeError, OSError, MediaDownloadError): - limitations.append(f"{candidate.kind}_unavailable") + cached = find_cached(self.budget, key) + if cached: + data, paths = cached + pins.enter_context(self.budget.pin(paths)) + if not cached: + if candidate.kind == "image": + outputs = [prepare_image(raw)] + data = [{"file": outputs[0].name, "timestamp": None, + "hash": hashlib.sha256(outputs[0].read_bytes()).hexdigest()}] + else: + frames = extract_frames(raw, settings.media_frames_per_video, reservation.directory / "frames", + settings.media_decode_timeout, settings.media_ffmpeg_threads, + max_seconds=settings.media_video_max_seconds) + outputs = [f.path for f in frames] + data = [{"file": f.path.relative_to(reservation.directory).as_posix(), "timestamp": f.timestamp_ms, + "hash": f.content_hash} for f in frames] + raw.unlink(missing_ok=True) + manifest = reservation.directory / f"{key}.json" + manifest.write_text(json.dumps(data), encoding="utf-8") + with self.budget.state.lock: + self.budget.commit(reservation.token, outputs + [manifest]) + paths = reservation.committed_paths[:-1] + pins.enter_context(self.budget.pin(paths)) + for row, path in zip(data, paths, strict=True): + identity = hashlib.sha256(f"{row['hash']}:{candidate.offer_key}:{row['timestamp']}".encode()).hexdigest()[:32] + evidence.append(MediaEvidence(identity, "image" if candidate.kind == "image" else "frame", + row["hash"], path, candidate.offer_key, + candidate.key if candidate.kind == "video" else None, row["timestamp"])) + except (ValueError, RuntimeError, OSError, MediaDownloadError): + limitations.append(f"{candidate.kind}_unavailable") yield evidence, tuple(dict.fromkeys(limitations)) diff --git a/goodprice/models.py b/goodprice/models.py index 09099ed..4823203 100644 --- a/goodprice/models.py +++ b/goodprice/models.py @@ -157,6 +157,10 @@ class ListingOffer(Base): last_notified_score: Mapped[Optional[float]] = mapped_column(Float, nullable=True) last_notified_amount: Mapped[Optional[int]] = mapped_column(Integer, nullable=True) last_notified_source: Mapped[Optional[str]] = mapped_column(String(30), nullable=True) + effective_price_state: Mapped[str] = mapped_column(String(20), default="unknown") + effective_price_cents: Mapped[Optional[int]] = mapped_column(Integer, nullable=True) + price_valid_until: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True) + price_task_version: Mapped[Optional[int]] = mapped_column(Integer, nullable=True) observations: Mapped[list["OfferObservation"]] = relationship(cascade="all, delete-orphan") quotes: Mapped[list["ManualQuote"]] = relationship(cascade="all, delete-orphan") evaluations: Mapped[list["OfferEvaluationRecord"]] = relationship(cascade="all, delete-orphan") diff --git a/goodprice/offers/service.py b/goodprice/offers/service.py index 347c1a2..28ba7ab 100644 --- a/goodprice/offers/service.py +++ b/goodprice/offers/service.py @@ -1,3 +1,4 @@ +from contextlib import nullcontext from datetime import datetime, timezone from goodprice.models import ( @@ -8,7 +9,7 @@ OfferObservation, WatchTask, ) -from goodprice.offers.pricing import offer_fingerprint, utc +from goodprice.offers.pricing import decide_price, offer_fingerprint, utc from goodprice.offers.types import OfferCandidate, PriceClaim, QuoteInput @@ -28,10 +29,23 @@ class OfferService: def __init__(self, session_factory): self._session_factory = session_factory - def record_observation(self, listing_id: int, offer: OfferCandidate) -> int: + def _refresh_price_index(self, session, row, listing, candidate, quote=None): + task = session.get(WatchTask, listing.task_id) if listing.task_id else None + if quote is None: + quote = session.query(ManualQuote).filter_by(offer_id=row.id, revoked=False).order_by(ManualQuote.id.desc()).first() + value = None if quote is None else QuoteInput(row.offer_key, quote.amount_cents, + utc(quote.confirmed_at), utc(quote.expires_at), quote.evidence_fingerprint, quote.revoked) + ttl = task.interval_minutes * 60 if task and task.fetch_detail else 0 + decision = decide_price(candidate, value, datetime.now(timezone.utc), ttl) + row.effective_price_state, row.effective_price_cents = decision.state, decision.amount_cents + row.price_valid_until = utc(decision.fresh_until).replace(tzinfo=None) if decision.fresh_until else None + row.price_task_version = task.version if task else None + + def record_observation(self, listing_id: int, offer: OfferCandidate, *, session=None) -> int: if not offer.key or not offer.claims: raise ValueError("规格或证据不能为空") - with self._session_factory() as session: + own = session is None + with self._session_factory() if own else nullcontext(session) as session: listing = session.get(Listing, listing_id) if listing is None: raise LookupError("商品不存在") @@ -48,7 +62,10 @@ def record_observation(self, listing_id: int, offer: OfferCandidate) -> int: session.add(OfferObservation(offer_id=row.id, fingerprint=offer_fingerprint(offer), payload=encode_offer(offer), observed_at=min(utc(c.observed_at) for c in offer.claims))) - session.commit() + session.flush() + self._refresh_price_index(session, row, listing, offer) + if own: + session.commit() return row.id def confirm_quote(self, offer_id: int, quote: QuoteInput) -> int: @@ -80,6 +97,7 @@ def confirm_quote(self, offer_id: int, quote: QuoteInput) -> int: evidence_fingerprint=quote.evidence_fingerprint, revoked=False) session.add(row) listing.offer_verified_until = None + self._refresh_price_index(session, offer, listing, candidate, quote=row) session.commit() return row.id @@ -90,5 +108,10 @@ def revoke_quote(self, quote_id: int) -> None: raise LookupError("报价不存在") quote.revoked = True offer = session.get(ListingOffer, quote.offer_id) - session.get(Listing, offer.listing_id).offer_verified_until = None + listing = session.get(Listing, offer.listing_id) + listing.offer_verified_until = None + session.flush() + observation = session.query(OfferObservation).filter_by(offer_id=offer.id).order_by(OfferObservation.id.desc()).first() + if observation: + self._refresh_price_index(session, offer, listing, decode_offer(observation.payload)) session.commit() diff --git a/goodprice/resources.py b/goodprice/resources.py new file mode 100644 index 0000000..bb875ad --- /dev/null +++ b/goodprice/resources.py @@ -0,0 +1,144 @@ +"""Local, read-only capability detection. Recommendations never apply themselves.""" +import platform +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path, PurePosixPath + + +@dataclass(frozen=True) +class ResourceSnapshot: + effective_cpu: float | None + memory_limit_bytes: int | None + available_memory_bytes: int | None + source: str + warnings: tuple[str, ...] + detected_at: datetime + + +def resource_profile(name): + if name not in {"low", "balanced"}: + raise ValueError("Unknown resource profile") + return {"execution_active_runs": 2, "execution_crawler_workers": 1, + "execution_model_workers": 2 if name == "balanced" else 1, + "execution_media_workers": 1, "media_ffmpeg_threads": 1} + + +def recommend_profile(snapshot): + balanced = (snapshot.source != "unknown" and (snapshot.effective_cpu or 0) >= 4 + and (snapshot.memory_limit_bytes or 0) >= 6 * 1024**3 + and (snapshot.available_memory_bytes or 0) >= 3 * 1024**3) + name = "balanced" if balanced else "low" + return {"name": name, "values": resource_profile(name), + "reason": "有效CPU及内存余量足够,可选择均衡" if balanced else "设备余量较小或检测不完整,建议低占用"} + + +def _cpus(raw): + values = set() + for part in raw.strip().split(","): + if not part: + continue + bounds = part.split("-") + start, end = int(bounds[0]), int(bounds[-1]) + if start < 0 or end < start or end - start > 65536: + raise ValueError() + values.update(range(start, end + 1)) + return values + + +def _unescape(raw): + for escaped, value in [("\\040", " "), ("\\011", "\t"), ("\\134", "\\")]: + raw = raw.replace(escaped, value) + return raw + + +def detect_resources(*, system=None, read=None, host=None, affinity=None): + system = system or platform.system() + read = read or (lambda p: Path(p).read_text(encoding="utf-8")) + warnings = [] + if host is None: + try: + import psutil + memory = psutil.virtual_memory() + host = (psutil.cpu_count(logical=True), memory.total, memory.available) + try: + affinity = set(psutil.Process().cpu_affinity()) + except (AttributeError, NotImplementedError, psutil.Error): + pass + except (ImportError, OSError): + host = (None, None, None) + warnings.append("系统资源检测不可用") + cpu, memory_limit, available = host + if affinity: + cpu = min(cpu, len(affinity)) if cpu else len(affinity) + source = "host" if all(v is not None for v in host) else "unknown" + def optional(path): + try: + return read(str(path)).strip() + except (OSError, KeyError, ValueError): + return "" + if system == "Linux": + groups = optional("/proc/self/cgroup") + mounts = optional("/proc/self/mountinfo") + if not groups or not mounts: + source = "unknown" + warnings.append("无法读取容器层级,保守推荐") + else: + mappings = [] + for line in groups.splitlines(): + parts = line.split(":", 2) + if len(parts) == 3: + mappings.append((set(parts[1].split(",")), PurePosixPath(parts[2]))) + for line in mounts.splitlines(): + fields = line.split() + try: + sep = fields.index("-") + fs = fields[sep+1] + if fs not in {"cgroup", "cgroup2"}: + continue + mount_root, mount = PurePosixPath(_unescape(fields[3])), PurePosixPath(_unescape(fields[4])) + controllers = set(fields[sep+3].split(",")) + for names, process in mappings: + if fs == "cgroup" and not (controllers & names): + continue + if fs == "cgroup2" and names != {""}: + continue + if not process.is_relative_to(mount_root): + continue + directory = mount / process.relative_to(mount_root) + for ancestor in (directory, *directory.parents): + if not ancestor.is_relative_to(mount): + break + quota = optional(ancestor / "cpu.max") if fs == "cgroup2" else " ".join([ + optional(ancestor / "cpu.cfs_quota_us"), optional(ancestor / "cpu.cfs_period_us")]) + parts = quota.split() + if len(parts) == 2 and parts[0] not in {"max", "-1"}: + limit = int(parts[0]) / int(parts[1]) + if limit > 0: + cpu = min(cpu, limit) if cpu else limit + source = "cgroup" + cpuset = optional(ancestor / ("cpuset.cpus.effective" if fs == "cgroup2" else "cpuset.cpus")) + if cpuset: + ids = _cpus(cpuset) + if affinity: + ids &= affinity + if ids: + cpu = min(cpu, len(ids)) if cpu else len(ids) + source = "cgroup" + raw_limit = optional(ancestor / ("memory.max" if fs == "cgroup2" else "memory.limit_in_bytes")) + raw_current = optional(ancestor / ("memory.current" if fs == "cgroup2" else "memory.usage_in_bytes")) + if raw_limit and raw_limit != "max": + limit = int(raw_limit) + if 0 < limit < 2**60: + memory_limit = min(memory_limit, limit) if memory_limit else limit + source = "cgroup" + if raw_current: + remaining = max(0, limit - int(raw_current)) + available = min(available, remaining) if available is not None else remaining + else: + available = 0 + warnings.append("容器内存占用不可读") + except (ValueError, IndexError, ZeroDivisionError): + source = "unknown" + warnings.append("容器资源格式无法解析") + return ResourceSnapshot(float(cpu) if cpu is not None else None, memory_limit, available, + source, tuple(dict.fromkeys(warnings)), datetime.now(timezone.utc)) diff --git a/goodprice/services/crawl_service.py b/goodprice/services/crawl_service.py index 4ca61ad..73c7481 100644 --- a/goodprice/services/crawl_service.py +++ b/goodprice/services/crawl_service.py @@ -66,8 +66,12 @@ def run_task(self, task_id: int) -> dict[str, Any]: return {"found": 0, "new": 0, "notified": 0, "skipped": "already_running"} try: from goodprice.services.offer_crawl import OfferCrawlRunner + from goodprice.services.offer_pipeline import StaleEvaluation - return OfferCrawlRunner(self).run(task_id) + try: + return OfferCrawlRunner(self).run(task_id) + except StaleEvaluation: + return {"found": 0, "new": 0, "notified": 0, "skipped": "superseded"} except Exception as exc: self._record_error(task_id, f"处理商品时出错: {exc}"[:1000]) raise @@ -105,8 +109,22 @@ def _upsert_listing(self, session, task: WatchTask, data: ListingData): listing.price = data.price session.add(PriceSnapshot(listing_id=listing.id, price=data.price)) listing.last_seen_at = datetime.now() + listing.title, listing.url = data.title, data.url return listing, False + def _commit_if_current(self, session, task) -> bool: + """Acquire the short write lock and gate all pending result mutations.""" + from sqlalchemy import update + + result = session.execute(update(WatchTask).where(WatchTask.id == task.id, + WatchTask.version == task.version, WatchTask.enabled.is_(True)).values( + version=WatchTask.version).execution_options(synchronize_session=False)) + if result.rowcount != 1: + session.rollback() + return False + session.commit() + return True + def _is_blocked(self, session, listing: Listing) -> bool: if listing.blocked: return True @@ -157,8 +175,12 @@ def _fetch_detail(self, session, listing: Listing, force: bool = False) -> Listi def reanalyze_listing(self, listing_id: int) -> dict[str, Any]: """Revalidate offer evidence before analysis, without sending notifications.""" from goodprice.services.offer_crawl import OfferCrawlRunner + from goodprice.services.offer_pipeline import StaleEvaluation - return OfferCrawlRunner(self).reanalyze(listing_id) + try: + return OfferCrawlRunner(self).reanalyze(listing_id) + except StaleEvaluation: + return {"updated": 0, "skipped": "superseded"} def _seller_check(self, session, listing: Listing, task: WatchTask) -> None: if not listing.seller_uid or self.seller_service is None: diff --git a/goodprice/services/media_service.py b/goodprice/services/media_service.py index 820abbc..efd0541 100644 --- a/goodprice/services/media_service.py +++ b/goodprice/services/media_service.py @@ -23,6 +23,7 @@ def record(self, listing_id, candidates): row = MediaAsset(listing_id=listing_id, key=candidate.key, kind=candidate.kind) session.add(row) row.source_url, row.source_locator = candidate.url, candidate.source_locator + row.relative_path, row.content_hash = None, None row.offer_key, row.status = candidate.offer_key, candidate.status row.observed_at = datetime.now() session.commit() diff --git a/goodprice/services/offer_crawl.py b/goodprice/services/offer_crawl.py index 37196dc..276a701 100644 --- a/goodprice/services/offer_crawl.py +++ b/goodprice/services/offer_crawl.py @@ -81,7 +81,7 @@ def run(self, task_id: int) -> dict: changed = is_new or was_missing or len(candidates) > 1 or any( offer_fingerprint(c) != old_fingerprint for c in candidates) # Freshness renews each observation; content identity does not. - ids = self.pipeline.evaluate_listing(listing_id, task_id, candidates, now) + ids = self.pipeline.evaluate_listing(listing_id, task_id, candidates, now, expected_version=task.version) with self.factory() as session: records = [session.get(OfferEvaluationRecord, i) for i in ids] potential = any(r.payload["state"] == "exact" and r.payload["match"] == "matched" @@ -97,7 +97,7 @@ def run(self, task_id: int) -> dict: changed = True listing = session.get(Listing, listing_id) settings = self.crawl.settings_service.get() - inputs = {"description": listing.description, "images": listing.image_urls, + inputs = {"title": listing.title, "description": listing.description, "images": listing.image_urls, "requirement": task.condition_requirement, "target": task.target_variant, "vision": self.crawl.vision.enabled, "model": settings.llm_model, "vision_model": settings.vision_model, "prompt": "offer-v1", @@ -105,7 +105,7 @@ def run(self, task_id: int) -> dict: "llm_url": settings.llm_base_url, "vision_url": settings.vision_base_url} analysis_fp = hashlib.sha256(json.dumps(inputs, sort_keys=True).encode()).hexdigest() content_fp = hashlib.sha256(json.dumps( - {"description": listing.description, "images": listing.image_urls}, sort_keys=True).encode()).hexdigest() + {"title": listing.title, "description": listing.description, "images": listing.image_urls}, sort_keys=True).encode()).hexdigest() unchanged_content = all((previous_evaluations.get(c.key) or {}).get("analysis_content_fingerprint") == content_fp for c in candidates) needs_backfill = (not changed and unchanged_content and self.crawl.vision.enabled @@ -130,8 +130,9 @@ def run(self, task_id: int) -> dict: if self.crawl.media_service and (previous_media != (listing.condition_detail or {}).get("media_cache_key") or previous_score != listing.condition_score): changed = True - session.commit() - ids = self.pipeline.evaluate_listing(listing_id, task_id, candidates, now) + if not self.crawl._commit_if_current(session, task): + return {**stats, "skipped": "superseded"} + ids = self.pipeline.evaluate_listing(listing_id, task_id, candidates, now, expected_version=task.version) stats["reevaluated"] += int(not is_new and changed) stats["backfilled"] += int(needs_backfill) for record_id in ids: @@ -237,6 +238,9 @@ def _notify_pending(self, session, task, listing, offer, payload): def _fetch(self, listing_id, task): detail = None + if not task.fetch_detail: + return [OfferCandidate("default", claims=(PriceClaim("dom", "unknown", + "详情抓取关闭,价格和规格待核验", datetime.now(timezone.utc), "detail:disabled"),))] if task.fetch_detail: with self.factory() as session: listing = session.get(Listing, listing_id) @@ -268,7 +272,7 @@ def reanalyze(self, listing_id): raise RuntimeError("原监控任务已删除,无法按旧条件重新分析") candidates = self._fetch(listing_id, task) now = datetime.now(timezone.utc) - ids = self.pipeline.evaluate_listing(listing_id, task.id, candidates, now) + ids = self.pipeline.evaluate_listing(listing_id, task.id, candidates, now, expected_version=task.version) with self.factory() as session: listing = session.get(Listing, listing_id) potential = any(session.get(OfferEvaluationRecord, i).payload["state"] == "exact" for i in ids) @@ -283,8 +287,9 @@ def reanalyze(self, listing_id): self.crawl._seller_check(session, listing, task) if not potential: listing.value_score, listing.best_of_batch = None, False - session.commit() - ids = self.pipeline.evaluate_listing(listing_id, task.id, candidates, datetime.now(timezone.utc)) + if not self.crawl._commit_if_current(session, task): + return {"updated": 0, "skipped": "superseded"} + ids = self.pipeline.evaluate_listing(listing_id, task.id, candidates, datetime.now(timezone.utc), expected_version=task.version) pending = [] with self.factory() as session: listing = session.get(Listing, listing_id) diff --git a/goodprice/services/offer_pipeline.py b/goodprice/services/offer_pipeline.py index 32187da..b23380a 100644 --- a/goodprice/services/offer_pipeline.py +++ b/goodprice/services/offer_pipeline.py @@ -16,6 +16,10 @@ from goodprice.offers.types import OfferCandidate, QuoteInput +class StaleEvaluation(RuntimeError): + pass + + class OfferPipeline: def __init__(self, session_factory, vision_enabled: bool = True): self._session_factory = session_factory @@ -23,14 +27,21 @@ def __init__(self, session_factory, vision_enabled: bool = True): self.offers = OfferService(session_factory) def evaluate_listing(self, listing_id: int, task_id: int, - candidates: list[OfferCandidate], now: datetime) -> list[int]: + candidates: list[OfferCandidate], now: datetime, *, expected_version=None) -> list[int]: # Network/AI callers must finish their work before this short write phase. - ids = [self.offers.record_observation(listing_id, o) for o in candidates[:20]] result = [] with self._session_factory() as session: listing, task = session.get(Listing, listing_id), session.get(WatchTask, task_id) if listing is None or task is None or listing.task_id != task.id: raise LookupError("任务或商品不存在/关联不一致") + if expected_version is not None: + from sqlalchemy import update + gate = session.execute(update(WatchTask).where(WatchTask.id == task_id, + WatchTask.version == expected_version, WatchTask.enabled.is_(True)).values( + version=WatchTask.version).execution_options(synchronize_session=False)) + if gate.rowcount != 1: + raise StaleEvaluation("任务已暂停或条件已变化") + ids = [self.offers.record_observation(listing_id, o, session=session) for o in candidates[:20]] for candidate, offer_id in zip(candidates, ids, strict=False): quote = session.query(ManualQuote).filter_by(offer_id=offer_id, revoked=False).order_by( ManualQuote.id.desc()).first() @@ -79,39 +90,71 @@ def latest(self, listing_id: int) -> list[dict]: return result def snapshot(self, listing_id: int, now: datetime) -> list[dict]: - """Read-time validity: stale stored recommendations must not stay eligible.""" + result = self.snapshot_many([listing_id], now) + if listing_id not in result: + raise LookupError("商品不存在") + return result[listing_id] + + def snapshot_many(self, listing_ids, now: datetime) -> dict[int, list[dict]]: + """One bounded set of latest-row queries for the whole visible page.""" + from sqlalchemy import func + from goodprice.offers.service import decode_offer - result = self.latest(listing_id) + if not listing_ids: + return {} with self._session_factory() as session: - listing = session.get(Listing, listing_id) - if listing is None: - raise LookupError("商品不存在") - task = session.get(WatchTask, listing.task_id) if listing.task_id else None - for item in result: - observation = session.query(OfferObservation).filter_by(offer_id=item["id"]).order_by( - OfferObservation.observed_at.desc(), OfferObservation.id.desc()).first() - quote = session.query(ManualQuote).filter_by(offer_id=item["id"], revoked=False).order_by(ManualQuote.id.desc()).first() + listings = {row.id: row for row in session.query(Listing).filter(Listing.id.in_(listing_ids))} + task_ids = {row.task_id for row in listings.values() if row.task_id} + tasks = {row.id: row for row in session.query(WatchTask).filter(WatchTask.id.in_(task_ids))} if task_ids else {} + offers = session.query(ListingOffer).filter(ListingOffer.listing_id.in_(listing_ids)).order_by(ListingOffer.id).all() + offer_ids = [row.id for row in offers] + result = {key: [] for key in listings} + if not offer_ids: + return result + def latest(model, active=False): + ids = session.query(func.max(model.id)).filter(model.offer_id.in_(offer_ids)) + if active: + ids = ids.filter(model.revoked.is_(False)) + ids = ids.group_by(model.offer_id) + return {row.offer_id: row for row in session.query(model).filter(model.id.in_(ids))} + observations = latest(OfferObservation) + evaluations = latest(OfferEvaluationRecord) + quotes = latest(ManualQuote, active=True) + for offer in offers: + listing = listings[offer.listing_id] + task = tasks.get(listing.task_id) + observation, quote, record = observations.get(offer.id), quotes.get(offer.id), evaluations.get(offer.id) + item = {"id": offer.id, "key": offer.offer_key, "name": offer.name, + "attributes": offer.attributes, "availability": offer.availability} + if observation is None: + item.update(price_state="unknown", effective_price_cents=None, source=None, + fingerprint="", reason="缺少价格证据", quote_id=None, + evaluation={"eligible": False}) + result[listing.id].append(item) + continue candidate = decode_offer(observation.payload) + item["target_match"] = match_variant(task.target_variant, candidate.attributes, listing.title) if task else "uncertain" value = None if quote is None else QuoteInput(candidate.key, quote.amount_cents, utc(quote.confirmed_at), utc(quote.expires_at), quote.evidence_fingerprint, quote.revoked) - decision = decide_price(candidate, value, now, (task.interval_minutes if task else 20) * 60) + ttl = (task.interval_minutes if task else 20) * 60 + decision = decide_price(candidate, value, now, 0 if task and not task.fetch_detail else ttl) item.update(fingerprint=observation.fingerprint, price_state=decision.state, - effective_price_cents=decision.amount_cents, source=decision.source, - fresh_until=decision.fresh_until.isoformat() if decision.fresh_until else None, - reason=decision.reason, quote_id=quote.id if quote else None) - evaluation = dict(item.get("evaluation") or {}) + effective_price_cents=decision.amount_cents, source=decision.source, + fresh_until=decision.fresh_until.isoformat() if decision.fresh_until else None, + reason=decision.reason, quote_id=quote.id if quote else None) + evaluation = dict(record.payload or {}) if record else {} valid = (task is not None and task.enabled and not listing.blocked - and decision.state == "exact" - and evaluation.get("task_version") == task.version - and evaluation.get("fingerprint") == observation.fingerprint - and evaluation.get("source") == decision.source - and evaluation.get("amount_cents") == decision.amount_cents - and (not task.min_condition_score or self.vision_enabled)) + and decision.state == "exact" and evaluation.get("task_version") == task.version + and evaluation.get("fingerprint") == observation.fingerprint + and evaluation.get("source") == decision.source + and evaluation.get("amount_cents") == decision.amount_cents + and (not task.min_condition_score or self.vision_enabled)) if not valid: evaluation.update(eligible=False, value_score=None, best_of_batch=False, satisfaction=None) item["evaluation"] = evaluation - return result + result[listing.id].append(item) + return result def present_listings(listings, factory, vision_enabled=True): @@ -119,13 +162,20 @@ def present_listings(listings, factory, vision_enabled=True): pipeline = OfferPipeline(factory, vision_enabled=vision_enabled) result = [] + snapshots = pipeline.snapshot_many([listing.id for listing in listings], datetime.now().astimezone()) for listing in listings: view = SimpleNamespace(**{col.name: getattr(listing, col.name) for col in Listing.__table__.columns}) - offers = pipeline.snapshot(listing.id, datetime.now().astimezone()) + view.image_urls = [f"/media/legacy-listing/{listing.id}/image/{i}" for i, _ in enumerate((listing.image_urls or [])[:32])] + offers = snapshots.get(listing.id, []) eligible = [o for o in offers if (o.get("evaluation") or {}).get("eligible")] exact = [o for o in offers if o["price_state"] == "exact"] + matched_exact = [o for o in exact if o.get("target_match") == "matched"] + cheapest = min(matched_exact, key=lambda o: o["effective_price_cents"], default=None) view.price_confirmed = bool(eligible) - view.effective_price_cents = min((o["effective_price_cents"] for o in exact), default=None) + view.price_state = "exact" if exact else "pending" + view.exact_offer_count = len(matched_exact) + view.effective_price_cents = cheapest["effective_price_cents"] if cheapest else None + view.effective_price_source = cheapest["source"] if cheapest else None if not eligible: view.value_score, view.best_of_batch, view.satisfaction = None, False, 0.0 result.append(view) diff --git a/goodprice/services/settings_service.py b/goodprice/services/settings_service.py index 7527c6f..d1693b5 100644 --- a/goodprice/services/settings_service.py +++ b/goodprice/services/settings_service.py @@ -1,6 +1,6 @@ from typing import Optional -from goodprice.config import Settings +from goodprice.config import SECRET_FIELDS, Settings from goodprice.models import AppSetting, SchemaVersion # 设置字段只在 goodprice.config.Settings 定义一次。运行时值与数据库覆盖都经同一个 @@ -22,7 +22,7 @@ def _merge(self, overrides: dict[str, str], legacy: bool = False) -> RuntimeSett if legacy and "media_total_limit" not in self._base.model_fields_set: values["media_total_limit"] = 4 for key, value in overrides.items(): - if key in values and value != "": + if key in values and (value != "" or key in SECRET_FIELDS): values[key] = value return Settings(_env_file=None, **values) @@ -31,15 +31,22 @@ def get(self) -> RuntimeSettings: marker = session.get(SchemaVersion, 1) return self._merge(self._overrides(session), bool(marker and marker.legacy_install)) - def set_many(self, values: dict[str, str]) -> RuntimeSettings: + def set_many(self, values: dict[str, str], *, clear_keys=frozenset()) -> RuntimeSettings: + if set(clear_keys) - SECRET_FIELDS: + raise ValueError("Only credentials can be explicitly cleared") + values = {**values, **dict.fromkeys(clear_keys, "")} with self._session_factory() as session: overrides = self._overrides(session) - overrides.update(values) + for key, value in values.items(): + if value == "" and key not in clear_keys: + overrides.pop(key, None) + else: + overrides[key] = value marker = session.get(SchemaVersion, 1) validated = self._merge(overrides, bool(marker and marker.legacy_install)) for key, value in values.items(): row = session.get(AppSetting, key) - if value == "": + if value == "" and key not in clear_keys: if row: session.delete(row) elif row: diff --git a/goodprice/web/list_context.py b/goodprice/web/list_context.py new file mode 100644 index 0000000..7f31cfc --- /dev/null +++ b/goodprice/web/list_context.py @@ -0,0 +1,64 @@ +from dataclasses import dataclass +from urllib.parse import parse_qsl, unquote, urlencode, urlsplit + +FIELDS = {"task_id", "show", "sort", "price_state", "loaded"} + + +@dataclass(frozen=True) +class ListContext: + task_id: int | None = None + show: str = "active" + sort: str = "satisfaction" + price_state: str = "all" + loaded: int = 100 + + @classmethod + def from_query(cls, query): + def single(name, default): + values = query.getlist(name) if hasattr(query, "getlist") else [query.get(name, default)] + return str(values[0])[:100] if len(values) == 1 else default + raw_id = single("task_id", "") + task_id = int(raw_id) if raw_id.isascii() and raw_id.isdigit() and 0 < int(raw_id) <= 2**31-1 else None + raw_loaded = single("loaded", "100") + loaded = min(1000, max(100, int(raw_loaded))) if raw_loaded.isascii() and raw_loaded.isdigit() else 100 + def choice(name, options, default): + raw = single(name, default) + return raw if raw in options else default + return cls(task_id, choice("show", {"active", "gone", "blocked", "all", "unseen"}, "active"), + choice("sort", {"satisfaction", "price_asc", "price_desc", "newest"}, "satisfaction"), + choice("price_state", {"all", "exact", "pending"}, "all"), loaded) + + def values(self): + values = {"show": self.show, "sort": self.sort} + if self.task_id: + values = {"task_id": str(self.task_id), **values} + if self.price_state != "all": + values["price_state"] = self.price_state + if self.loaded != 100: + values["loaded"] = str(self.loaded) + return values + + def url(self, path="/listings"): + return path + "?" + urlencode(self.values()) + + +def safe_return_to(raw, fallback="/listings"): + if not raw or len(raw) > 2048: + return fallback + decoded = raw + for _ in range(3): + if any(ord(c) < 32 for c in decoded) or "\\" in decoded: + return fallback + decoded = unquote(decoded) + try: + parts = urlsplit(raw) + except ValueError: + return fallback + if parts.scheme or parts.netloc or parts.path != "/listings" or parts.fragment: + return fallback + pairs = parse_qsl(parts.query, keep_blank_values=True) + if any(k not in FIELDS for k, v in pairs) or len({k for k, v in pairs}) != len(pairs): + return fallback + context = ListContext.from_query(dict(pairs)) + canonical = context.values() + return "/listings" + ("?" + urlencode({k: canonical[k] for k, v in pairs if k in canonical}) if pairs else "") diff --git a/goodprice/web/media_routes.py b/goodprice/web/media_routes.py index bf4127f..16db47c 100644 --- a/goodprice/web/media_routes.py +++ b/goodprice/web/media_routes.py @@ -1,3 +1,4 @@ +import hashlib import shutil from fastapi import APIRouter, HTTPException, Request @@ -6,7 +7,7 @@ from goodprice.config import Settings from goodprice.media.budget import MediaBudget -from goodprice.models import MediaAsset +from goodprice.models import Listing, MediaAsset router = APIRouter() MEDIA_FIELDS = {name for name in Settings.model_fields if name.startswith("media_")} | {"video_analysis_enabled"} @@ -98,6 +99,33 @@ def media_file(asset_id: int, request: Request): raise HTTPException(404, "证据文件不可用或已清理") +@router.get("/media/legacy-listing/{listing_id}/image/{index}") +def legacy_listing_image(listing_id: int, index: int, request: Request): + # An internal listing/index reference migrates a legacy image lazily on view; + # neither pages nor APIs publish expiring source URLs or take caller URLs. + with request.app.state.session_factory() as session: + listing = session.get(Listing, listing_id) + if listing is None or not 0 <= index < min(32, len(listing.image_urls or [])): + raise HTTPException(404, "图片不存在") + source = listing.image_urls[index] + key = "legacy:" + hashlib.sha256(source.encode()).hexdigest()[:32] + row = session.query(MediaAsset).filter(MediaAsset.listing_id == listing_id, + MediaAsset.source_url == source, MediaAsset.kind == "image", + MediaAsset.status != "superseded").order_by(MediaAsset.id.desc()).first() + if row is None: + row = session.query(MediaAsset).filter_by(listing_id=listing_id, key=key).first() + if row: + row.status, row.relative_path, row.content_hash = "pending", None, None + session.commit() + if row is None: + row = MediaAsset(listing_id=listing_id, key=key, kind="image", source_url=source, + source_locator=f"legacy-image:{index}") + session.add(row) + session.commit() + asset_id = row.id + return media_file(asset_id, request) + + @router.post("/settings/media") async def save_media(request: Request): form = await request.form() diff --git a/goodprice/web/offer_routes.py b/goodprice/web/offer_routes.py index 31ada68..970853b 100644 --- a/goodprice/web/offer_routes.py +++ b/goodprice/web/offer_routes.py @@ -1,4 +1,5 @@ from datetime import datetime, timedelta, timezone +from urllib.parse import urlencode from fastapi import APIRouter, HTTPException, Request from fastapi.responses import RedirectResponse @@ -9,6 +10,7 @@ from goodprice.offers.service import OfferService from goodprice.offers.types import QuoteInput from goodprice.services.offer_pipeline import OfferPipeline +from goodprice.web.list_context import safe_return_to router = APIRouter() @@ -45,7 +47,7 @@ async def confirm_quote(request: Request, offer_id: int): raise HTTPException(422, "报价无效或价格证据已变化,请刷新后确认") from exc if json_request: return {"id": row_id} - return RedirectResponse(f"/listings/{listing_id}", status_code=303) + return RedirectResponse(f"/listings/{listing_id}?" + urlencode({"return_to": safe_return_to(request.query_params.get("return_to"))}), status_code=303) @router.post("/quotes/{quote_id}/revoke") @@ -61,4 +63,4 @@ async def revoke_quote(request: Request, quote_id: int): OfferService(factory).revoke_quote(quote_id) if "application/json" in request.headers.get("content-type", ""): return {"revoked": True} - return RedirectResponse(f"/listings/{listing_id}", status_code=303) + return RedirectResponse(f"/listings/{listing_id}?" + urlencode({"return_to": safe_return_to(request.query_params.get("return_to"))}), status_code=303) diff --git a/goodprice/web/resource_routes.py b/goodprice/web/resource_routes.py new file mode 100644 index 0000000..f88e1bb --- /dev/null +++ b/goodprice/web/resource_routes.py @@ -0,0 +1,44 @@ +from dataclasses import asdict + +from fastapi import APIRouter, HTTPException, Request +from fastapi.responses import RedirectResponse + +from goodprice.execution.limits import ResourceLimits +from goodprice.resources import detect_resources, recommend_profile, resource_profile + +router = APIRouter() +RESOURCE_LABELS = { + "execution_active_runs": "同时推进的任务数(1–4)", + "execution_crawler_workers": "抓取工作线程(1–2)", + "execution_model_workers": "模型请求并发(1–8)", + "execution_media_workers": "媒体处理并发(1–2)", + "execution_platform_interval_seconds": "平台导航最小间隔(5–300秒)", + "execution_model_rpm": "模型 HTTP 请求上限(1–60次/分钟)", + "execution_global_items": "全局未完成步骤(1–200)", + "execution_task_items": "单任务未完成步骤(1–60)", + "execution_run_details": "每轮详情预算(1–100)", + "execution_run_http_attempts": "每轮模型 HTTP 尝试(1–200)", + "execution_run_minutes": "每轮墙钟预算(5–60分钟)", +} + + +@router.get("/api/resources/recommendation") +def resource_recommendation(): + snapshot = detect_resources() + return {"snapshot": asdict(snapshot), "recommendation": recommend_profile(snapshot)} + + +@router.post("/settings/resources/apply-profile") +async def apply_profile(request: Request): + form = await request.form() + if form.get("confirm") != "1" or set(form) - {"name", "confirm"}: + raise HTTPException(422, "请明确确认应用资源配置") + try: + values = resource_profile(form.get("name", "")) + except ValueError: + raise HTTPException(422, "未知资源配置") from None + runtime = request.app.state.settings_service.set_many({k: str(v) for k, v in values.items()}) + coordinator = getattr(request.app.state, "coordinator", None) + if coordinator: + coordinator.update_limits(ResourceLimits.from_settings(runtime)) + return RedirectResponse("/settings#resources", status_code=303) diff --git a/goodprice/web/routes.py b/goodprice/web/routes.py index 4bdcbaf..917bd8e 100644 --- a/goodprice/web/routes.py +++ b/goodprice/web/routes.py @@ -11,6 +11,7 @@ from sqlalchemy import and_, case, func, select, text from goodprice.constants import SCORE_MAX +from goodprice.web.list_context import ListContext, safe_return_to from goodprice.web.media_routes import MEDIA_LABELS, capabilities, public_assets logger = logging.getLogger(__name__) @@ -23,17 +24,17 @@ def _optional_int(value: str) -> Optional[int]: """把查询参数安全转成 int;非数字返回 None,避免非法输入导致 500。""" - return int(value) if str(value).isdigit() else None + return int(value) if str(value).isascii() and str(value).isdigit() else None -def _apply_listing_filters(query, *, task_id, show, sort): +def _apply_listing_filters(query, *, task_id, show, sort, price_state="all"): """列表页 / 加载更多 / API 共用的过滤与排序,避免三处 if/elif 语义分叉。 show 的取值由模板下拉决定;未知值(含 "all")不追加过滤条件。 """ from datetime import datetime, timezone - from goodprice.models import Listing, WatchTask + from goodprice.models import Listing, ListingOffer, WatchTask if task_id: query = query.filter(Listing.task_id == task_id) @@ -43,7 +44,14 @@ def _apply_listing_filters(query, *, task_id, show, sort): query = query.filter(Listing.status == "gone") elif show == "blocked": query = query.filter(Listing.blocked.is_(True)) + elif show == "unseen": + query = query.filter(Listing.status == "unseen") version = select(WatchTask.version).where(WatchTask.id == Listing.task_id).correlate(Listing).scalar_subquery() + if price_state in {"exact", "pending"}: + exact = select(ListingOffer.id).where(ListingOffer.listing_id == Listing.id, + ListingOffer.effective_price_state == "exact", ListingOffer.price_valid_until > datetime.now(timezone.utc), + ListingOffer.price_task_version == version).correlate(Listing).exists() + query = query.filter(exact if price_state == "exact" else ~exact) valid = and_(Listing.offer_verified_until > datetime.now(timezone.utc), Listing.offer_price_cents.is_not(None), Listing.offer_task_version == version) query = query.order_by(case((valid, 0), else_=1)) @@ -146,10 +154,16 @@ def dashboard(request: Request): return templates.TemplateResponse( request, "dashboard.html", - {"stats": stats, "recent": recent, "active": "dashboard"}, + {"stats": stats, "recent": _present_recent(request, recent), "active": "dashboard"}, ) +def _present_recent(request, rows): + from goodprice.services.offer_pipeline import present_listings + return present_listings(rows, request.app.state.session_factory, + request.app.state.settings_service.get().vision_enabled) + + @router.get("/tasks", response_class=HTMLResponse) def tasks_page(request: Request): task_service, _ = _services(request) @@ -422,9 +436,15 @@ def toggle_task(request: Request, task_id: int): @router.post("/tasks/{task_id}/run") def run_task(request: Request, task_id: int): - threading.Thread( - target=request.app.state.run_job, args=(task_id,), daemon=True - ).start() + from fastapi import HTTPException + task = request.app.state.task_service.get_task(task_id) + if task is None: + raise HTTPException(404, "任务不存在") + if not task.enabled: + raise HTTPException(409, "任务已停用,请先启用后执行") + queued = request.app.state.run_job(task_id) + if queued is False: + return RedirectResponse("/tasks?" + urlencode({"toast": "任务已在队列中,无需重复提交"}), status_code=303) return RedirectResponse(f"/tasks?run={task_id}", status_code=303) @@ -445,27 +465,31 @@ def listings_page( sort: str = Query("satisfaction"), show: str = Query("active"), ): + context = ListContext.from_query(request.query_params) + task_id, sort, show = context.task_id, context.sort, context.show + offset = min(10000, max(0, offset)) with request.app.state.session_factory() as session: from goodprice.models import Listing, WatchTask query = session.query(Listing) tasks = session.query(WatchTask).order_by(WatchTask.id).all() - task_id_int = _optional_int(task_id) - query = _apply_listing_filters(query, task_id=task_id_int, show=show, sort=sort) + task_id_int = task_id + query = _apply_listing_filters(query, task_id=task_id_int, show=show, sort=sort, price_state=context.price_state) query = query.offset(offset) - listings = query.limit(PAGE_SIZE).all() + listings = query.limit(context.loaded).all() from goodprice.services.offer_pipeline import present_listings listings = present_listings(listings, request.app.state.session_factory, request.app.state.settings_service.get().vision_enabled) notify_counts = _notify_counts(session, listings) - more = len(listings) >= PAGE_SIZE - partial_url = f"/listings?partial=1&task_id={task_id or ''}&sort={sort}&show={show}" + more = len(listings) >= context.loaded and context.loaded < 1000 + partial_url = context.url() + "&partial=1" + return_to = context.url() if partial: return templates.TemplateResponse( request, "listings_grid.html", - {"listings": listings, "notify_counts": notify_counts}, + {"listings": listings, "notify_counts": notify_counts, "return_to": return_to}, ) running_ids = request.app.state.guard.running_ids() return templates.TemplateResponse( @@ -478,13 +502,13 @@ def listings_page( "task_id": task_id, "sort": sort, "show": show, + "price_state": context.price_state, "partial_url": partial_url, "notify_counts": notify_counts, "more": more, - "load_more_url": ( - f"/listings/more?offset={offset + len(listings)}" - f"&task_id={task_id or ''}&sort={sort}&show={show}" - ), + "return_to": return_to, + "load_more_url": context.url("/listings/more") + f"&offset={offset + len(listings)}", + "load_more_fallback": ListContext(context.task_id, show, sort, context.price_state, min(1000, context.loaded + PAGE_SIZE)).url(), "active": "listings", }, ) @@ -515,21 +539,30 @@ def listings_more( sort: str = Query("satisfaction"), show: str = Query("active"), ): + context = ListContext.from_query(request.query_params) + task_id, sort, show = context.task_id, context.sort, context.show + offset = min(1000, max(0, offset)) with request.app.state.session_factory() as session: from goodprice.models import Listing query = session.query(Listing) - task_id_int = _optional_int(task_id) - query = _apply_listing_filters(query, task_id=task_id_int, show=show, sort=sort) - listings = query.offset(offset).limit(PAGE_SIZE).all() + task_id_int = task_id + query = _apply_listing_filters(query, task_id=task_id_int, show=show, sort=sort, price_state=context.price_state) + listings = query.offset(offset).limit(min(PAGE_SIZE, 1000-offset)).all() from goodprice.services.offer_pipeline import present_listings listings = present_listings(listings, request.app.state.session_factory, request.app.state.settings_service.get().vision_enabled) notify_counts = _notify_counts(session, listings) - return templates.TemplateResponse( - request, "listings_cards.html", {"listings": listings, "notify_counts": notify_counts} - ) + loaded = min(1000, max(100, offset + len(listings))) + current = ListContext(task_id, show, sort, context.price_state, loaded) + response = templates.TemplateResponse(request, "listings_more.html", { + "listings": listings, "notify_counts": notify_counts, "return_to": current.url(), + "more": len(listings) == PAGE_SIZE and loaded < 1000, + "load_more_url": current.url("/listings/more") + f"&offset={loaded}", + "load_more_fallback": ListContext(task_id, show, sort, context.price_state, min(1000, loaded + PAGE_SIZE)).url()}) + response.headers["HX-Replace-Url"] = current.url() + return response @router.get("/listings/{listing_id}", response_class=HTMLResponse) @@ -579,6 +612,7 @@ def listing_detail_page(request: Request, listing_id: int): "drop_pct_threshold": DROP_PCT_DISPLAY_THRESHOLD, "offers": OfferPipeline(request.app.state.session_factory).snapshot(listing_id, dt.now(timezone.utc)), "media_assets": public_assets(request, listing_id), + "return_to": safe_return_to(request.query_params.get("return_to"), "/listings"), "active": "listings", }, ) @@ -589,7 +623,8 @@ def reanalyze_listing(request: Request, listing_id: int): threading.Thread( target=request.app.state.run_reanalyze, args=(listing_id,), daemon=True ).start() - return RedirectResponse(f"/listings/{listing_id}?toast=已提交重新分析", status_code=303) + return RedirectResponse(f"/listings/{listing_id}?" + urlencode({"toast": "已提交重新分析", + "return_to": safe_return_to(request.query_params.get("return_to"), "/listings")}), status_code=303) @router.post("/listings/delete-batch") @@ -640,12 +675,18 @@ def unblock_seller(request: Request, listing_id: int): def _listings_back(request: Request, toast: str) -> str: - params = {} - ref = request.headers.get("referer", "") - if "?" in ref: - params = { - k: v for k, v in parse_qsl(ref.split("?", 1)[1]) if k in ("task_id", "sort", "show") - } + raw = request.query_params.get("return_to") + if raw is None: + # Legacy same-origin forms: reduce Referer to a whitelisted relative path. + from urllib.parse import urlsplit + try: + ref = urlsplit(request.headers.get("referer", "")) + except ValueError: + ref = urlsplit("") + if (not ref.netloc and not ref.scheme or ref.netloc == request.url.netloc and ref.scheme == request.url.scheme) and ref.path == "/listings": + raw = ref.path + ("?" + ref.query if ref.query else "") + target = safe_return_to(raw, "/listings") + params = dict(parse_qsl(target.split("?", 1)[1])) if "?" in target else {} params["toast"] = toast return "/listings?" + urlencode(params) @@ -678,6 +719,13 @@ def _set_blocked(request: Request, listing_id: int, blocked: bool, seller: bool) @router.get("/settings", response_class=HTMLResponse) def settings_page(request: Request): + return render_settings(request) + + +def render_settings(request, overrides=None, error=None, status_code=200): + from goodprice.resources import detect_resources, recommend_profile + from goodprice.web.resource_routes import RESOURCE_LABELS + resource_snapshot = detect_resources() _, settings_service = _services(request) settings = settings_service.get() login_session = getattr(request.app.state, "login_session", None) @@ -688,7 +736,11 @@ def settings_page(request: Request): request, "settings.html", { - "settings": settings, + "settings": settings.model_dump() | (overrides or {}), + "settings_error": error, + "resource_snapshot": resource_snapshot, + "resource_recommendation": recommend_profile(resource_snapshot), + "resource_labels": RESOURCE_LABELS, "media_capabilities": capabilities(settings), "media_fields": [name for name in type(settings).model_fields if name.startswith("media_")], "media_labels": MEDIA_LABELS, @@ -696,7 +748,7 @@ def settings_page(request: Request): "login_message": login_message, **_login_context(login_session, settings.novnc_port), "active": "settings", - }, + }, status_code=status_code, ) @@ -1020,13 +1072,14 @@ def api_list_listings( task_id: str = Query(""), sort: str = Query("satisfaction"), show: str = Query("active"), + price_state: str = Query("all"), ): with request.app.state.session_factory() as session: from goodprice.models import Listing query = session.query(Listing) task_id_int = _optional_int(task_id) - query = _apply_listing_filters(query, task_id=task_id_int, show=show, sort=sort) + query = _apply_listing_filters(query, task_id=task_id_int, show=show, sort=sort, price_state=price_state) rows = query.offset(offset).limit(PAGE_SIZE).all() from goodprice.services.offer_pipeline import present_listings @@ -1040,6 +1093,7 @@ def api_list_listings( "title": row.title, "price": row.price, "effective_price_cents": row.effective_price_cents, + "price_state": row.price_state, "price_confirmed": row.price_confirmed, "value_score": row.value_score, "url": row.url, diff --git a/goodprice/web/settings_sections.py b/goodprice/web/settings_sections.py new file mode 100644 index 0000000..28f71d8 --- /dev/null +++ b/goodprice/web/settings_sections.py @@ -0,0 +1,54 @@ +from fastapi import APIRouter, Request +from fastapi.responses import RedirectResponse +from pydantic import ValidationError + +from goodprice.config import SECRET_FIELDS, Settings + +router = APIRouter() +SECTIONS = { + "account": frozenset({"xianyu_cookie", "proxy", "default_crawl_interval_minutes", "default_crawl_jitter_minutes"}), + "models": frozenset({"llm_base_url", "llm_api_key", "llm_model", "llm_api_format", "vision_base_url", "vision_api_key", + "vision_model", "vision_api_format", "vision_enabled", "jev_enabled", "jev_auto_threshold", "jev_backend", "jev_api_key"}), + "notifications": frozenset({"serverchan_sendkey", "serverchan_enabled", "wecom_webhook", "wecom_robot_enabled", + "feishu_webhook", "feishu_secret", "feishu_enabled", "gotify_url", "gotify_token", "gotify_priority", "gotify_enabled"}), + "deployment_readonly": frozenset(), + "resources": frozenset({name for name in Settings.model_fields if name.startswith("execution_")}), +} + + +def save_section(service, section, form): + fields = SECTIONS.get(section) + if not fields: + raise ValueError("设置分类不可修改") + extras = {f"clear_{name}" for name in fields & SECRET_FIELDS} | {"confirm_clear", "confirm_resources"} + if set(form) - fields - extras: + raise ValueError("分类包含不允许修改的字段") + clear = {name for name in fields & SECRET_FIELDS if form.get(f"clear_{name}") == "1"} + if clear and form.get("confirm_clear") != "1": + raise ValueError("清除凭据需要明确确认") + values = {k: str(v) for k, v in form.items() if k in fields and (k not in SECRET_FIELDS or str(v) != "")} + if section == "resources": + before = service.get() + proposed = Settings(_env_file=None, **(before.model_dump() | values)) + increase = any(getattr(proposed, k) > getattr(before, k) for k in fields + if k != "execution_platform_interval_seconds") + increase |= proposed.execution_platform_interval_seconds < before.execution_platform_interval_seconds + if increase and form.get("confirm_resources") != "1": + raise ValueError("提高资源或访问频率需确认代价") + for name in fields: + if name.endswith("_enabled"): + values[name] = "1" if str(form.get(name, "")).lower() in {"1", "true", "on", "yes"} else "0" + return service.set_many(values, clear_keys=clear) + + +@router.post("/settings/sections/{section}") +async def save_settings_section(request: Request, section: str): + from goodprice.web.routes import render_settings + + form = await request.form() + try: + save_section(request.app.state.settings_service, section, form) + except (ValueError, ValidationError): + public = {k: str(v) for k, v in form.items() if k in SECTIONS.get(section, ()) and k not in SECRET_FIELDS} + return render_settings(request, overrides=public, error="设置值无效,未保存;请检查范围和分类字段。", status_code=422) + return RedirectResponse(f"/settings#{section}", status_code=303) diff --git a/goodprice/web/static/THIRD_PARTY.md b/goodprice/web/static/THIRD_PARTY.md new file mode 100644 index 0000000..8fe680e --- /dev/null +++ b/goodprice/web/static/THIRD_PARTY.md @@ -0,0 +1,14 @@ +# Bundled browser assets + +- HTMX 1.9.12: https://github.com/bigskysoftware/htmx/tree/v1.9.12 + `htmx.min.js` SHA-256: `449317ade7881e949510db614991e195c3a099c4c791c24dacec55f9f4a2a452`. + License: `htmx.LICENSE` (Zero-Clause BSD). +- Tailwind CSS 3.4.17: https://github.com/tailwindlabs/tailwindcss/tree/v3.4.17 + License: `tailwind.LICENSE` (MIT). `app.css` is generated from the existing templates. + +Normal application startup needs neither Node.js nor a CDN connection. Maintainers +changing utility classes can regenerate the committed stylesheet with: + +```sh +npx --yes tailwindcss@3.4.17 -i ./goodprice/web/static/tailwind.input.css -o ./goodprice/web/static/app.css --content './goodprice/web/templates/**/*.html' --minify +``` diff --git a/goodprice/web/static/app.css b/goodprice/web/static/app.css new file mode 100644 index 0000000..bc141e7 --- /dev/null +++ b/goodprice/web/static/app.css @@ -0,0 +1 @@ +*,:after,:before{--tw-border-spacing-x:0;--tw-border-spacing-y:0;--tw-translate-x:0;--tw-translate-y:0;--tw-rotate:0;--tw-skew-x:0;--tw-skew-y:0;--tw-scale-x:1;--tw-scale-y:1;--tw-pan-x: ;--tw-pan-y: ;--tw-pinch-zoom: ;--tw-scroll-snap-strictness:proximity;--tw-gradient-from-position: ;--tw-gradient-via-position: ;--tw-gradient-to-position: ;--tw-ordinal: ;--tw-slashed-zero: ;--tw-numeric-figure: ;--tw-numeric-spacing: ;--tw-numeric-fraction: ;--tw-ring-inset: ;--tw-ring-offset-width:0px;--tw-ring-offset-color:#fff;--tw-ring-color:rgba(59,130,246,.5);--tw-ring-offset-shadow:0 0 #0000;--tw-ring-shadow:0 0 #0000;--tw-shadow:0 0 #0000;--tw-shadow-colored:0 0 #0000;--tw-blur: ;--tw-brightness: ;--tw-contrast: ;--tw-grayscale: ;--tw-hue-rotate: ;--tw-invert: ;--tw-saturate: ;--tw-sepia: ;--tw-drop-shadow: ;--tw-backdrop-blur: ;--tw-backdrop-brightness: ;--tw-backdrop-contrast: ;--tw-backdrop-grayscale: ;--tw-backdrop-hue-rotate: ;--tw-backdrop-invert: ;--tw-backdrop-opacity: ;--tw-backdrop-saturate: ;--tw-backdrop-sepia: ;--tw-contain-size: ;--tw-contain-layout: ;--tw-contain-paint: ;--tw-contain-style: }::backdrop{--tw-border-spacing-x:0;--tw-border-spacing-y:0;--tw-translate-x:0;--tw-translate-y:0;--tw-rotate:0;--tw-skew-x:0;--tw-skew-y:0;--tw-scale-x:1;--tw-scale-y:1;--tw-pan-x: ;--tw-pan-y: ;--tw-pinch-zoom: ;--tw-scroll-snap-strictness:proximity;--tw-gradient-from-position: ;--tw-gradient-via-position: ;--tw-gradient-to-position: ;--tw-ordinal: ;--tw-slashed-zero: ;--tw-numeric-figure: ;--tw-numeric-spacing: ;--tw-numeric-fraction: ;--tw-ring-inset: ;--tw-ring-offset-width:0px;--tw-ring-offset-color:#fff;--tw-ring-color:rgba(59,130,246,.5);--tw-ring-offset-shadow:0 0 #0000;--tw-ring-shadow:0 0 #0000;--tw-shadow:0 0 #0000;--tw-shadow-colored:0 0 #0000;--tw-blur: ;--tw-brightness: ;--tw-contrast: ;--tw-grayscale: ;--tw-hue-rotate: ;--tw-invert: ;--tw-saturate: ;--tw-sepia: ;--tw-drop-shadow: ;--tw-backdrop-blur: ;--tw-backdrop-brightness: ;--tw-backdrop-contrast: ;--tw-backdrop-grayscale: ;--tw-backdrop-hue-rotate: ;--tw-backdrop-invert: ;--tw-backdrop-opacity: ;--tw-backdrop-saturate: ;--tw-backdrop-sepia: ;--tw-contain-size: ;--tw-contain-layout: ;--tw-contain-paint: ;--tw-contain-style: }/*! tailwindcss v3.4.17 | MIT License | https://tailwindcss.com*/*,:after,:before{box-sizing:border-box;border:0 solid #e5e7eb}:after,:before{--tw-content:""}:host,html{line-height:1.5;-webkit-text-size-adjust:100%;-moz-tab-size:4;-o-tab-size:4;tab-size:4;font-family:ui-sans-serif,system-ui,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji;font-feature-settings:normal;font-variation-settings:normal;-webkit-tap-highlight-color:transparent}body{margin:0;line-height:inherit}hr{height:0;color:inherit;border-top-width:1px}abbr:where([title]){-webkit-text-decoration:underline dotted;text-decoration:underline dotted}h1,h2,h3,h4,h5,h6{font-size:inherit;font-weight:inherit}a{color:inherit;text-decoration:inherit}b,strong{font-weight:bolder}code,kbd,pre,samp{font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace;font-feature-settings:normal;font-variation-settings:normal;font-size:1em}small{font-size:80%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sub{bottom:-.25em}sup{top:-.5em}table{text-indent:0;border-color:inherit;border-collapse:collapse}button,input,optgroup,select,textarea{font-family:inherit;font-feature-settings:inherit;font-variation-settings:inherit;font-size:100%;font-weight:inherit;line-height:inherit;letter-spacing:inherit;color:inherit;margin:0;padding:0}button,select{text-transform:none}button,input:where([type=button]),input:where([type=reset]),input:where([type=submit]){-webkit-appearance:button;background-color:transparent;background-image:none}:-moz-focusring{outline:auto}:-moz-ui-invalid{box-shadow:none}progress{vertical-align:baseline}::-webkit-inner-spin-button,::-webkit-outer-spin-button{height:auto}[type=search]{-webkit-appearance:textfield;outline-offset:-2px}::-webkit-search-decoration{-webkit-appearance:none}::-webkit-file-upload-button{-webkit-appearance:button;font:inherit}summary{display:list-item}blockquote,dd,dl,figure,h1,h2,h3,h4,h5,h6,hr,p,pre{margin:0}fieldset{margin:0}fieldset,legend{padding:0}menu,ol,ul{list-style:none;margin:0;padding:0}dialog{padding:0}textarea{resize:vertical}input::-moz-placeholder,textarea::-moz-placeholder{opacity:1;color:#9ca3af}input::placeholder,textarea::placeholder{opacity:1;color:#9ca3af}[role=button],button{cursor:pointer}:disabled{cursor:default}audio,canvas,embed,iframe,img,object,svg,video{display:block;vertical-align:middle}img,video{max-width:100%;height:auto}[hidden]:where(:not([hidden=until-found])){display:none}.fixed{position:fixed}.sticky{position:sticky}.inset-0{inset:0}.bottom-4{bottom:1rem}.right-4{right:1rem}.top-4{top:1rem}.z-10{z-index:10}.z-50{z-index:50}.col-span-full{grid-column:1/-1}.mx-auto{margin-left:auto;margin-right:auto}.mb-1{margin-bottom:.25rem}.mb-1\.5{margin-bottom:.375rem}.mb-2{margin-bottom:.5rem}.mb-3{margin-bottom:.75rem}.mb-4{margin-bottom:1rem}.mb-5{margin-bottom:1.25rem}.mb-6{margin-bottom:1.5rem}.mt-0\.5{margin-top:.125rem}.mt-1{margin-top:.25rem}.mt-1\.5{margin-top:.375rem}.mt-2{margin-top:.5rem}.mt-3{margin-top:.75rem}.mt-4{margin-top:1rem}.mt-5{margin-top:1.25rem}.mt-6{margin-top:1.5rem}.block{display:block}.inline{display:inline}.flex{display:flex}.inline-flex{display:inline-flex}.table{display:table}.grid{display:grid}.hidden{display:none}.aspect-\[3\/4\]{aspect-ratio:3/4}.aspect-square{aspect-ratio:1/1}.h-14{height:3.5rem}.h-full{height:100%}.max-h-full{max-height:100%}.w-14{width:3.5rem}.w-16{width:4rem}.w-20{width:5rem}.w-full{width:100%}.min-w-0{min-width:0}.max-w-4xl{max-width:56rem}.max-w-6xl{max-width:72rem}.max-w-full{max-width:100%}.flex-1{flex:1 1 0%}.basis-full{flex-basis:100%}.transform{transform:translate(var(--tw-translate-x),var(--tw-translate-y)) rotate(var(--tw-rotate)) skewX(var(--tw-skew-x)) skewY(var(--tw-skew-y)) scaleX(var(--tw-scale-x)) scaleY(var(--tw-scale-y))}.cursor-pointer{cursor:pointer}.cursor-zoom-in{cursor:zoom-in}.cursor-zoom-out{cursor:zoom-out}.select-none{-webkit-user-select:none;-moz-user-select:none;user-select:none}.select-all{-webkit-user-select:all;-moz-user-select:all;user-select:all}.resize-y{resize:vertical}.resize{resize:both}.grid-cols-1{grid-template-columns:repeat(1,minmax(0,1fr))}.grid-cols-2{grid-template-columns:repeat(2,minmax(0,1fr))}.flex-col{flex-direction:column}.flex-wrap{flex-wrap:wrap}.items-center{align-items:center}.justify-center{justify-content:center}.justify-between{justify-content:space-between}.gap-1{gap:.25rem}.gap-1\.5{gap:.375rem}.gap-2{gap:.5rem}.gap-3{gap:.75rem}.gap-4{gap:1rem}.gap-5{gap:1.25rem}.gap-6{gap:1.5rem}.space-y-2>:not([hidden])~:not([hidden]){--tw-space-y-reverse:0;margin-top:calc(.5rem*(1 - var(--tw-space-y-reverse)));margin-bottom:calc(.5rem*var(--tw-space-y-reverse))}.space-y-3>:not([hidden])~:not([hidden]){--tw-space-y-reverse:0;margin-top:calc(.75rem*(1 - var(--tw-space-y-reverse)));margin-bottom:calc(.75rem*var(--tw-space-y-reverse))}.space-y-4>:not([hidden])~:not([hidden]){--tw-space-y-reverse:0;margin-top:calc(1rem*(1 - var(--tw-space-y-reverse)));margin-bottom:calc(1rem*var(--tw-space-y-reverse))}.space-y-6>:not([hidden])~:not([hidden]){--tw-space-y-reverse:0;margin-top:calc(1.5rem*(1 - var(--tw-space-y-reverse)));margin-bottom:calc(1.5rem*var(--tw-space-y-reverse))}.divide-y>:not([hidden])~:not([hidden]){--tw-divide-y-reverse:0;border-top-width:calc(1px*(1 - var(--tw-divide-y-reverse)));border-bottom-width:calc(1px*var(--tw-divide-y-reverse))}.overflow-hidden{overflow:hidden}.overflow-x-auto{overflow-x:auto}.truncate{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.whitespace-pre-wrap{white-space:pre-wrap}.break-words{overflow-wrap:break-word}.break-all{word-break:break-all}.rounded{border-radius:.25rem}.rounded-full{border-radius:9999px}.rounded-lg{border-radius:.5rem}.rounded-md{border-radius:.375rem}.border{border-width:1px}.border-b{border-bottom-width:1px}.border-t{border-top-width:1px}.border-amber-200{--tw-border-opacity:1;border-color:rgb(253 230 138/var(--tw-border-opacity,1))}.border-blue-200{--tw-border-opacity:1;border-color:rgb(191 219 254/var(--tw-border-opacity,1))}.border-gray-100{--tw-border-opacity:1;border-color:rgb(243 244 246/var(--tw-border-opacity,1))}.border-gray-200{--tw-border-opacity:1;border-color:rgb(229 231 235/var(--tw-border-opacity,1))}.border-green-200{--tw-border-opacity:1;border-color:rgb(187 247 208/var(--tw-border-opacity,1))}.border-purple-100{--tw-border-opacity:1;border-color:rgb(243 232 255/var(--tw-border-opacity,1))}.border-red-300{--tw-border-opacity:1;border-color:rgb(252 165 165/var(--tw-border-opacity,1))}.border-yellow-200{--tw-border-opacity:1;border-color:rgb(254 240 138/var(--tw-border-opacity,1))}.bg-amber-100{--tw-bg-opacity:1;background-color:rgb(254 243 199/var(--tw-bg-opacity,1))}.bg-black\/80{background-color:rgba(0,0,0,.8)}.bg-blue-100{--tw-bg-opacity:1;background-color:rgb(219 234 254/var(--tw-bg-opacity,1))}.bg-blue-50{--tw-bg-opacity:1;background-color:rgb(239 246 255/var(--tw-bg-opacity,1))}.bg-gray-100{--tw-bg-opacity:1;background-color:rgb(243 244 246/var(--tw-bg-opacity,1))}.bg-gray-200{--tw-bg-opacity:1;background-color:rgb(229 231 235/var(--tw-bg-opacity,1))}.bg-gray-300{--tw-bg-opacity:1;background-color:rgb(209 213 219/var(--tw-bg-opacity,1))}.bg-gray-50{--tw-bg-opacity:1;background-color:rgb(249 250 251/var(--tw-bg-opacity,1))}.bg-gray-50\/80{background-color:rgba(249,250,251,.8)}.bg-green-100{--tw-bg-opacity:1;background-color:rgb(220 252 231/var(--tw-bg-opacity,1))}.bg-green-50{--tw-bg-opacity:1;background-color:rgb(240 253 244/var(--tw-bg-opacity,1))}.bg-green-700{--tw-bg-opacity:1;background-color:rgb(21 128 61/var(--tw-bg-opacity,1))}.bg-indigo-100{--tw-bg-opacity:1;background-color:rgb(224 231 255/var(--tw-bg-opacity,1))}.bg-purple-100{--tw-bg-opacity:1;background-color:rgb(243 232 255/var(--tw-bg-opacity,1))}.bg-purple-50\/40{background-color:rgba(250,245,255,.4)}.bg-red-100{--tw-bg-opacity:1;background-color:rgb(254 226 226/var(--tw-bg-opacity,1))}.bg-red-50{--tw-bg-opacity:1;background-color:rgb(254 242 242/var(--tw-bg-opacity,1))}.bg-white{--tw-bg-opacity:1;background-color:rgb(255 255 255/var(--tw-bg-opacity,1))}.bg-white\/95{background-color:hsla(0,0%,100%,.95)}.bg-yellow-100{--tw-bg-opacity:1;background-color:rgb(254 249 195/var(--tw-bg-opacity,1))}.bg-yellow-50{--tw-bg-opacity:1;background-color:rgb(254 252 232/var(--tw-bg-opacity,1))}.object-contain{-o-object-fit:contain;object-fit:contain}.object-cover{-o-object-fit:cover;object-fit:cover}.p-2{padding:.5rem}.p-3{padding:.75rem}.p-3\.5{padding:.875rem}.p-4{padding:1rem}.p-5{padding:1.25rem}.px-1\.5{padding-left:.375rem;padding-right:.375rem}.px-2{padding-left:.5rem;padding-right:.5rem}.px-2\.5{padding-left:.625rem;padding-right:.625rem}.px-3{padding-left:.75rem;padding-right:.75rem}.px-4{padding-left:1rem;padding-right:1rem}.px-6{padding-left:1.5rem;padding-right:1.5rem}.py-0\.5{padding-top:.125rem;padding-bottom:.125rem}.py-1{padding-top:.25rem;padding-bottom:.25rem}.py-1\.5{padding-top:.375rem;padding-bottom:.375rem}.py-2{padding-top:.5rem;padding-bottom:.5rem}.py-2\.5{padding-top:.625rem;padding-bottom:.625rem}.py-3{padding-top:.75rem;padding-bottom:.75rem}.py-6{padding-top:1.5rem;padding-bottom:1.5rem}.pb-3{padding-bottom:.75rem}.pt-3{padding-top:.75rem}.pt-4{padding-top:1rem}.text-left{text-align:left}.text-center{text-align:center}.font-mono{font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace}.text-2xl{font-size:1.5rem;line-height:2rem}.text-base{font-size:1rem;line-height:1.5rem}.text-lg{font-size:1.125rem;line-height:1.75rem}.text-sm{font-size:.875rem;line-height:1.25rem}.text-xl{font-size:1.25rem;line-height:1.75rem}.text-xs{font-size:.75rem;line-height:1rem}.font-bold{font-weight:700}.font-medium{font-weight:500}.font-semibold{font-weight:600}.leading-snug{line-height:1.375}.text-amber-700{--tw-text-opacity:1;color:rgb(180 83 9/var(--tw-text-opacity,1))}.text-amber-800{--tw-text-opacity:1;color:rgb(146 64 14/var(--tw-text-opacity,1))}.text-blue-600{--tw-text-opacity:1;color:rgb(37 99 235/var(--tw-text-opacity,1))}.text-blue-700{--tw-text-opacity:1;color:rgb(29 78 216/var(--tw-text-opacity,1))}.text-blue-800{--tw-text-opacity:1;color:rgb(30 64 175/var(--tw-text-opacity,1))}.text-gray-400{--tw-text-opacity:1;color:rgb(156 163 175/var(--tw-text-opacity,1))}.text-gray-500{--tw-text-opacity:1;color:rgb(107 114 128/var(--tw-text-opacity,1))}.text-gray-600{--tw-text-opacity:1;color:rgb(75 85 99/var(--tw-text-opacity,1))}.text-gray-700{--tw-text-opacity:1;color:rgb(55 65 81/var(--tw-text-opacity,1))}.text-gray-800{--tw-text-opacity:1;color:rgb(31 41 55/var(--tw-text-opacity,1))}.text-gray-900{--tw-text-opacity:1;color:rgb(17 24 39/var(--tw-text-opacity,1))}.text-green-600{--tw-text-opacity:1;color:rgb(22 163 74/var(--tw-text-opacity,1))}.text-green-700{--tw-text-opacity:1;color:rgb(21 128 61/var(--tw-text-opacity,1))}.text-green-800{--tw-text-opacity:1;color:rgb(22 101 52/var(--tw-text-opacity,1))}.text-indigo-700{--tw-text-opacity:1;color:rgb(67 56 202/var(--tw-text-opacity,1))}.text-purple-600{--tw-text-opacity:1;color:rgb(147 51 234/var(--tw-text-opacity,1))}.text-purple-700{--tw-text-opacity:1;color:rgb(126 34 206/var(--tw-text-opacity,1))}.text-red-600{--tw-text-opacity:1;color:rgb(220 38 38/var(--tw-text-opacity,1))}.text-red-700{--tw-text-opacity:1;color:rgb(185 28 28/var(--tw-text-opacity,1))}.text-white{--tw-text-opacity:1;color:rgb(255 255 255/var(--tw-text-opacity,1))}.text-yellow-600{--tw-text-opacity:1;color:rgb(202 138 4/var(--tw-text-opacity,1))}.text-yellow-700{--tw-text-opacity:1;color:rgb(161 98 7/var(--tw-text-opacity,1))}.text-yellow-800{--tw-text-opacity:1;color:rgb(133 77 14/var(--tw-text-opacity,1))}.underline{text-decoration-line:underline}.accent-purple-600{accent-color:#9333ea}.opacity-50{opacity:.5}.shadow{--tw-shadow:0 1px 3px 0 rgba(0,0,0,.1),0 1px 2px -1px rgba(0,0,0,.1);--tw-shadow-colored:0 1px 3px 0 var(--tw-shadow-color),0 1px 2px -1px var(--tw-shadow-color)}.shadow,.shadow-md{box-shadow:var(--tw-ring-offset-shadow,0 0 #0000),var(--tw-ring-shadow,0 0 #0000),var(--tw-shadow)}.shadow-md{--tw-shadow:0 4px 6px -1px rgba(0,0,0,.1),0 2px 4px -2px rgba(0,0,0,.1);--tw-shadow-colored:0 4px 6px -1px var(--tw-shadow-color),0 2px 4px -2px var(--tw-shadow-color)}.shadow-sm{--tw-shadow:0 1px 2px 0 rgba(0,0,0,.05);--tw-shadow-colored:0 1px 2px 0 var(--tw-shadow-color);box-shadow:var(--tw-ring-offset-shadow,0 0 #0000),var(--tw-ring-shadow,0 0 #0000),var(--tw-shadow)}.backdrop-blur{--tw-backdrop-blur:blur(8px);-webkit-backdrop-filter:var(--tw-backdrop-blur) var(--tw-backdrop-brightness) var(--tw-backdrop-contrast) var(--tw-backdrop-grayscale) var(--tw-backdrop-hue-rotate) var(--tw-backdrop-invert) var(--tw-backdrop-opacity) var(--tw-backdrop-saturate) var(--tw-backdrop-sepia);backdrop-filter:var(--tw-backdrop-blur) var(--tw-backdrop-brightness) var(--tw-backdrop-contrast) var(--tw-backdrop-grayscale) var(--tw-backdrop-hue-rotate) var(--tw-backdrop-invert) var(--tw-backdrop-opacity) var(--tw-backdrop-saturate) var(--tw-backdrop-sepia)}.transition{transition-property:color,background-color,border-color,text-decoration-color,fill,stroke,opacity,box-shadow,transform,filter,-webkit-backdrop-filter;transition-property:color,background-color,border-color,text-decoration-color,fill,stroke,opacity,box-shadow,transform,filter,backdrop-filter;transition-property:color,background-color,border-color,text-decoration-color,fill,stroke,opacity,box-shadow,transform,filter,backdrop-filter,-webkit-backdrop-filter;transition-timing-function:cubic-bezier(.4,0,.2,1);transition-duration:.15s}.transition-all{transition-property:all;transition-timing-function:cubic-bezier(.4,0,.2,1);transition-duration:.15s}.transition-colors{transition-property:color,background-color,border-color,text-decoration-color,fill,stroke;transition-timing-function:cubic-bezier(.4,0,.2,1);transition-duration:.15s}.transition-transform{transition-property:transform;transition-timing-function:cubic-bezier(.4,0,.2,1);transition-duration:.15s}.hover\:bg-blue-100:hover{--tw-bg-opacity:1;background-color:rgb(219 234 254/var(--tw-bg-opacity,1))}.hover\:bg-gray-100:hover{--tw-bg-opacity:1;background-color:rgb(243 244 246/var(--tw-bg-opacity,1))}.hover\:bg-gray-100\/70:hover{background-color:rgba(243,244,246,.7)}.hover\:bg-gray-200:hover{--tw-bg-opacity:1;background-color:rgb(229 231 235/var(--tw-bg-opacity,1))}.hover\:text-blue-600:hover{--tw-text-opacity:1;color:rgb(37 99 235/var(--tw-text-opacity,1))}.hover\:opacity-90:hover{opacity:.9}.hover\:shadow-md:hover{--tw-shadow:0 4px 6px -1px rgba(0,0,0,.1),0 2px 4px -2px rgba(0,0,0,.1);--tw-shadow-colored:0 4px 6px -1px var(--tw-shadow-color),0 2px 4px -2px var(--tw-shadow-color);box-shadow:var(--tw-ring-offset-shadow,0 0 #0000),var(--tw-ring-shadow,0 0 #0000),var(--tw-shadow)}.focus\:ring-1:focus{--tw-ring-offset-shadow:var(--tw-ring-inset) 0 0 0 var(--tw-ring-offset-width) var(--tw-ring-offset-color);--tw-ring-shadow:var(--tw-ring-inset) 0 0 0 calc(1px + var(--tw-ring-offset-width)) var(--tw-ring-color);box-shadow:var(--tw-ring-offset-shadow),var(--tw-ring-shadow),var(--tw-shadow,0 0 #0000)}.focus\:ring-blue-500:focus{--tw-ring-opacity:1;--tw-ring-color:rgb(59 130 246/var(--tw-ring-opacity,1))}.group[open] .group-open\:rotate-180{--tw-rotate:180deg;transform:translate(var(--tw-translate-x),var(--tw-translate-y)) rotate(var(--tw-rotate)) skewX(var(--tw-skew-x)) skewY(var(--tw-skew-y)) scaleX(var(--tw-scale-x)) scaleY(var(--tw-scale-y))}@media (min-width:640px){.sm\:grid-cols-2{grid-template-columns:repeat(2,minmax(0,1fr))}}@media (min-width:768px){.md\:col-span-2{grid-column:span 2/span 2}.md\:col-span-3{grid-column:span 3/span 3}.md\:basis-auto{flex-basis:auto}.md\:grid-cols-2{grid-template-columns:repeat(2,minmax(0,1fr))}.md\:grid-cols-3{grid-template-columns:repeat(3,minmax(0,1fr))}.md\:grid-cols-4{grid-template-columns:repeat(4,minmax(0,1fr))}}@media (min-width:1024px){.lg\:col-span-2{grid-column:span 2/span 2}.lg\:grid-cols-3{grid-template-columns:repeat(3,minmax(0,1fr))}.lg\:grid-cols-4{grid-template-columns:repeat(4,minmax(0,1fr))}}@media (min-width:1280px){.xl\:grid-cols-3{grid-template-columns:repeat(3,minmax(0,1fr))}} \ No newline at end of file diff --git a/goodprice/web/static/htmx.LICENSE b/goodprice/web/static/htmx.LICENSE new file mode 100644 index 0000000..d3061a3 --- /dev/null +++ b/goodprice/web/static/htmx.LICENSE @@ -0,0 +1,13 @@ +Zero-Clause BSD +============= + +Permission to use, copy, modify, and/or distribute this software for +any purpose with or without fee is hereby granted. + +THE SOFTWARE IS PROVIDED “AS IS” AND THE AUTHOR DISCLAIMS ALL +WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES +OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE +FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY +DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN +AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT +OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. diff --git a/goodprice/web/static/htmx.min.js b/goodprice/web/static/htmx.min.js new file mode 100644 index 0000000..de5f0f1 --- /dev/null +++ b/goodprice/web/static/htmx.min.js @@ -0,0 +1 @@ +(function(e,t){if(typeof define==="function"&&define.amd){define([],t)}else if(typeof module==="object"&&module.exports){module.exports=t()}else{e.htmx=e.htmx||t()}})(typeof self!=="undefined"?self:this,function(){return function(){"use strict";var Q={onLoad:F,process:zt,on:de,off:ge,trigger:ce,ajax:Nr,find:C,findAll:f,closest:v,values:function(e,t){var r=dr(e,t||"post");return r.values},remove:_,addClass:z,removeClass:n,toggleClass:$,takeClass:W,defineExtension:Ur,removeExtension:Br,logAll:V,logNone:j,logger:null,config:{historyEnabled:true,historyCacheSize:10,refreshOnHistoryMiss:false,defaultSwapStyle:"innerHTML",defaultSwapDelay:0,defaultSettleDelay:20,includeIndicatorStyles:true,indicatorClass:"htmx-indicator",requestClass:"htmx-request",addedClass:"htmx-added",settlingClass:"htmx-settling",swappingClass:"htmx-swapping",allowEval:true,allowScriptTags:true,inlineScriptNonce:"",attributesToSettle:["class","style","width","height"],withCredentials:false,timeout:0,wsReconnectDelay:"full-jitter",wsBinaryType:"blob",disableSelector:"[hx-disable], [data-hx-disable]",useTemplateFragments:false,scrollBehavior:"smooth",defaultFocusScroll:false,getCacheBusterParam:false,globalViewTransitions:false,methodsThatUseUrlParams:["get"],selfRequestsOnly:false,ignoreTitle:false,scrollIntoViewOnBoost:true,triggerSpecsCache:null},parseInterval:d,_:t,createEventSource:function(e){return new EventSource(e,{withCredentials:true})},createWebSocket:function(e){var t=new WebSocket(e,[]);t.binaryType=Q.config.wsBinaryType;return t},version:"1.9.12"};var r={addTriggerHandler:Lt,bodyContains:se,canAccessLocalStorage:U,findThisElement:xe,filterValues:yr,hasAttribute:o,getAttributeValue:te,getClosestAttributeValue:ne,getClosestMatch:c,getExpressionVars:Hr,getHeaders:xr,getInputValues:dr,getInternalData:ae,getSwapSpecification:wr,getTriggerSpecs:it,getTarget:ye,makeFragment:l,mergeObjects:le,makeSettleInfo:T,oobSwap:Ee,querySelectorExt:ue,selectAndSwap:je,settleImmediately:nr,shouldCancel:ut,triggerEvent:ce,triggerErrorEvent:fe,withExtensions:R};var w=["get","post","put","delete","patch"];var i=w.map(function(e){return"[hx-"+e+"], [data-hx-"+e+"]"}).join(", ");var S=e("head"),q=e("title"),H=e("svg",true);function e(e,t){return new RegExp("<"+e+"(\\s[^>]*>|>)([\\s\\S]*?)<\\/"+e+">",!!t?"gim":"im")}function d(e){if(e==undefined){return undefined}let t=NaN;if(e.slice(-2)=="ms"){t=parseFloat(e.slice(0,-2))}else if(e.slice(-1)=="s"){t=parseFloat(e.slice(0,-1))*1e3}else if(e.slice(-1)=="m"){t=parseFloat(e.slice(0,-1))*1e3*60}else{t=parseFloat(e)}return isNaN(t)?undefined:t}function ee(e,t){return e.getAttribute&&e.getAttribute(t)}function o(e,t){return e.hasAttribute&&(e.hasAttribute(t)||e.hasAttribute("data-"+t))}function te(e,t){return ee(e,t)||ee(e,"data-"+t)}function u(e){return e.parentElement}function re(){return document}function c(e,t){while(e&&!t(e)){e=u(e)}return e?e:null}function L(e,t,r){var n=te(t,r);var i=te(t,"hx-disinherit");if(e!==t&&i&&(i==="*"||i.split(" ").indexOf(r)>=0)){return"unset"}else{return n}}function ne(t,r){var n=null;c(t,function(e){return n=L(t,e,r)});if(n!=="unset"){return n}}function h(e,t){var r=e.matches||e.matchesSelector||e.msMatchesSelector||e.mozMatchesSelector||e.webkitMatchesSelector||e.oMatchesSelector;return r&&r.call(e,t)}function A(e){var t=/<([a-z][^\/\0>\x20\t\r\n\f]*)/i;var r=t.exec(e);if(r){return r[1].toLowerCase()}else{return""}}function s(e,t){var r=new DOMParser;var n=r.parseFromString(e,"text/html");var i=n.body;while(t>0){t--;i=i.firstChild}if(i==null){i=re().createDocumentFragment()}return i}function N(e){return/",0);var a=i.querySelector("template").content;if(Q.config.allowScriptTags){oe(a.querySelectorAll("script"),function(e){if(Q.config.inlineScriptNonce){e.nonce=Q.config.inlineScriptNonce}e.htmxExecuted=navigator.userAgent.indexOf("Firefox")===-1})}else{oe(a.querySelectorAll("script"),function(e){_(e)})}return a}switch(r){case"thead":case"tbody":case"tfoot":case"colgroup":case"caption":return s("
时间价格
"+n+"
",1);case"col":return s(""+n+"
",2);case"tr":return s(""+n+"
",2);case"td":case"th":return s(""+n+"
",3);case"script":case"style":return s("
"+n+"
",1);default:return s(n,0)}}function ie(e){if(e){e()}}function I(e,t){return Object.prototype.toString.call(e)==="[object "+t+"]"}function k(e){return I(e,"Function")}function P(e){return I(e,"Object")}function ae(e){var t="htmx-internal-data";var r=e[t];if(!r){r=e[t]={}}return r}function M(e){var t=[];if(e){for(var r=0;r=0}function se(e){if(e.getRootNode&&e.getRootNode()instanceof window.ShadowRoot){return re().body.contains(e.getRootNode().host)}else{return re().body.contains(e)}}function D(e){return e.trim().split(/\s+/)}function le(e,t){for(var r in t){if(t.hasOwnProperty(r)){e[r]=t[r]}}return e}function E(e){try{return JSON.parse(e)}catch(e){b(e);return null}}function U(){var e="htmx:localStorageTest";try{localStorage.setItem(e,e);localStorage.removeItem(e);return true}catch(e){return false}}function B(t){try{var e=new URL(t);if(e){t=e.pathname+e.search}if(!/^\/$/.test(t)){t=t.replace(/\/+$/,"")}return t}catch(e){return t}}function t(e){return Tr(re().body,function(){return eval(e)})}function F(t){var e=Q.on("htmx:load",function(e){t(e.detail.elt)});return e}function V(){Q.logger=function(e,t,r){if(console){console.log(t,e,r)}}}function j(){Q.logger=null}function C(e,t){if(t){return e.querySelector(t)}else{return C(re(),e)}}function f(e,t){if(t){return e.querySelectorAll(t)}else{return f(re(),e)}}function _(e,t){e=p(e);if(t){setTimeout(function(){_(e);e=null},t)}else{e.parentElement.removeChild(e)}}function z(e,t,r){e=p(e);if(r){setTimeout(function(){z(e,t);e=null},r)}else{e.classList&&e.classList.add(t)}}function n(e,t,r){e=p(e);if(r){setTimeout(function(){n(e,t);e=null},r)}else{if(e.classList){e.classList.remove(t);if(e.classList.length===0){e.removeAttribute("class")}}}}function $(e,t){e=p(e);e.classList.toggle(t)}function W(e,t){e=p(e);oe(e.parentElement.children,function(e){n(e,t)});z(e,t)}function v(e,t){e=p(e);if(e.closest){return e.closest(t)}else{do{if(e==null||h(e,t)){return e}}while(e=e&&u(e));return null}}function g(e,t){return e.substring(0,t.length)===t}function G(e,t){return e.substring(e.length-t.length)===t}function J(e){var t=e.trim();if(g(t,"<")&&G(t,"/>")){return t.substring(1,t.length-2)}else{return t}}function Z(e,t){if(t.indexOf("closest ")===0){return[v(e,J(t.substr(8)))]}else if(t.indexOf("find ")===0){return[C(e,J(t.substr(5)))]}else if(t==="next"){return[e.nextElementSibling]}else if(t.indexOf("next ")===0){return[K(e,J(t.substr(5)))]}else if(t==="previous"){return[e.previousElementSibling]}else if(t.indexOf("previous ")===0){return[Y(e,J(t.substr(9)))]}else if(t==="document"){return[document]}else if(t==="window"){return[window]}else if(t==="body"){return[document.body]}else{return re().querySelectorAll(J(t))}}var K=function(e,t){var r=re().querySelectorAll(t);for(var n=0;n=0;n--){var i=r[n];if(i.compareDocumentPosition(e)===Node.DOCUMENT_POSITION_FOLLOWING){return i}}};function ue(e,t){if(t){return Z(e,t)[0]}else{return Z(re().body,e)[0]}}function p(e){if(I(e,"String")){return C(e)}else{return e}}function ve(e,t,r){if(k(t)){return{target:re().body,event:e,listener:t}}else{return{target:p(e),event:t,listener:r}}}function de(t,r,n){jr(function(){var e=ve(t,r,n);e.target.addEventListener(e.event,e.listener)});var e=k(r);return e?r:n}function ge(t,r,n){jr(function(){var e=ve(t,r,n);e.target.removeEventListener(e.event,e.listener)});return k(r)?r:n}var pe=re().createElement("output");function me(e,t){var r=ne(e,t);if(r){if(r==="this"){return[xe(e,t)]}else{var n=Z(e,r);if(n.length===0){b('The selector "'+r+'" on '+t+" returned no matches!");return[pe]}else{return n}}}}function xe(e,t){return c(e,function(e){return te(e,t)!=null})}function ye(e){var t=ne(e,"hx-target");if(t){if(t==="this"){return xe(e,"hx-target")}else{return ue(e,t)}}else{var r=ae(e);if(r.boosted){return re().body}else{return e}}}function be(e){var t=Q.config.attributesToSettle;for(var r=0;r0){o=e.substr(0,e.indexOf(":"));t=e.substr(e.indexOf(":")+1,e.length)}else{o=e}var r=re().querySelectorAll(t);if(r){oe(r,function(e){var t;var r=i.cloneNode(true);t=re().createDocumentFragment();t.appendChild(r);if(!Se(o,e)){t=r}var n={shouldSwap:true,target:e,fragment:t};if(!ce(e,"htmx:oobBeforeSwap",n))return;e=n.target;if(n["shouldSwap"]){Fe(o,e,e,t,a)}oe(a.elts,function(e){ce(e,"htmx:oobAfterSwap",n)})});i.parentNode.removeChild(i)}else{i.parentNode.removeChild(i);fe(re().body,"htmx:oobErrorNoTarget",{content:i})}return e}function Ce(e,t,r){var n=ne(e,"hx-select-oob");if(n){var i=n.split(",");for(var a=0;a0){var r=t.replace("'","\\'");var n=e.tagName.replace(":","\\:");var i=o.querySelector(n+"[id='"+r+"']");if(i&&i!==o){var a=e.cloneNode();we(e,i);s.tasks.push(function(){we(e,a)})}}})}function Oe(e){return function(){n(e,Q.config.addedClass);zt(e);Nt(e);qe(e);ce(e,"htmx:load")}}function qe(e){var t="[autofocus]";var r=h(e,t)?e:e.querySelector(t);if(r!=null){r.focus()}}function a(e,t,r,n){Te(e,r,n);while(r.childNodes.length>0){var i=r.firstChild;z(i,Q.config.addedClass);e.insertBefore(i,t);if(i.nodeType!==Node.TEXT_NODE&&i.nodeType!==Node.COMMENT_NODE){n.tasks.push(Oe(i))}}}function He(e,t){var r=0;while(r-1){var t=e.replace(H,"");var r=t.match(q);if(r){return r[2]}}}function je(e,t,r,n,i,a){i.title=Ve(n);var o=l(n);if(o){Ce(r,o,i);o=Be(r,o,a);Re(o);return Fe(e,r,t,o,i)}}function _e(e,t,r){var n=e.getResponseHeader(t);if(n.indexOf("{")===0){var i=E(n);for(var a in i){if(i.hasOwnProperty(a)){var o=i[a];if(!P(o)){o={value:o}}ce(r,a,o)}}}else{var s=n.split(",");for(var l=0;l0){var o=t[0];if(o==="]"){n--;if(n===0){if(a===null){i=i+"true"}t.shift();i+=")})";try{var s=Tr(e,function(){return Function(i)()},function(){return true});s.source=i;return s}catch(e){fe(re().body,"htmx:syntax:error",{error:e,source:i});return null}}}else if(o==="["){n++}if(Qe(o,a,r)){i+="(("+r+"."+o+") ? ("+r+"."+o+") : (window."+o+"))"}else{i=i+o}a=t.shift()}}}function y(e,t){var r="";while(e.length>0&&!t.test(e[0])){r+=e.shift()}return r}function tt(e){var t;if(e.length>0&&Ze.test(e[0])){e.shift();t=y(e,Ke).trim();e.shift()}else{t=y(e,x)}return t}var rt="input, textarea, select";function nt(e,t,r){var n=[];var i=Ye(t);do{y(i,Je);var a=i.length;var o=y(i,/[,\[\s]/);if(o!==""){if(o==="every"){var s={trigger:"every"};y(i,Je);s.pollInterval=d(y(i,/[,\[\s]/));y(i,Je);var l=et(e,i,"event");if(l){s.eventFilter=l}n.push(s)}else if(o.indexOf("sse:")===0){n.push({trigger:"sse",sseEvent:o.substr(4)})}else{var u={trigger:o};var l=et(e,i,"event");if(l){u.eventFilter=l}while(i.length>0&&i[0]!==","){y(i,Je);var f=i.shift();if(f==="changed"){u.changed=true}else if(f==="once"){u.once=true}else if(f==="consume"){u.consume=true}else if(f==="delay"&&i[0]===":"){i.shift();u.delay=d(y(i,x))}else if(f==="from"&&i[0]===":"){i.shift();if(Ze.test(i[0])){var c=tt(i)}else{var c=y(i,x);if(c==="closest"||c==="find"||c==="next"||c==="previous"){i.shift();var h=tt(i);if(h.length>0){c+=" "+h}}}u.from=c}else if(f==="target"&&i[0]===":"){i.shift();u.target=tt(i)}else if(f==="throttle"&&i[0]===":"){i.shift();u.throttle=d(y(i,x))}else if(f==="queue"&&i[0]===":"){i.shift();u.queue=y(i,x)}else if(f==="root"&&i[0]===":"){i.shift();u[f]=tt(i)}else if(f==="threshold"&&i[0]===":"){i.shift();u[f]=y(i,x)}else{fe(e,"htmx:syntax:error",{token:i.shift()})}}n.push(u)}}if(i.length===a){fe(e,"htmx:syntax:error",{token:i.shift()})}y(i,Je)}while(i[0]===","&&i.shift());if(r){r[t]=n}return n}function it(e){var t=te(e,"hx-trigger");var r=[];if(t){var n=Q.config.triggerSpecsCache;r=n&&n[t]||nt(e,t,n)}if(r.length>0){return r}else if(h(e,"form")){return[{trigger:"submit"}]}else if(h(e,'input[type="button"], input[type="submit"]')){return[{trigger:"click"}]}else if(h(e,rt)){return[{trigger:"change"}]}else{return[{trigger:"click"}]}}function at(e){ae(e).cancelled=true}function ot(e,t,r){var n=ae(e);n.timeout=setTimeout(function(){if(se(e)&&n.cancelled!==true){if(!ct(r,e,Wt("hx:poll:trigger",{triggerSpec:r,target:e}))){t(e)}ot(e,t,r)}},r.pollInterval)}function st(e){return location.hostname===e.hostname&&ee(e,"href")&&ee(e,"href").indexOf("#")!==0}function lt(t,r,e){if(t.tagName==="A"&&st(t)&&(t.target===""||t.target==="_self")||t.tagName==="FORM"){r.boosted=true;var n,i;if(t.tagName==="A"){n="get";i=ee(t,"href")}else{var a=ee(t,"method");n=a?a.toLowerCase():"get";if(n==="get"){}i=ee(t,"action")}e.forEach(function(e){ht(t,function(e,t){if(v(e,Q.config.disableSelector)){m(e);return}he(n,i,e,t)},r,e,true)})}}function ut(e,t){if(e.type==="submit"||e.type==="click"){if(t.tagName==="FORM"){return true}if(h(t,'input[type="submit"], button')&&v(t,"form")!==null){return true}if(t.tagName==="A"&&t.href&&(t.getAttribute("href")==="#"||t.getAttribute("href").indexOf("#")!==0)){return true}}return false}function ft(e,t){return ae(e).boosted&&e.tagName==="A"&&t.type==="click"&&(t.ctrlKey||t.metaKey)}function ct(e,t,r){var n=e.eventFilter;if(n){try{return n.call(t,r)!==true}catch(e){fe(re().body,"htmx:eventFilter:error",{error:e,source:n.source});return true}}return false}function ht(a,o,e,s,l){var u=ae(a);var t;if(s.from){t=Z(a,s.from)}else{t=[a]}if(s.changed){t.forEach(function(e){var t=ae(e);t.lastValue=e.value})}oe(t,function(n){var i=function(e){if(!se(a)){n.removeEventListener(s.trigger,i);return}if(ft(a,e)){return}if(l||ut(e,a)){e.preventDefault()}if(ct(s,a,e)){return}var t=ae(e);t.triggerSpec=s;if(t.handledFor==null){t.handledFor=[]}if(t.handledFor.indexOf(a)<0){t.handledFor.push(a);if(s.consume){e.stopPropagation()}if(s.target&&e.target){if(!h(e.target,s.target)){return}}if(s.once){if(u.triggeredOnce){return}else{u.triggeredOnce=true}}if(s.changed){var r=ae(n);if(r.lastValue===n.value){return}r.lastValue=n.value}if(u.delayed){clearTimeout(u.delayed)}if(u.throttle){return}if(s.throttle>0){if(!u.throttle){o(a,e);u.throttle=setTimeout(function(){u.throttle=null},s.throttle)}}else if(s.delay>0){u.delayed=setTimeout(function(){o(a,e)},s.delay)}else{ce(a,"htmx:trigger");o(a,e)}}};if(e.listenerInfos==null){e.listenerInfos=[]}e.listenerInfos.push({trigger:s.trigger,listener:i,on:n});n.addEventListener(s.trigger,i)})}var vt=false;var dt=null;function gt(){if(!dt){dt=function(){vt=true};window.addEventListener("scroll",dt);setInterval(function(){if(vt){vt=false;oe(re().querySelectorAll("[hx-trigger='revealed'],[data-hx-trigger='revealed']"),function(e){pt(e)})}},200)}}function pt(t){if(!o(t,"data-hx-revealed")&&X(t)){t.setAttribute("data-hx-revealed","true");var e=ae(t);if(e.initHash){ce(t,"revealed")}else{t.addEventListener("htmx:afterProcessNode",function(e){ce(t,"revealed")},{once:true})}}}function mt(e,t,r){var n=D(r);for(var i=0;i=0){var t=wt(n);setTimeout(function(){xt(s,r,n+1)},t)}};t.onopen=function(e){n=0};ae(s).webSocket=t;t.addEventListener("message",function(e){if(yt(s)){return}var t=e.data;R(s,function(e){t=e.transformResponse(t,null,s)});var r=T(s);var n=l(t);var i=M(n.children);for(var a=0;a0){ce(u,"htmx:validation:halted",i);return}t.send(JSON.stringify(l));if(ut(e,u)){e.preventDefault()}})}else{fe(u,"htmx:noWebSocketSourceError")}}function wt(e){var t=Q.config.wsReconnectDelay;if(typeof t==="function"){return t(e)}if(t==="full-jitter"){var r=Math.min(e,6);var n=1e3*Math.pow(2,r);return n*Math.random()}b('htmx.config.wsReconnectDelay must either be a function or the string "full-jitter"')}function St(e,t,r){var n=D(r);for(var i=0;i0){setTimeout(i,n)}else{i()}}function Ht(t,i,e){var a=false;oe(w,function(r){if(o(t,"hx-"+r)){var n=te(t,"hx-"+r);a=true;i.path=n;i.verb=r;e.forEach(function(e){Lt(t,e,i,function(e,t){if(v(e,Q.config.disableSelector)){m(e);return}he(r,n,e,t)})})}});return a}function Lt(n,e,t,r){if(e.sseEvent){Rt(n,r,e.sseEvent)}else if(e.trigger==="revealed"){gt();ht(n,r,t,e);pt(n)}else if(e.trigger==="intersect"){var i={};if(e.root){i.root=ue(n,e.root)}if(e.threshold){i.threshold=parseFloat(e.threshold)}var a=new IntersectionObserver(function(e){for(var t=0;t0){t.polling=true;ot(n,r,e)}else{ht(n,r,t,e)}}function At(e){if(!e.htmxExecuted&&Q.config.allowScriptTags&&(e.type==="text/javascript"||e.type==="module"||e.type==="")){var t=re().createElement("script");oe(e.attributes,function(e){t.setAttribute(e.name,e.value)});t.textContent=e.textContent;t.async=false;if(Q.config.inlineScriptNonce){t.nonce=Q.config.inlineScriptNonce}var r=e.parentElement;try{r.insertBefore(t,e)}catch(e){b(e)}finally{if(e.parentElement){e.parentElement.removeChild(e)}}}}function Nt(e){if(h(e,"script")){At(e)}oe(f(e,"script"),function(e){At(e)})}function It(e){var t=e.attributes;if(!t){return false}for(var r=0;r0){var o=n.shift();var s=o.match(/^\s*([a-zA-Z:\-\.]+:)(.*)/);if(a===0&&s){o.split(":");i=s[1].slice(0,-1);r[i]=s[2]}else{r[i]+=o}a+=Bt(o)}for(var l in r){Ft(e,l,r[l])}}}function jt(e){Ae(e);for(var t=0;tQ.config.historyCacheSize){i.shift()}while(i.length>0){try{localStorage.setItem("htmx-history-cache",JSON.stringify(i));break}catch(e){fe(re().body,"htmx:historyCacheError",{cause:e,cache:i});i.shift()}}}function Yt(e){if(!U()){return null}e=B(e);var t=E(localStorage.getItem("htmx-history-cache"))||[];for(var r=0;r=200&&this.status<400){ce(re().body,"htmx:historyCacheMissLoad",o);var e=l(this.response);e=e.querySelector("[hx-history-elt],[data-hx-history-elt]")||e;var t=Zt();var r=T(t);var n=Ve(this.response);if(n){var i=C("title");if(i){i.innerHTML=n}else{window.document.title=n}}Ue(t,e,r);nr(r.tasks);Jt=a;ce(re().body,"htmx:historyRestore",{path:a,cacheMiss:true,serverResponse:this.response})}else{fe(re().body,"htmx:historyCacheMissLoadError",o)}};e.send()}function ar(e){er();e=e||location.pathname+location.search;var t=Yt(e);if(t){var r=l(t.content);var n=Zt();var i=T(n);Ue(n,r,i);nr(i.tasks);document.title=t.title;setTimeout(function(){window.scrollTo(0,t.scroll)},0);Jt=e;ce(re().body,"htmx:historyRestore",{path:e,item:t})}else{if(Q.config.refreshOnHistoryMiss){window.location.reload(true)}else{ir(e)}}}function or(e){var t=me(e,"hx-indicator");if(t==null){t=[e]}oe(t,function(e){var t=ae(e);t.requestCount=(t.requestCount||0)+1;e.classList["add"].call(e.classList,Q.config.requestClass)});return t}function sr(e){var t=me(e,"hx-disabled-elt");if(t==null){t=[]}oe(t,function(e){var t=ae(e);t.requestCount=(t.requestCount||0)+1;e.setAttribute("disabled","")});return t}function lr(e,t){oe(e,function(e){var t=ae(e);t.requestCount=(t.requestCount||0)-1;if(t.requestCount===0){e.classList["remove"].call(e.classList,Q.config.requestClass)}});oe(t,function(e){var t=ae(e);t.requestCount=(t.requestCount||0)-1;if(t.requestCount===0){e.removeAttribute("disabled")}})}function ur(e,t){for(var r=0;r=0}function wr(e,t){var r=t?t:ne(e,"hx-swap");var n={swapStyle:ae(e).boosted?"innerHTML":Q.config.defaultSwapStyle,swapDelay:Q.config.defaultSwapDelay,settleDelay:Q.config.defaultSettleDelay};if(Q.config.scrollIntoViewOnBoost&&ae(e).boosted&&!br(e)){n["show"]="top"}if(r){var i=D(r);if(i.length>0){for(var a=0;a0?l.join(":"):null;n["scroll"]=u;n["scrollTarget"]=f}else if(o.indexOf("show:")===0){var c=o.substr(5);var l=c.split(":");var h=l.pop();var f=l.length>0?l.join(":"):null;n["show"]=h;n["showTarget"]=f}else if(o.indexOf("focus-scroll:")===0){var v=o.substr("focus-scroll:".length);n["focusScroll"]=v=="true"}else if(a==0){n["swapStyle"]=o}else{b("Unknown modifier in hx-swap: "+o)}}}}return n}function Sr(e){return ne(e,"hx-encoding")==="multipart/form-data"||h(e,"form")&&ee(e,"enctype")==="multipart/form-data"}function Er(t,r,n){var i=null;R(r,function(e){if(i==null){i=e.encodeParameters(t,n,r)}});if(i!=null){return i}else{if(Sr(r)){return mr(n)}else{return pr(n)}}}function T(e){return{tasks:[],elts:[e]}}function Cr(e,t){var r=e[0];var n=e[e.length-1];if(t.scroll){var i=null;if(t.scrollTarget){i=ue(r,t.scrollTarget)}if(t.scroll==="top"&&(r||i)){i=i||r;i.scrollTop=0}if(t.scroll==="bottom"&&(n||i)){i=i||n;i.scrollTop=i.scrollHeight}}if(t.show){var i=null;if(t.showTarget){var a=t.showTarget;if(t.showTarget==="window"){a="body"}i=ue(r,a)}if(t.show==="top"&&(r||i)){i=i||r;i.scrollIntoView({block:"start",behavior:Q.config.scrollBehavior})}if(t.show==="bottom"&&(n||i)){i=i||n;i.scrollIntoView({block:"end",behavior:Q.config.scrollBehavior})}}}function Rr(e,t,r,n){if(n==null){n={}}if(e==null){return n}var i=te(e,t);if(i){var a=i.trim();var o=r;if(a==="unset"){return null}if(a.indexOf("javascript:")===0){a=a.substr(11);o=true}else if(a.indexOf("js:")===0){a=a.substr(3);o=true}if(a.indexOf("{")!==0){a="{"+a+"}"}var s;if(o){s=Tr(e,function(){return Function("return ("+a+")")()},{})}else{s=E(a)}for(var l in s){if(s.hasOwnProperty(l)){if(n[l]==null){n[l]=s[l]}}}}return Rr(u(e),t,r,n)}function Tr(e,t,r){if(Q.config.allowEval){return t()}else{fe(e,"htmx:evalDisallowedError");return r}}function Or(e,t){return Rr(e,"hx-vars",true,t)}function qr(e,t){return Rr(e,"hx-vals",false,t)}function Hr(e){return le(Or(e),qr(e))}function Lr(t,r,n){if(n!==null){try{t.setRequestHeader(r,n)}catch(e){t.setRequestHeader(r,encodeURIComponent(n));t.setRequestHeader(r+"-URI-AutoEncoded","true")}}}function Ar(t){if(t.responseURL&&typeof URL!=="undefined"){try{var e=new URL(t.responseURL);return e.pathname+e.search}catch(e){fe(re().body,"htmx:badResponseUrl",{url:t.responseURL})}}}function O(e,t){return t.test(e.getAllResponseHeaders())}function Nr(e,t,r){e=e.toLowerCase();if(r){if(r instanceof Element||I(r,"String")){return he(e,t,null,null,{targetOverride:p(r),returnPromise:true})}else{return he(e,t,p(r.source),r.event,{handler:r.handler,headers:r.headers,values:r.values,targetOverride:p(r.target),swapOverride:r.swap,select:r.select,returnPromise:true})}}else{return he(e,t,null,null,{returnPromise:true})}}function Ir(e){var t=[];while(e){t.push(e);e=e.parentElement}return t}function kr(e,t,r){var n;var i;if(typeof URL==="function"){i=new URL(t,document.location.href);var a=document.location.origin;n=a===i.origin}else{i=t;n=g(t,document.location.origin)}if(Q.config.selfRequestsOnly){if(!n){return false}}return ce(e,"htmx:validateUrl",le({url:i,sameHost:n},r))}function he(t,r,n,i,a,e){var o=null;var s=null;a=a!=null?a:{};if(a.returnPromise&&typeof Promise!=="undefined"){var l=new Promise(function(e,t){o=e;s=t})}if(n==null){n=re().body}var M=a.handler||Mr;var X=a.select||null;if(!se(n)){ie(o);return l}var u=a.targetOverride||ye(n);if(u==null||u==pe){fe(n,"htmx:targetError",{target:te(n,"hx-target")});ie(s);return l}var f=ae(n);var c=f.lastButtonClicked;if(c){var h=ee(c,"formaction");if(h!=null){r=h}var v=ee(c,"formmethod");if(v!=null){if(v.toLowerCase()!=="dialog"){t=v}}}var d=ne(n,"hx-confirm");if(e===undefined){var D=function(e){return he(t,r,n,i,a,!!e)};var U={target:u,elt:n,path:r,verb:t,triggeringEvent:i,etc:a,issueRequest:D,question:d};if(ce(n,"htmx:confirm",U)===false){ie(o);return l}}var g=n;var p=ne(n,"hx-sync");var m=null;var x=false;if(p){var B=p.split(":");var F=B[0].trim();if(F==="this"){g=xe(n,"hx-sync")}else{g=ue(n,F)}p=(B[1]||"drop").trim();f=ae(g);if(p==="drop"&&f.xhr&&f.abortable!==true){ie(o);return l}else if(p==="abort"){if(f.xhr){ie(o);return l}else{x=true}}else if(p==="replace"){ce(g,"htmx:abort")}else if(p.indexOf("queue")===0){var V=p.split(" ");m=(V[1]||"last").trim()}}if(f.xhr){if(f.abortable){ce(g,"htmx:abort")}else{if(m==null){if(i){var y=ae(i);if(y&&y.triggerSpec&&y.triggerSpec.queue){m=y.triggerSpec.queue}}if(m==null){m="last"}}if(f.queuedRequests==null){f.queuedRequests=[]}if(m==="first"&&f.queuedRequests.length===0){f.queuedRequests.push(function(){he(t,r,n,i,a)})}else if(m==="all"){f.queuedRequests.push(function(){he(t,r,n,i,a)})}else if(m==="last"){f.queuedRequests=[];f.queuedRequests.push(function(){he(t,r,n,i,a)})}ie(o);return l}}var b=new XMLHttpRequest;f.xhr=b;f.abortable=x;var w=function(){f.xhr=null;f.abortable=false;if(f.queuedRequests!=null&&f.queuedRequests.length>0){var e=f.queuedRequests.shift();e()}};var j=ne(n,"hx-prompt");if(j){var S=prompt(j);if(S===null||!ce(n,"htmx:prompt",{prompt:S,target:u})){ie(o);w();return l}}if(d&&!e){if(!confirm(d)){ie(o);w();return l}}var E=xr(n,u,S);if(t!=="get"&&!Sr(n)){E["Content-Type"]="application/x-www-form-urlencoded"}if(a.headers){E=le(E,a.headers)}var _=dr(n,t);var C=_.errors;var R=_.values;if(a.values){R=le(R,a.values)}var z=Hr(n);var $=le(R,z);var T=yr($,n);if(Q.config.getCacheBusterParam&&t==="get"){T["org.htmx.cache-buster"]=ee(u,"id")||"true"}if(r==null||r===""){r=re().location.href}var O=Rr(n,"hx-request");var W=ae(n).boosted;var q=Q.config.methodsThatUseUrlParams.indexOf(t)>=0;var H={boosted:W,useUrlParams:q,parameters:T,unfilteredParameters:$,headers:E,target:u,verb:t,errors:C,withCredentials:a.credentials||O.credentials||Q.config.withCredentials,timeout:a.timeout||O.timeout||Q.config.timeout,path:r,triggeringEvent:i};if(!ce(n,"htmx:configRequest",H)){ie(o);w();return l}r=H.path;t=H.verb;E=H.headers;T=H.parameters;C=H.errors;q=H.useUrlParams;if(C&&C.length>0){ce(n,"htmx:validation:halted",H);ie(o);w();return l}var G=r.split("#");var J=G[0];var L=G[1];var A=r;if(q){A=J;var Z=Object.keys(T).length!==0;if(Z){if(A.indexOf("?")<0){A+="?"}else{A+="&"}A+=pr(T);if(L){A+="#"+L}}}if(!kr(n,A,H)){fe(n,"htmx:invalidPath",H);ie(s);return l}b.open(t.toUpperCase(),A,true);b.overrideMimeType("text/html");b.withCredentials=H.withCredentials;b.timeout=H.timeout;if(O.noHeaders){}else{for(var N in E){if(E.hasOwnProperty(N)){var K=E[N];Lr(b,N,K)}}}var I={xhr:b,target:u,requestConfig:H,etc:a,boosted:W,select:X,pathInfo:{requestPath:r,finalRequestPath:A,anchor:L}};b.onload=function(){try{var e=Ir(n);I.pathInfo.responsePath=Ar(b);M(n,I);lr(k,P);ce(n,"htmx:afterRequest",I);ce(n,"htmx:afterOnLoad",I);if(!se(n)){var t=null;while(e.length>0&&t==null){var r=e.shift();if(se(r)){t=r}}if(t){ce(t,"htmx:afterRequest",I);ce(t,"htmx:afterOnLoad",I)}}ie(o);w()}catch(e){fe(n,"htmx:onLoadError",le({error:e},I));throw e}};b.onerror=function(){lr(k,P);fe(n,"htmx:afterRequest",I);fe(n,"htmx:sendError",I);ie(s);w()};b.onabort=function(){lr(k,P);fe(n,"htmx:afterRequest",I);fe(n,"htmx:sendAbort",I);ie(s);w()};b.ontimeout=function(){lr(k,P);fe(n,"htmx:afterRequest",I);fe(n,"htmx:timeout",I);ie(s);w()};if(!ce(n,"htmx:beforeRequest",I)){ie(o);w();return l}var k=or(n);var P=sr(n);oe(["loadstart","loadend","progress","abort"],function(t){oe([b,b.upload],function(e){e.addEventListener(t,function(e){ce(n,"htmx:xhr:"+t,{lengthComputable:e.lengthComputable,loaded:e.loaded,total:e.total})})})});ce(n,"htmx:beforeSend",I);var Y=q?null:Er(b,n,T);b.send(Y);return l}function Pr(e,t){var r=t.xhr;var n=null;var i=null;if(O(r,/HX-Push:/i)){n=r.getResponseHeader("HX-Push");i="push"}else if(O(r,/HX-Push-Url:/i)){n=r.getResponseHeader("HX-Push-Url");i="push"}else if(O(r,/HX-Replace-Url:/i)){n=r.getResponseHeader("HX-Replace-Url");i="replace"}if(n){if(n==="false"){return{}}else{return{type:i,path:n}}}var a=t.pathInfo.finalRequestPath;var o=t.pathInfo.responsePath;var s=ne(e,"hx-push-url");var l=ne(e,"hx-replace-url");var u=ae(e).boosted;var f=null;var c=null;if(s){f="push";c=s}else if(l){f="replace";c=l}else if(u){f="push";c=o||a}if(c){if(c==="false"){return{}}if(c==="true"){c=o||a}if(t.pathInfo.anchor&&c.indexOf("#")===-1){c=c+"#"+t.pathInfo.anchor}return{type:f,path:c}}else{return{}}}function Mr(l,u){var f=u.xhr;var c=u.target;var e=u.etc;var t=u.requestConfig;var h=u.select;if(!ce(l,"htmx:beforeOnLoad",u))return;if(O(f,/HX-Trigger:/i)){_e(f,"HX-Trigger",l)}if(O(f,/HX-Location:/i)){er();var r=f.getResponseHeader("HX-Location");var v;if(r.indexOf("{")===0){v=E(r);r=v["path"];delete v["path"]}Nr("GET",r,v).then(function(){tr(r)});return}var n=O(f,/HX-Refresh:/i)&&"true"===f.getResponseHeader("HX-Refresh");if(O(f,/HX-Redirect:/i)){location.href=f.getResponseHeader("HX-Redirect");n&&location.reload();return}if(n){location.reload();return}if(O(f,/HX-Retarget:/i)){if(f.getResponseHeader("HX-Retarget")==="this"){u.target=l}else{u.target=ue(l,f.getResponseHeader("HX-Retarget"))}}var d=Pr(l,u);var i=f.status>=200&&f.status<400&&f.status!==204;var g=f.response;var a=f.status>=400;var p=Q.config.ignoreTitle;var o=le({shouldSwap:i,serverResponse:g,isError:a,ignoreTitle:p},u);if(!ce(c,"htmx:beforeSwap",o))return;c=o.target;g=o.serverResponse;a=o.isError;p=o.ignoreTitle;u.target=c;u.failed=a;u.successful=!a;if(o.shouldSwap){if(f.status===286){at(l)}R(l,function(e){g=e.transformResponse(g,f,l)});if(d.type){er()}var s=e.swapOverride;if(O(f,/HX-Reswap:/i)){s=f.getResponseHeader("HX-Reswap")}var v=wr(l,s);if(v.hasOwnProperty("ignoreTitle")){p=v.ignoreTitle}c.classList.add(Q.config.swappingClass);var m=null;var x=null;var y=function(){try{var e=document.activeElement;var t={};try{t={elt:e,start:e?e.selectionStart:null,end:e?e.selectionEnd:null}}catch(e){}var r;if(h){r=h}if(O(f,/HX-Reselect:/i)){r=f.getResponseHeader("HX-Reselect")}if(d.type){ce(re().body,"htmx:beforeHistoryUpdate",le({history:d},u));if(d.type==="push"){tr(d.path);ce(re().body,"htmx:pushedIntoHistory",{path:d.path})}else{rr(d.path);ce(re().body,"htmx:replacedInHistory",{path:d.path})}}var n=T(c);je(v.swapStyle,c,l,g,n,r);if(t.elt&&!se(t.elt)&&ee(t.elt,"id")){var i=document.getElementById(ee(t.elt,"id"));var a={preventScroll:v.focusScroll!==undefined?!v.focusScroll:!Q.config.defaultFocusScroll};if(i){if(t.start&&i.setSelectionRange){try{i.setSelectionRange(t.start,t.end)}catch(e){}}i.focus(a)}}c.classList.remove(Q.config.swappingClass);oe(n.elts,function(e){if(e.classList){e.classList.add(Q.config.settlingClass)}ce(e,"htmx:afterSwap",u)});if(O(f,/HX-Trigger-After-Swap:/i)){var o=l;if(!se(l)){o=re().body}_e(f,"HX-Trigger-After-Swap",o)}var s=function(){oe(n.tasks,function(e){e.call()});oe(n.elts,function(e){if(e.classList){e.classList.remove(Q.config.settlingClass)}ce(e,"htmx:afterSettle",u)});if(u.pathInfo.anchor){var e=re().getElementById(u.pathInfo.anchor);if(e){e.scrollIntoView({block:"start",behavior:"auto"})}}if(n.title&&!p){var t=C("title");if(t){t.innerHTML=n.title}else{window.document.title=n.title}}Cr(n.elts,v);if(O(f,/HX-Trigger-After-Settle:/i)){var r=l;if(!se(l)){r=re().body}_e(f,"HX-Trigger-After-Settle",r)}ie(m)};if(v.settleDelay>0){setTimeout(s,v.settleDelay)}else{s()}}catch(e){fe(l,"htmx:swapError",u);ie(x);throw e}};var b=Q.config.globalViewTransitions;if(v.hasOwnProperty("transition")){b=v.transition}if(b&&ce(l,"htmx:beforeTransition",u)&&typeof Promise!=="undefined"&&document.startViewTransition){var w=new Promise(function(e,t){m=e;x=t});var S=y;y=function(){document.startViewTransition(function(){S();return w})}}if(v.swapDelay>0){setTimeout(y,v.swapDelay)}else{y()}}if(a){fe(l,"htmx:responseError",le({error:"Response Status Error Code "+f.status+" from "+u.pathInfo.requestPath},u))}}var Xr={};function Dr(){return{init:function(e){return null},onEvent:function(e,t){return true},transformResponse:function(e,t,r){return e},isInlineSwap:function(e){return false},handleSwap:function(e,t,r,n){return false},encodeParameters:function(e,t,r){return null}}}function Ur(e,t){if(t.init){t.init(r)}Xr[e]=le(Dr(),t)}function Br(e){delete Xr[e]}function Fr(e,r,n){if(e==undefined){return r}if(r==undefined){r=[]}if(n==undefined){n=[]}var t=te(e,"hx-ext");if(t){oe(t.split(","),function(e){e=e.replace(/ /g,"");if(e.slice(0,7)=="ignore:"){n.push(e.slice(7));return}if(n.indexOf(e)<0){var t=Xr[e];if(t&&r.indexOf(t)<0){r.push(t)}}})}return Fr(u(e),r,n)}var Vr=false;re().addEventListener("DOMContentLoaded",function(){Vr=true});function jr(e){if(Vr||re().readyState==="complete"){e()}else{re().addEventListener("DOMContentLoaded",e)}}function _r(){if(Q.config.includeIndicatorStyles!==false){re().head.insertAdjacentHTML("beforeend","")}}function zr(){var e=re().querySelector('meta[name="htmx-config"]');if(e){return E(e.content)}else{return null}}function $r(){var e=zr();if(e){Q.config=le(Q.config,e)}}jr(function(){$r();_r();var e=re().body;zt(e);var t=re().querySelectorAll("[hx-trigger='restored'],[data-hx-trigger='restored']");e.addEventListener("htmx:abort",function(e){var t=e.target;var r=ae(t);if(r&&r.xhr){r.xhr.abort()}});const r=window.onpopstate?window.onpopstate.bind(window):null;window.onpopstate=function(e){if(e.state&&e.state.htmx){ar();oe(t,function(e){ce(e,"htmx:restored",{document:re(),triggerEvent:ce})})}else{if(r){r(e)}}};setTimeout(function(){ce(e,"htmx:load",{});e=null},0)});return Q}()}); \ No newline at end of file diff --git a/goodprice/web/static/navigation.js b/goodprice/web/static/navigation.js new file mode 100644 index 0000000..109aec8 --- /dev/null +++ b/goodprice/web/static/navigation.js @@ -0,0 +1,81 @@ +(function () { + 'use strict'; + if (location.pathname !== '/listings') return; + function currentContext() { + var raw = new URLSearchParams(location.search), clean = new URLSearchParams(); + var task = raw.get('task_id'); + if (task && /^[0-9]+$/.test(task) && Number(task) > 0 && Number(task) <= 2147483647) clean.set('task_id', String(Number(task))); + var show = raw.get('show'), sort = raw.get('sort'); + clean.set('show', ['active','gone','blocked','all','unseen'].includes(show) ? show : 'active'); + clean.set('sort', ['satisfaction','price_asc','price_desc','newest'].includes(sort) ? sort : 'satisfaction'); + var state = raw.get('price_state'); + if (['exact','pending'].includes(state)) clean.set('price_state', state); + var loaded = raw.get('loaded'); + if (loaded && /^[0-9]+$/.test(loaded) && Number(loaded) > 100) clean.set('loaded', String(Math.min(1000, Number(loaded)))); + return '/listings?' + clean.toString(); + } + var key = function () { return 'second-eye:list:' + currentContext(); }; + var lastInteraction = 0; + function save(anchor) { + try { + sessionStorage.setItem(key(), JSON.stringify({y: window.scrollY, anchor: anchor || null})); + } catch (_) {} + } + function restore() { + try { + var state = JSON.parse(sessionStorage.getItem(key()) || 'null'); + if (!state) return; + requestAnimationFrame(function () { requestAnimationFrame(function () { + var anchor = state.anchor && document.getElementById('listing-' + state.anchor); + if (anchor && Math.abs(anchor.offsetTop - state.y) < 500) anchor.scrollIntoView(); + else window.scrollTo(0, state.y || 0); + }); }); + } catch (_) {} + } + function updateLinks() { + var context = currentContext(); + document.querySelectorAll('#listings-grid a[href^="/listings/"], #listings-grid form[action^="/listings/"]').forEach(function(el) { + var attr = el.tagName === 'FORM' ? 'action' : 'href'; + var url = new URL(el.getAttribute(attr), location.origin); + url.searchParams.set('return_to', context); + el.setAttribute(attr, url.pathname + url.search); + }); + var poller = document.getElementById('list-poller'); + if (poller) { + var pollURL = new URL(context, location.origin); + pollURL.searchParams.set('partial', '1'); + poller.setAttribute('hx-get', pollURL.pathname + pollURL.search); + if (window.htmx) htmx.process(poller); + } + var refresh = document.getElementById('new-list-results'); + if (refresh) refresh.href = context; + } + document.addEventListener('click', function(event) { + lastInteraction = Date.now(); + var card = event.target.closest('[data-listing-id]'); + save(card && card.dataset.listingId); + }); + document.addEventListener('input', function() { lastInteraction = Date.now(); }); + window.addEventListener('pagehide', function() { save(); }); + window.addEventListener('pageshow', restore); + document.addEventListener('htmx:beforeSwap', function(event) { + if (!event.detail.target || event.detail.target.id !== 'listings-grid') return; + var poller = event.detail.requestConfig && event.detail.requestConfig.elt; + if (poller && poller.id === 'list-poller' && ( + document.querySelector('.row-check:checked') || window.scrollY > 100 || + Date.now() - lastInteraction < 15000 || document.activeElement.closest('form'))) { + event.detail.shouldSwap = false; + document.getElementById('new-list-results').hidden = false; + } + }); + document.addEventListener('htmx:afterSettle', function() { + var seen = new Set(); + document.querySelectorAll('[data-listing-id]').forEach(function(card) { + if (seen.has(card.dataset.listingId)) card.remove(); + else seen.add(card.dataset.listingId); + }); + updateLinks(); + save(); + }); + updateLinks(); +})(); diff --git a/goodprice/web/static/tailwind.LICENSE b/goodprice/web/static/tailwind.LICENSE new file mode 100644 index 0000000..d6a8229 --- /dev/null +++ b/goodprice/web/static/tailwind.LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) Tailwind Labs, Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/goodprice/web/static/tailwind.input.css b/goodprice/web/static/tailwind.input.css new file mode 100644 index 0000000..b5c61c9 --- /dev/null +++ b/goodprice/web/static/tailwind.input.css @@ -0,0 +1,3 @@ +@tailwind base; +@tailwind components; +@tailwind utilities; diff --git a/goodprice/web/templates/base.html b/goodprice/web/templates/base.html index cfc91da..afc57f3 100644 --- a/goodprice/web/templates/base.html +++ b/goodprice/web/templates/base.html @@ -4,8 +4,9 @@ {% block title %}闲鱼盯价助手{% endblock %} - - + + +