Skip to content

feat(pack): 提供文本 Ftrace 类型化 Parquet Provider - #231

Open
qiqingzhixin wants to merge 30 commits into
codex/issue-226-postgresql-pr2from
codex/text-ftrace-to-parquet
Open

feat(pack): 提供文本 Ftrace 类型化 Parquet Provider#231
qiqingzhixin wants to merge 30 commits into
codex/issue-226-postgresql-pr2from
codex/text-ftrace-to-parquet

Conversation

@qiqingzhixin

@qiqingzhixin qiqingzhixin commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

关联

依赖:#229
Refs #219#217
Closes #230

结果

提供文本 Ftrace 的完整 PACK Provider 纵向切片:独立 Rust ftrace2parquet 把未压缩 UTF-8 文本转换为 Proto 派生 Parquet 表;PACK 自有 Ftrace2ParquetProvider 在当前 Workflow 的临时 workspace 中调用转换器、校验必需关系,并通过 ds.DataFusionProvider 提供 query() -> ds.Table

交付内容

  • 提供 ftrace2parquet --input <trace.ftrace> --output <directory> --clock-domain <name>
  • 首批强类型支持 sched_switchsched_wakeupsched_wakeup_newtracing_mark_write
  • 合法未知事件保留 source_event_sequence 间隙,但不发布无类型载荷。
  • 逐行有界读取、十进制定点解析 Clock value、8,192 行分批写入,成功后整体发布 Parquet 目录。
  • 严格校验 tracer、buffer 计数、CPU 数、flags 图例、事件列标题和正文事件数。
  • 发布 text_ftrace_headertext_ftrace_event_occurrencetext_ftrace_event,以及来源实际出现的类型化 payload 关系。
  • 新增 External PACK ftrace2parquet-provider
    • Workflow 从 KAT_FTRACE2PARQUET_EXECUTABLE 取得部署提供的转换器;
    • Provider 不继承 KAT 类型,不注册或自动发现;
    • 转换、Catalog 打开或必需关系校验失败时保持未 ready 并清理独占目标;
    • query() eager 返回与进程和 Parquet reader 生命周期脱离的 ds.Table
  • 新增 summarize-ftrace Workflow,发布来源事件数、已支持事件数、tracer 和 CPU 数。

架构边界

text Ftrace
  -> Rust ftrace2parquet
  -> typed Parquet relations
  -> ds.open(root=...)
  -> ds.DataFusionProvider
  -> Ftrace2ParquetProvider.query()
  -> eager ds.Table
  -> Workflow Run Output
  • Rust 二进制拥有文本语法、Proto 类型合同和 Parquet 原子发布。
  • Python Provider 拥有进程调用、当前 Workflow 的 Catalog 生命周期和查询适配。
  • 不恢复旧 Data Import/Dataset writer 路径,不建立 Provider 基类、registry 或自动发现。
  • 不根据文件名、数值或运行环境猜测 Clock domain。

验证证据

  • cargo +stable test --locked -p ftrace2parquet:8 passed。
  • kat test --pack-dir examples/packs/ftrace2parquet-provider:9 passed。
  • cargo +stable test --locked -p kat-cli --lib:53 passed。
  • 真实纵向运行:Workflow 调用 Rust 二进制转换 fixture,发布 source_event_count=5supported_event_count=4cpu_count=4
  • cargo +stable fmt --all -- --check:通过。
  • git diff --check:通过。

仓库锁定的 Rust 1.95.0 在当前镜像返回 404;本地验证使用已安装的 Rust 1.97.1,未修改 rust-toolchain.toml

shijifei and others added 18 commits August 27, 2026 21:45
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[15%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
Refs #226

🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[88%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
Refs #226

🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[19%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[11%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[30%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[54%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[5%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[4%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[13%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[72%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[14%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[100%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[100%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[100%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[100%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
shijifei and others added 11 commits August 29, 2026 11:08
🤖 AI[0%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[0%]
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[100%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[100%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[100%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[100%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
🤖 AI[31%] 🔧 Human Fixed[0%] 🧑 Human[69%] 👌 AI Adopted[100%]
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[100%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
…pr1' into codex/issue-226-postgresql-pr2-rework

# Conflicts:
#	docs/sdd/2026-08-28-pack-datasource-toolkit.md
🤖 AI[100%] 🔧 Human Fixed[0%] 🧑 Human[0%] 👌 AI Adopted[100%]
Co-authored-by: codex (gpt-5.6-sol) <ai@local>
@qiqingzhixin qiqingzhixin changed the title feat:新增文本ftrace转换与头部校验 feat(pack): 提供文本 Ftrace 类型化 Parquet Provider Aug 31, 2026
@qiqingzhixin
qiqingzhixin changed the base branch from main to codex/issue-226-postgresql-pr2 August 31, 2026 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[slice]校验并保留文本ftrace头部信息

1 participant