[eas-cli] Ingest realtime incrementally instead of rebuilding full logs on every message - #4293
Conversation
|
Subscribed to pull request
Generated by CodeMention Warning: The preamble and epilogue options in commentConfiguration are deprecated. Use template instead. |
There was a problem hiding this comment.
Pull request overview
Improves the performance of workflow:run --wait log rendering by switching workflow job log state management from “rebuild everything on every realtime message” to incremental in-place updates, reducing repeated parsing/grouping work for large log streams.
Changes:
- Refactored
WorkflowJobLogsStateto maintain grouped logs incrementally and reveal buffered realtime logs at the appropriate time (catch-up or completion). - Updated log parsing/grouping to support an accumulator (
groupLogLinesIntoSteps) and tracked step result separately to avoid recomputation. - Adjusted rendering/helpers and expanded Jest coverage for the new incremental behavior.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| packages/eas-cli/src/commandUtils/workflow/utils.ts | Uses precomputed step result, optimizes tail rendering, and marks completed jobs to reveal buffered realtime logs without rebuilding. |
| packages/eas-cli/src/commandUtils/workflow/types.ts | Introduces StepLogs and extends per-step log state with optional result. |
| packages/eas-cli/src/commandUtils/workflow/logs/watcher.ts | Implements incremental ingestion/deduping of realtime log lines and incremental grouping into a persistent WorkflowLogs map. |
| packages/eas-cli/src/commandUtils/workflow/logs/parseLogs.ts | Adds accumulator support to groupLogLinesIntoSteps and captures first end-marker result per step. |
| packages/eas-cli/src/commandUtils/workflow/logs/tests/watcher-test.ts | Updates/extends tests to validate incremental ingestion, completion reveal, and deduplication. |
| packages/eas-cli/src/commandUtils/workflow/logs/tests/parseLogs-test.ts | Adds tests for step result capture semantics (first end marker wins, undefined before end). |
| packages/eas-cli/src/commandUtils/workflow/tests/utils-test.ts | Expands formatting tests (default tail size, current step selection, skipped handling, non-in-progress behavior, failed-step output). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
4fec1e7 to
d3f0579
Compare
6bac8a1 to
310d0f5
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## realtime-logs #4293 +/- ##
================================================
Coverage ? 64.74%
================================================
Files ? 1052
Lines ? 48694
Branches ? 10257
================================================
Hits ? 31523
Misses ? 17055
Partials ? 116 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…gs on every message
310d0f5 to
7318bc9
Compare
|
❌ It looks like a changelog entry is missing for this PR. Add it manually to CHANGELOG.md. |
Why
Logs displayed by
workflow:run --waitrebuild the entire stored logs on every realtime logs message, which hurts the performance when displaying large workflow runs with many MB of logs.How
WorkflowJobLogsStateto maintain the current parsed logs and update them in place on realtime logs publications instead of rebuilding them from scratch on every publication.groupLogLinesIntoStepsto take an accumulator argument for the result instead of building it from scratchstepLogTailto slice then map, to prevent iterating over log linesSeparated this out of #4228 because I didn't want to complicate that PR too much. We should merge this into #4228 before merging to main.
Test Plan
CI passes with new tests,
workflow:run --waittested manuallyStack created with GitHub Stacks CLI • Give Feedback 💬