Skip to content

fix(picod): cap stdout/stderr kept per execution - #494

Open
GuTS805 wants to merge 2 commits into
volcano-sh:mainfrom
GuTS805:fix/picod-cap-execute-output
Open

GuTS805 wants to merge 2 commits into
volcano-sh:mainfrom
GuTS805:fix/picod-cap-execute-output

Conversation

@GuTS805

@GuTS805 GuTS805 commented Oct 28, 2026

Copy link
Copy Markdown

What type of PR is this?

/kind bug

What this PR does / why we need it:

POST /api/execute buffered stdout/stderr in unbounded bytes.Buffers. A command with large output (cat of a big file, a print loop, yes) could grow PicoD's memory until the sandbox container was OOM-killed, losing the session's workspace and state.

This PR:

  • Adds a cappedBuffer writer that keeps the first N bytes per stream and discards the rest. Write always reports the full length, so the command keeps running instead of dying on a broken pipe.
  • Adds the --max-output-size flag (default 10 MB per stream; 0 or negative means the default).
  • Adds stdout_truncated / stderr_truncated to ExecuteResponse. Both are omitempty, so untruncated responses are byte-for-byte unchanged. A short note is also appended to stderr.
  • Writes the timeout and error messages to stderr outside the limit, so "Command timed out" is never lost when stderr is full.
  • Python SDK: execute_command_result returns both flags and logs a warning on truncation. The LangChain integration now passes the real truncated value instead of a hardcoded False.
  • Documents the limit in docs/design/picod-proposal.md.

Measured on Linux with 300 MB of command output:

Response size Memory obtained from OS (Sys)
main 300 MB +1934 MB
this PR 10 MB +68 MB

Which issue(s) this PR fixes:
Fixes #493

Special notes for your reviewer:

  • cappedBuffer deliberately does not embed bytes.Buffer. os/exec copies output with io.Copy, which would use an embedded ReadFrom and bypass the limit. TestCappedBuffer_IOCopyRespectsLimit covers this.
  • The truncation logic is in small helpers (outputLimit, appendTruncationNotes), so ExecuteHandler stays within the gocyclo limit of 15.
  • This touches the same lines in execute.go as fix(picod): stop child processes when execute times out #473 (process-group kill on timeout). Whichever merges second needs a small rebase around cmd.Stdout/cmd.Stderr.
  • Tests:
    • New table-driven cappedBuffer tests.
    • Handler tests for output within the limit, truncation of both streams, 1 MiB output through a 1 KiB limit (command still exits 0), and the timeout message kept when stderr is full.
    • SDK tests for the new flags.
    • go test ./pkg/picod/ passes on Linux, golangci-lint is clean, and ruff plus SDK unit tests pass.

Does this PR introduce a user-facing change?:

picod: command output kept per execution is now capped at `--max-output-size` bytes per stream (default 10 MB). Truncated responses include `stdout_truncated` / `stderr_truncated`, and the Python SDK exposes these flags.

@volcano-sh-bot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hzxuzhonghu for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ExecuteHandler buffered command output without limit, so a command
with large output could OOM-kill PicoD and lose the sandbox session.
Keep at most --max-output-size bytes (default 10 MB) per stream,
discard the rest without breaking the pipe, and report truncation via
stdout_truncated/stderr_truncated and a note in stderr.

Signed-off-by: GuTS805 <alok020505@gmail.com>
@codecov-commenter

codecov-commenter commented Oct 28, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 94.73684% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.91%. Comparing base (5903457) to head (1813345).
⚠️ Report is 42 commits behind head on main.

Files with missing lines Patch % Lines
pkg/picod/execute.go 94.73% 1 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #494      +/-   ##
==========================================
+ Coverage   59.98%   61.91%   +1.92%     
==========================================
  Files          36       35       -1     
  Lines        3589     3684      +95     
==========================================
+ Hits         2153     2281     +128     
+ Misses       1216     1178      -38     
- Partials      220      225       +5     
Flag Coverage Δ
unittests 61.91% <94.73%> (+1.92%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: GuTS805 <alok020505@gmail.com>
@acsoto

acsoto commented Oct 29, 2026

Copy link
Copy Markdown
Member

Has this occurred in actual usage? The reproduction demonstrates the risk, but I’m not yet convinced we need a configurable output limit and API changes at this stage.

@GuTS805

GuTS805 commented Oct 29, 2026

Copy link
Copy Markdown
Author

Not from a user report, but I reproduced it on the e2e kind setup (72763cc, 512Mi limit) via SDK and router with head -c 1G /dev/zero | tr '\0' a:

  1. main: 503 after 18.1s, PicoD OOMKilled (exit 137), and files written earlier in the session were gone.
  2. This PR: exit_code: 0 with the first 10 MB and a truncation note, no restart.

If the flag and API changes are too much for now, I can reduce it to a fixed limit plus a stderr note only. Let me know.

@zhzhuang-zju

Copy link
Copy Markdown
Contributor

This seems like a common challenge for code execution services. Could we look at how other projects handle large command output before settling on an approach?

@GuTS805

GuTS805 commented Oct 29, 2026

Copy link
Copy Markdown
Author

well I checked a few (Piston, LangChain shell tool, kubelet logs). Most of them cap the output per stream and set a truncated flag, which is what this PR does. Streaming is the other option, but it's a bigger change and already listed as a future enhancement in the PicoD proposal.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Something isn't working size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PicoD execute buffers stdout/stderr without a size limit; large output can OOM-kill the sandbox

5 participants