Skip to content

Runtime: add an optional memory profiler #1100

Closed

Description

@josephjclark

In the runtime, if an option is set, we should profile memory usage.

It is useful to know the maximum/peak memory used by a step (and indeed by a whole workflow)

At the moment we log the final heap memory at the end of each step. But this doesn't tell us much really - GC has probably run after the most expensive operations. I don't actually think we log a "workflow total"?

For this to work we need to manually poll process.memoryUsage() and save the highest/peak memory used while the workflow is running. We might want a count per step and per workflow. We should poll often - ever 5 or 2 s. We can take an argument to set the poll rate.

Polling is cheap but even so, the more you do it the more it'll slow down runs.

ALSO! Rename "final memory usage" log to "step memory usage", because the final thing is super confusing

We need to decide whether to default this in the worker. It would be good to compare some jobs to assess performance impact.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    • Status
      Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions