[josephjclark]

Short Description

This PR adds two new tools for tracking memory usage in the worker:

1. When the Worker claims, it'll log memory usage in the main thread
2. If setting profile to true, the runtime will poll memory usage while it's running and log the peak stack it detects. This should be a lot more accurate than the old memory analysis, but might have a small impact on performance (probably negligible)

Profiling is disabled by default. Set PROFILE=true to enable it.

I have not added profiling options to the CLI, and I have not added an integration test. I'm trying to keep the cost of this down.

Fixes #1100
Fixes #1101
Fixes #1022 (I was in the neighbourhood)

Old logs

R/T Transform data completed in 161ms
R/T Step memory usage: [step 105mb] [system 263mb]
R/T Run complete with status: success

New Logs

R/T Transform data completed in 152ms
R/T Step memory usage: peak 70.6mb
R/T Workflow complete: peak memory 71.37mb
R/T Run complete with status: success

Claim logs

[SRV] ❯ Claiming runs :: demand 1 | capacity 0/5 | memory 1% (21/4144mb)

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy