Sudden explosion of lost runs #1096

Suddenly we're losing 10 runs a day?? And they all seem to be caused by heap errors

We know there's still a memory leak in production, and we have seen that occasionally something is triggering a heap error in the main thread. Presumably something processing large payloads when the memory is ticking up a bit high.

We recently appended the final state object to run:complete. For large payloads, that's just duplicating the cost of message processing. An 8mb state object has to be parsed and serialized on step:complete AND run:complete.

We also have QA runs which download the docsite HTML (about 5mb of data) and send it back to Lightning every 15 minutes.

Basically we've just increased the strain on the server significantly, giving it so much more data to process. And that's presumably just causing the memory to tick up above its available threshold.

I'm going to first try and repro this locally. And then I need to work out a) why this is so expensive in memory, b) whether it's memory leak related, and c) what we can do about it.