[FeathBow]

Why

A worker on another host runs behind an SSH client, so the local child the core observes is the client, and its exit and end of file describe the connection. The remote side needs its own process owner that records which process it started, how it ended and what it wrote, in a form the core can read separately from the worker's protocol stream. Refs #17.

What

• invar remote run creates an instance directory, takes a lock in it for its whole life, and writes launching.json. It starts its child, invar remote exec, in a new process group led by that child, with exactly the declared environment and working directory. It publishes instance.json (host, nonce, wrapper and worker process ids, process group) before it releases a barrier, so the worker runs only after its instance is on disk. A close-on-exec status pipe separates a failed start, recorded in failure.json, from any exit of the worker.
• The wrapper forwards the worker's stdout unchanged and adds nothing to it, while counting and digesting every byte. When the reader goes away it keeps draining, so the count covers everything the worker wrote. It ignores hangups, so it keeps owning the child after a connection drops.
• One loop owns the child. It keeps the group leader unreaped while the worker's stdout is open or a termination is under way, so the group id cannot be reused while a signal may still go to it. It answers a matching termination request by sending TERM to the worker's group, and KILL to the group when the request's grace ends, so a member that ignores TERM and holds stdout after the leader exits is still killed. terminated.json and killed.json record the signals that were delivered. After an admitted request, the outcome is therefore written no sooner than the grace after it. When the child has ended and its stdout is drained, outcome.json records the instance, the exit or signal, the time and the stdout count and digest.
• Every control file is written under a staging name of its own, synchronized and renamed into place without replacing anything, so a staging file left by an interrupted writer never blocks another.
• An instance admits one termination request, which stays in place. A request whose nonce or process ids do not match is refused: the wrapper records it in refused.<n>.json with its own instance, the request and the reason, then removes it, so a request naming this instance can still terminate it. invar remote terminate writes the request when a live wrapper holds the instance and holds no request, waits for the answer to this request and reports it as confirmed, refused, unanswered or unrequested. A repeated call writes nothing and reports the confirmation of its own request; while another request is admitted, a call writes nothing and reports its own request as unrequested, never the answer to the other. Without a live wrapper it writes nothing and signals nothing.
• invar remote status reads the files in the reverse order of writing, counts a file removed before it is read as absent, and reports whether a live wrapper holds the lock.
• Invar.Remote.Control holds the record types, their encoders and one decoder. It refuses a file with another format or a missing or extra field, a file that names another launch or instance, an outcome without its instance, a launch failure together with an outcome, a kill without its termination, a confirmation without its accepted request, a refusal that names another instance, and a refusal of a request that would be accepted. Control.admitRequest is the one place that decodes a request and matches it against the instance.

Liveness is a lock held by the wrapper, released by the kernel when the wrapper ends. This replaces comparing a process id with a start time and boot identity: a reused process id cannot make a dead wrapper look alive, and the same check works on Linux and macOS. The nonce chosen per launch already makes the instance identity unique, so the identity is the host, the nonce and the two process ids.

The child is placed in its own process group as it is created, before the wrapper publishes the instance. If the child created its own group after it started, a termination request could arrive before that group existed.

Group signals reach only processes that stay in the worker's group. A process that leaves the group while holding the worker's stdout is out of their reach, and the wrapper waits for it to close stdout. When only the exited leader is left in the group as the grace ends, Linux accepts the KILL and macOS refuses it, so killed.json is written on Linux only in that case.

Validation

The core suite, formatting, lint, the type fixtures and cabal check pass. The test suite runs the built executable with real local child processes. It covers the start barrier and the status before, during and after a run, a program that cannot start, unchanged forwarding with a final line that has no newline, a reader that goes away while the worker writes a mebibyte, termination of a worker that has its own child, a kill after the grace, a group member that ignores TERM and holds stdout after the leader exits, a hangup to the wrapper, a request with no live wrapper, a refused request followed by a matching one that terminates the worker, and a repeated request and a different one, neither of which replaces the admitted request or takes its confirmation. The decoder's refusals are tested in one place. The core suite passed in repeated consecutive runs.