[taylordowns2000]

Short Description

Workers were failing intermittently on startup in GKE with "Invalid worker path" errors. The worker would crash and retry several times before eventually succeeding.

Fixes #1016

Implementation Details

Root Cause

The worker validation happens immediately after engine initialization with only a 5-second timeout. In containerized environments, child process initialization can be slower due to:

• File system delays (overlay fs, network storage)
• CPU throttling during startup
• Process spawn timing

Solution

1. Increased validation timeout from 5s to 15s
2. Added retry logic with exponential backoff (5 attempts with graceful backoff)

Testing

• All existing tests pass with updated retry parameter
• No behavior changes for successful validation (returns immediately)
• Only affects error path by adding retries before failure

QA Notes

@josephjclark , what's up with my pnpm-lock? Why such a big diff when I run pnpm install? This looks like the thing that causes all the other changes, if i had to guess: https://github.com/OpenFn/kit/pull/1088/files#diff-32824c984905bb02bc7ffcef96a77addd1f1602cff71a11fbbfdd7f53ee026bbR470

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy

Release branch checklist

Delete this section if this is not a release PR.

If this IS a release branch:

• Run pnpm changeset version from root to bump versions
• Run pnpm install
• Commit the new version numbers
• Run pnpm changeset tag to generate tags
• Push tags git push --tags

Tags may need updating if commits come in after the tags are first generated.

[josephjclark]

Ok this should work well:

• I've reverted the diff on package lock - no idea what that's about
• I've tweaked the implementation, added logging of how long the validation takes, and driven the values through options
• The engine by default does NOT retry
• But the worker by default will retry three times