Submit Swift test executions through OpenTelemetry

[pda]

Why

The Swift collector currently buffers proprietary JSON executions inside the XCTest runner. When the runner is restarted or killed, completed results that have not reached /v1/uploads disappear; the XCUITest reproduction retains only 2 of 5 executions. An internal resilience assessment recommends handing each completed execution to the longer-lived Test Engine Client process through its OTLP relay.

What

• Remove the proprietary execution models and /v1/uploads client. Every completed XCTest is now a parentless test.execution root span marked with buildkite.execution.via=otlp, with failures represented as span status and exception events.
• Preserve run tags, execution-name settings, custom run-environment overrides, and the existing tagExecution API through OTLP resource/span attributes. Application spans created during a test are exported as children, including on Linux and when the application registers its tracer provider after collector startup.
• Support standard OTEL_EXPORTER_OTLP[_TRACES]_* HTTP configuration without disclosing the Buildkite suite token to standard endpoints. The default and collector-specific trusted endpoint retain suite-token authentication.
• Wait synchronously for root-span endpoint acceptance and keep failed batches in the collector process for retry. With the Test Engine Client relay, this hands completed tests out of a restartable XCTest runner before it exits.
• Support Swift 5.10 and macOS 12 or newer. Active tests without an XCTest completion callback, process-memory queues, and ambiguous accepted responses remain the documented loss/duplication limits.

Related end-to-end example: buildkite/test-engine-client-examples#68

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7aa65037d5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[buildsworth-bk-app]

Three issues need addressing before this is ready: CI cannot resolve the dependencies at the advertised Swift floor, OTLP endpoint selection can send the suite token to an unrelated collector, and child-span forwarding is ineffective on Linux. Details are inline.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20842, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

About buildsworth

Model: gpt-5.6-sol with xhigh thinking.

How to request a review: Comment @buildsworth-bk review on the PR, or request buildsworth-bk as a reviewer.

Risk labels (how buildsworth classifies risk) — buildsworth classifies risk itself from the diff. Unless repository policy already allows L2 approval, grant it by mentioning @buildsworth-bk (see approval ceiling and L2 approval grant):

• L1 — Low risk (dep bumps, docs/copy, lockfiles, small presentational fixes). buildsworth may approve by default.
• L2 — Standard risk (new UI, additive API fields, refactors). Approved only when repository policy or a verified grant allows it; otherwise comment-only.
• L3 — High risk (auth, migrations, payments, secrets, perf-critical paths). Human review always required.

[buildsworth-bk-app]

The three previous blockers remain open; the Swift 6.0 dependency adjustment still fails in build #69. I also found a duplicate-retry path introduced by the new processor queue.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20860, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

[buildsworth-bk-app]

The latest transport wrapper keeps failed-root retries owned by one layer, but the three earlier blockers remain open; I’ve updated their existing threads.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20869, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

[buildsworth-bk-app]

The Swift 6.0 dependency floor now resolves and build #73 passed. The OTLP credential and Linux child-span blockers remain open, and I found one additional provider-lifecycle blocker inline.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20873, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

[buildsworth-bk-app]

The three existing endpoint/context/provider blockers remain open; I’ve updated those threads. The new Swift 5.10 compatibility change also selects a dependency that Swift 5.10 cannot compile.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20875, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

[buildsworth-bk-app]

The endpoint-credential and Linux-context fixes are resolved. Two blockers remain in the existing threads; build #75 passed, but it only exercises Swift 6.0.3.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20879, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

[buildsworth-bk-app]

The late-provider forwarding blocker is fixed and covered by the new regression test. The existing Swift 5.10 compatibility blocker remains open; build #76 still only exercises Swift 6.0.3.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20884, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

Previous concerns have been addressed.

[buildsworth-bk-app]

The last open concern is resolved: build #77 passed the test and release-build commands under both Swift 5.10.1 and 6.0.3. I’m leaving this comment-only because the change replaces the execution transport and changes suite-token routing, which is outside the repository’s L1 approval ceiling.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20885, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

[pda]

🤖 Review (Amp / Fable 5 high)

I read the full diff, compared it line-by-line with Ruby's otel_only implementation, verified the opentelemetry-swift 2.2.0 dependency facts upstream, and ran the PR's test suite locally (33/33 pass on Swift 6.3.2 / macOS).

Verdict

Solid, well-designed port of the Ruby OTel approach — arguably better in places — but it will break some existing users automatically, because the breakage ships through SwiftPM's default version range, and the changes to toolchain floor, deployment target, and dependency graph are all breaking.

1. Likely to break existing users?

The README tells consumers to use .package(url: ..., from: "0.6.0"), and SwiftPM's upToNextMajor(from: 0.6.0) resolves to 0.6.0..<1.0.0 — SwiftPM does not treat 0.x minors as breaking. So a 0.7.0 release flows automatically into any CI job that resolves fresh (no committed Package.resolved). Users hit by that:

• Xcode < 15.3 / Swift < 5.10: Package.swift went from tools 5.7.1 → 5.10. Older toolchains can't even parse the manifest. Xcode 15.3 is March 2024, and Apple has required Xcode 16 for App Store submissions since April 2025, so actively maintained iOS apps are fine — but a stale CI pin isn't.
• macOS deployment target < 12: floor rises from 10.15 → 12. Any consumer package declaring .macOS("10.15")/11 fails to build against the dependency.
• Dependency-graph collisions: previously this package had zero dependencies. Now it pins opentelemetry-swift exact 2.2.0, opentelemetry-swift-core exact 2.2.0, and swift-metrics exact 2.7.1 — and the otel umbrella itself pins grpc-swift exact 1.26.1 plus swift-nio, swift-protobuf, swift-log (and Thrift/opentracing-objc on Darwin) at the resolution level, even though the HTTP exporter build graph doesn't compile the gRPC bits. Any consumer whose graph contains grpc-swift at another version, or opentelemetry-swift at any version other than exactly 2.2.0 (e.g. via Honeycomb/Embrace/Elastic mobile SDKs), gets a hard resolution failure. The exact pins are understandable (2.2.1+ pulls a grpc-swift whose manifest needs Swift 6.1 tools — verified upstream) but they're maximally conflict-prone.
• BUILDKITE_ANALYTICS_BASE_URL is silently ignored (EnvironmentValues.swift drops it; BUILDKITE_ANALYTICS_OTLP_ENDPOINT replaces it). Anyone proxying/redirecting uploads keeps "working" while sending to the default endpoint. A warning when the old var is set would be cheap and kind.
• New egress host: tests-otlp.buildkite.com instead of analytics-api.buildkite.com. Locked-down CI networks with allowlists break; README/CHANGELOG could call this out explicitly.
• Behavioral: synchronous per-test export. SynchronousExecutionSpanProcessor blocks each testCaseDidFinish on an HTTP round-trip (10s timeout), and on failure the pending batch is retried synchronously at every subsequent test end. With an unreachable/slow endpoint that's up to ~10s × N tests of added wall time, where the old collector uploaded asynchronously. This is the intended resilience tradeoff for the TEC relay, but it converts "analytics endpoint is down" into "everyone's suite is 10s/test slower". Consider a circuit breaker (e.g. stop retrying per-test after k consecutive failures, retry only at bundle end).

Public API is handled well: load, tagExecution, annotate unchanged; baseURL gets a deprecation shim. One nit: TestCollector.version still says "0.6.0".

Recommendation: given the auto-flow through from: "0.6.0", either release this as 1.0.0 — <1.0.0 excludes it, so existing users opt in deliberately — or accept the auto-upgrade and document the Xcode/macOS/network requirements loudly in the release notes.

2. Parity with Ruby's otel_only?

Very close — a faithful port, and in some ways a superset. Matching: test.execution parentless root spans on a private AlwaysOn provider; buildkite.execution.via=otlp; buildkite.test.scope/name, test.case.name, test.suite.name, buildkite.test.execution.external_id; buildkite.tag.* on spans and resource; identical resource vocabulary (buildkite.run_key, vcs.ref.head.*, service.*, collector name/version, framework name); job-span links from TRACEPARENT; pass/fail/skipped result values; failure as span status + semconv exception events; test.annotation events; child-span forwarding that only exports spans belonging to execution traces, attaching to a suite-owned SDK provider without replacing its sampler/exporters; Buildkite-Tests-Run-Key + Token token="..." auth; env OTLP header merging.

Deliberate/structural differences worth knowing:

• Ruby's otel_only is opt-in per suite; this PR is a full cutover — no legacy fallback. It's the equivalent of forcing otel_only on every Swift user.
• Root delivery: Ruby uses an async BatchSpanProcessor with a dropped-span warning; Swift exports each root synchronously with an in-process retry queue. Intentional (XCTest runner restarts), matches the PR's stated goal.
• Swift is stricter about credentials: it withholds the suite token from user-configured OTEL_* endpoints; Ruby sends its headers to whatever endpoint is configured. Swift also honors OTEL_EXPORTER_OTLP[_TRACES]_ENDPOINT/PROTOCOL, which Ruby's mode doesn't.
• code.file.path/code.line.number only on failure in Swift (XCTest doesn't expose source locations for passing tests); Ruby sets them always. Gap is inherent to XCTest.
• No buildkite.test.framework.version (Ruby reports the RSpec version). Minor.
• Ruby's ecosystem defenses have no Swift analog where one may be needed: VCR exemption. The Swift equivalent hazard is real — suites using OHHTTPStubs/Mocker (which swizzle URLSessionConfiguration.protocolClasses globally) will intercept the collector's export. Worth a README note or a defensive check.
• Ruby's SecureRandom ID generator, warm-worker token refresh, and run-mismatch warnings are Ruby-specific (seeded PRNG, forking test pools) and reasonably omitted.

3. Sensible range of Swift/XCTest/XCUITest versions?

Mostly yes, verified rather than assumed:

• opentelemetry-swift/-core 2.2.0 manifests declare tools 5.9 and platforms macOS 12 / iOS 13 / tvOS 13 / watchOS 6 — the PR's platform claims exactly match. Everything in the HTTP-exporter chain builds with Swift 5.10.
• CI matrix covers Linux Swift 5.10.1 and 6.0.3; resolution + all 33 tests also pass on Swift 6.3.2 (current Xcode era) locally. So 5.10 → 6.3 is covered in practice.
• Gap: this monorepo pipeline is Linux-only — .buildkite/pipeline.yml explicitly shadows the source repo's macOS/Xcode queue. Nothing here exercises Xcode, iOS simulator, or an actual XCUITest bundle, which is precisely where this collector's users live (env-var plumbing through test plans, UI test runner process behavior). The linked test-engine-client-examples PR presumably covers that end-to-end, but it's outside this repo's CI.
• Forward-compat is frozen: stuck on otel 2.2.0 until Swift 5.10/6.0 support is dropped (2.2.1+ needs Swift 6.1 tools via grpc-swift 1.27). Fine today; needs an explicit plan before it bit-rots.
• XCUITest nuance worth documenting: child-span forwarding only sees spans created in the test runner process. For unit tests hosted in an app that's the app's spans; for XCUITest, the app under test is a separate process, so its OTel spans will never appear as children — users only get execution roots there.

4. Conflicts with existing OpenTelemetry tracing in Swift codebases?

Direct opentelemetry-swift usage in app code is uncommon — iOS teams mostly get it indirectly via vendor SDKs (Honeycomb, Embrace, Elastic, Splunk RUM distributions all build on opentelemetry-swift), and server-side Swift overwhelmingly uses swift-otel (swift-distributed-tracing based), which is a completely separate library the collector neither conflicts with nor captures spans from. So the population is small but nonzero, and growing on mobile.

Where conflicts actually bite, in order of severity:

1. SwiftPM resolution, not runtime. The exact 2.2.0 pin means any consumer whose graph wants a different opentelemetry-swift (every vendor SDK pins its own range) fails to resolve at all — the whole project, since app and test-target dependencies resolve as one graph. This is the most likely real-world failure and it's loud, not subtle.
2. Global mutation of OpenTelemetry.instance. The collector replaces the global context manager (wrapping the existing provider as delegate) and registers a tracer provider if none exists. The handling is careful — an existing SDK provider gets the forwarding processor added without touching its sampler/exporters, late registration is detected and handled (there's a test for it), and an unsupported provider degrades to a logged error with roots still exported. But if the suite registers its own context manager after collector load, child capture silently degrades. Acceptable, matches Ruby's posture.
3. The child forwarder only exports spans belonging to execution traces, so an app's own exporters/sampling are untouched — same-trace filtering mirrors Ruby's ExecutionChildForwarder exactly.

Net: three things before merge — a release-versioning decision (1.0.0 vs documented auto-upgrade), a warning for the now-ignored BUILDKITE_ANALYTICS_BASE_URL, and a decision on the unbounded per-test synchronous retry when the endpoint is down. The rest is documentation polish.

[pda]

🤖 Delta review: 08e2099 → c00e7b6 (Amp / Fable 5 high)

This delta addresses all three pre-merge items from my previous review. I re-verified locally: the full suite passes at c00e7b6 on Swift 6.3.2 / macOS, including the new SynchronousExecutionSpanProcessorTests and the legacy-endpoint warning test. All three resolutions are sound; nothing blocking remains. Findings below are minor.

1. Versioning → 2.0.0-beta.1 (0903df8) ✅

Resolves the auto-upgrade hazard twice over: 2.0.0-beta.1 is outside the from: "0.6.0" (<1.0.0) range, and SwiftPM never selects prerelease versions from from: ranges at all. Existing consumers cannot be upgraded into this accidentally.

Follow-up for release time (not this PR): because from: ranges never match prereleases, beta adopters must pin .package(..., exact: "2.0.0-beta.1") — the README's install section should show that when the beta ships. The current note acknowledging the example still references 0.6.0 covers the interim.

2. BUILDKITE_ANALYTICS_BASE_URL warning (4977287) ✅

Correctly placed: fires whenever analytics is enabled and the variable is set, independent of whether a token is present, and the message names both replacements. Test asserts the exact rendered log line.

Nit: the new Logger.Level.warning uses rawValue 3 with a comment saying values are based on swift-log, but swift-log's warning is 4 (3 is notice). Ordering is still correct (info < warning < error), so purely cosmetic.

3. Failed-export circuit breaker (c00e7b6) ✅

I traced the semantics against the new test and they match the described design:

• First failure arms a 10s window; test ends during a window append to the in-memory queue and return without any network I/O — no per-test blocking during an outage.
• The first test end at/after window expiry makes one blocking attempt with the full accumulated batch; repeated failures escalate 10 → 20 → 30 → 60s, capped at 60.
• Success clears the queue and fully resets the backoff.
• flush()/shutdown() bypass the backoff for an immediate final attempt, so the bundle-end flush always gets one last shot before the runner exits — preserving the TEC-relay handoff goal.

Worst-case added wall time during a sustained outage is now roughly one HTTP timeout (10s) per elapsed backoff window rather than per test. This resolves my original concern. The test coverage is thorough — it pins both the batch accumulation ([1, 4, 6, 8, 10, 12, 1]) and every window boundary (9/10, 29/30, 59/60, 119/120, 179/180), and the injected clock plus systemUptime default (monotonic, immune to wall-clock steps) are the right choices.

Minor, non-blocking observations:

• Silent queueing during backoff: a test that finishes inside a backoff window produces no log output at any level. A debug-level line ("execution queued during export backoff, N pending") would help when supporting users with missing results, since the existing per-export error only fires on actual attempts.
• Backoff reset not directly asserted: the test ends with success → one fresh export; a failure after the success would pin that the delay restarts at 10s rather than staying at 60. The code plainly does reset (retryDelayIndex = 0), so this is test-completeness only.
• The pending queue remains unbounded in process memory during a sustained outage — already documented as a known limit and fine at test-suite scale.
• README title dropped "(Beta)" while the release itself is a beta; assuming that's intentional (the note explains the release status), just flagging.

Previous concerns have been addressed.

[buildsworth-bk-app]

The prior blockers remain resolved, but a standard-header-only configuration can still send a non-Buildkite credential to the Buildkite endpoint; details are inline.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20916, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

[buildsworth-bk-app]

The existing OTLP header-routing blocker remains open; the latest revision only changes export-backoff diagnostics and tests.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20917, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

Previous concerns have been addressed.

[buildsworth-bk-app]

The standard-header routing concern is fixed. One new endpoint-configuration blocker remains inline: standard protocol variables can still disable a trusted Buildkite export.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 21112, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

[buildsworth-bk-app]

The existing endpoint-configuration blocker remains open; no new findings this pass.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 21489, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

Previous concerns have been addressed.

[buildsworth-bk-app]

The endpoint-configuration blocker is fixed, and I found no new issues. This remains comment-only because the change replaces the execution transport and changes suite-token routing, beyond the repository’s L1 approval ceiling.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 21494, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

[pda]

🤖 Rebased swift-otel-executions onto latest origin/main (c8d234b) and force-updated the PR branch; final head is 497c5a8.

origin/main advanced once while the rebase was in progress, so I replayed the branch again onto the newer head. The only conflict was test-collector-swift/.buildkite/pipeline.yml during the first rebase. I preserved both sides’ intent:

• kept main’s hosted macOS coverage on Xcode 15.4, 16.4, and 26.2;
• kept this PR’s supported Linux matrix on Swift 5.10.1 and 6.0.3.

The final replay onto c8d234b was conflict-free, and git range-diff confirms all 17 PR commits are otherwise unchanged. Post-conflict verification passed: all 38 Swift tests and the release build.

Previous concerns have been addressed.

[buildsworth-bk-app]

The prior endpoint-configuration concern remains resolved. One new test-isolation blocker is reproducible across all three macOS CI jobs; details are inline.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 21500, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

[buildsworth-bk-app]

Blocking: Build #104 fails this test on all three macOS jobs because EnvironmentValues(values: [:]) falls back to the runner's actual environment; that supplies exporter configuration, so the new XCTAssertNil is false. Please isolate this no-configuration case from both environment and Info.plist lookups.

Suggested change

	let environment = EnvironmentValues(values: [:])
	let environment = EnvironmentValues(
	values: [:],
	getFromEnvironment: { _ in nil },
	getFromInfoDictionary: { _ in nil }
	)