Skip to content

Commit c4473cc

Browse files
sanchitrkclaude
andauthored
feat: export OpenTelemetry traces and metrics over OTLP (#74)
* feat: export OpenTelemetry traces and metrics over OTLP Every CLI command can export the harness's spans and metrics through Effect's built-in OTLP exporter, configured only by the standard OTEL_* environment variables. Without an endpoint nothing is exported, and a configuration that fails to build runs the command without export. - harness: GenAI attributes on the LLM, tool and agent spans; a failed provider request now ends its span with an error status. - harness: metrics for token usage, request/tool/agent duration, time to first token, time per output token, cost, retries, sandbox provisioning and tool output truncation, plus Effect's fiber runtime metrics. - cli: per-signal http/protobuf (default) or http/json, OTLP exporters default to otlp as the spec says, and resource attributes for the instance, release channel and command. - E2E test against a fake provider and collector, snapshotted to __artifacts__/telemetry.json. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: changelog for OpenTelemetry export Also adds the blank line the formatter expects before the Internal heading. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: name telemetry metrics under codework.* The GenAI semantic conventions are still in development and have renamed their metrics more than once, so metrics now use codework's own names and short labels; spans keep the gen_ai.* attributes trace views read. - codework.llm.{duration,ttft,tokens,cost}, codework.agent.duration, codework.tool.duration, codework.sandbox.{provision,exec}.duration and codework.sandbox.active, plus Effect's fiber runtime metrics. - Spans: gen_ai.usage.cache_write/reasoning.output_tokens names, conversation id, stream flag, time to first chunk, agent name. - Drop the retries and truncation counters and per-signal protocol vars. - Test drives a bash call so the sandbox exec metric is covered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: export only high-level spans, CODEWORK_TRACE_LEVEL for the rest A single run produced ~500 spans, a third of them Effect's own sql.execute. Spans now default to Debug and export starts at Info. The command, agent loop, model and tool calls, and sandbox create/wake are marked Info, so a run exports about a dozen spans. CODEWORK_TRACE_LEVEL=debug exports every span again. Effect drops a span's children with it, so every span on the path from a command to a model or tool call is Info, Loop.settleTools included. The E2E test pins the exported span tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(harness): probe instruction files with exists, not stat Most instruction candidates are missing, and a stat on a missing path fails its span even when the caller treats absence as normal: every debug trace carried ~54 SandboxFileSystem.stat errors per run. Candidates are now probed with exists, which answers absence without failing, and only the files that exist are stat'ed. Not-found stays silent and other failures still warn on every driver. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: address telemetry review findings - Export sandbox provisioning: CLI.selectSandbox and Sandbox.resolve are Info, and the server RPCs that provision a sandbox start their own Info trace, since RpcServer's Debug request span would drop them. - Record tokens and cost of an interrupted model call from the terminal the consumer drains. - E2E covers CODEWORK_TRACE_LEVEL=debug; the instruction test asserts stat only runs for existing files. - Drop comments that restated the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent d1cf9b0 commit c4473cc

19 files changed

Lines changed: 1198 additions & 77 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,11 @@ This file is the canonical source for unreleased changes and published release n
1111

1212
## [Unreleased]
1313

14+
### Added
15+
16+
- `@codeworksh/cli`: Added OpenTelemetry export. Set `OTEL_EXPORTER_OTLP_ENDPOINT` and every command sends its traces and metrics over OTLP/HTTP, as protobuf by default or JSON with `OTEL_EXPORTER_OTLP_PROTOCOL=http/json`. The standard `OTEL_*` variables configure it (headers, per-signal endpoints, `OTEL_SDK_DISABLED`, `OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES`); there is no settings key. Without an endpoint nothing is exported, and a bad configuration never stops a command. Only the high-level spans are exported (command, agent loop, model and tool calls, sandbox create and wake); set `CODEWORK_TRACE_LEVEL=debug` to export every span, including database queries.
17+
- `@codeworksh/harness`: Spans for model requests, tool calls and agent runs carry the OpenTelemetry GenAI attributes (provider, model, session, response id, finish reason, token counts, time to first token), and a failed model request ends its span with an error. Metrics: `codework.llm.duration`, `codework.llm.ttft`, `codework.llm.tokens` (by `type`: input, output, cache_read, cache_write, reasoning), `codework.llm.cost`, `codework.agent.duration`, `codework.tool.duration` (by `tool`), `codework.sandbox.provision.duration`, `codework.sandbox.exec.duration` and `codework.sandbox.active` (by `driver`), plus Effect's fiber runtime metrics (`child_fibers_*`). Prompts, file contents and model output are never exported.
18+
1419
### Fixed
1520

1621
- `@codeworksh/plugin`: Fixed tools whose parameters are `Schema.Struct({})` being rejected by OpenAI and Anthropic. Their JSON Schema is now `{ "type": "object", "properties": {} }` instead of `{ "not": { "type": "null" } }`.
@@ -20,6 +25,7 @@ This file is the canonical source for unreleased changes and published release n
2025

2126
### Internal
2227

28+
- Instruction-file discovery probes candidates with `exists` and only stats the files that exist, so a missing `AGENTS.md` or `CLAUDE.md` no longer records a failed span in debug traces.
2329
- The `codework tui` E2E artifact now records the CLI version as `<version>`, so it no longer goes stale on every version bump.
2430
- Tests load provider and sandbox credentials from the root `.env.local` instead of one file per package.
2531
- Added a live E2E test that runs `codework run` on a memory, Daytona and Vercel sandbox and checks the sandbox is stopped on exit.

‎packages/codework/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@
5151
"@effect/platform-node": "4.0.1",
5252
"effect": "4.0.1",
5353
"jsonc-parser": "3.3.1",
54+
"uuidv7": "^1.2.1",
5455
"zod": "4.6.5"
5556
},
5657
"devDependencies": {

‎packages/codework/src/cli/cmd/handlers/acp.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ import { Cmd } from "../cmd.ts";
77

88
export default Runtime.handler(
99
Cmd.commands.acp,
10-
Effect.fn("CLI.acp")(function* () {
10+
Effect.fn("CLI.acp", { level: "Info" })(function* () {
1111
const shared = yield* Cmd.spec;
1212
return yield* Server.serve(harnessOptions(shared)).pipe(Effect.catch(reportFailure));
1313
}),

‎packages/codework/src/cli/cmd/handlers/run.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ const awaitMessage = Effect.fn("CLI.awaitMessage")(function* (ended: Queue.Queue
6666
}
6767
});
6868

69-
const selectSandbox = Effect.fn("CLI.selectSandbox")(function* (input: Sandbox.Selection) {
69+
const selectSandbox = Effect.fn("CLI.selectSandbox", { level: "Info" })(function* (input: Sandbox.Selection) {
7070
const selection = yield* Sandbox.resolve(input);
7171
if (selection.created && selection.info !== undefined) {
7272
const id = selection.info.id;
@@ -77,7 +77,7 @@ const selectSandbox = Effect.fn("CLI.selectSandbox")(function* (input: Sandbox.S
7777

7878
export default Runtime.handler(
7979
Cmd.commands.run,
80-
Effect.fn("CLI.run")(function* ({
80+
Effect.fn("CLI.run", { level: "Info" })(function* ({
8181
prompt,
8282
session,
8383
cwd,

‎packages/codework/src/cli/cmd/handlers/serve.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ const report = (error: unknown) => writeError(renderError(error)).pipe(Effect.an
1616

1717
export default Runtime.handler(
1818
Cmd.commands.serve,
19-
Effect.fn("CLI.serve")(function* ({ host, port, service }) {
19+
Effect.fn("CLI.serve", { level: "Info" })(function* ({ host, port, service }) {
2020
const shared = yield* Cmd.spec;
2121
const harness = { ...harnessOptions(shared), watchModels: true };
2222
const serve = Layer.launch(Server.layer({ host, port, harness }));

‎packages/codework/src/cli/cmd/handlers/tui.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ const run = (binary: string, args: ReadonlyArray<string>, env: Record<string, st
7373

7474
export default Runtime.handler(
7575
Cmd.commands.tui,
76-
Effect.fn("CLI.tui")(function* ({ server, host, port, args }) {
76+
Effect.fn("CLI.tui", { level: "Info" })(function* ({ server, host, port, args }) {
7777
const shared = yield* Cmd.spec;
7878

7979
const program = Effect.gen(function* () {
Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
import { Config, ConfigProvider, Effect, Layer, Metric, Option, References, Schema } from "effect";
2+
import { FetchHttpClient } from "effect/http";
3+
import { OtlpMetrics, OtlpSerialization, OtlpTracer } from "effect/observability";
4+
import { uuidv7 } from "uuidv7";
5+
6+
// Configured by the standard `OTEL_*` variables alone. A configuration that fails
7+
// to build runs the command without export rather than failing it.
8+
9+
// The OTel spec defaults `OTEL_{TRACES,METRICS}_EXPORTER` to `otlp`; Effect reads an
10+
// unset list as none, so an endpoint alone would export nothing.
11+
const exporterDefaults = ConfigProvider.layerAdd(
12+
ConfigProvider.fromUnknown({ OTEL_TRACES_EXPORTER: ["otlp"], OTEL_METRICS_EXPORTER: ["otlp"] }),
13+
);
14+
15+
// Effect's fiber metrics hook every fiber start and end, so they follow metrics export.
16+
const metricsEnabled = Config.all({
17+
disabled: Config.Boolean("OTEL_SDK_DISABLED").pipe(Config.withDefault(false)),
18+
endpoint: Config.String("OTEL_EXPORTER_OTLP_METRICS_ENDPOINT").pipe(
19+
Config.orElse(() => Config.String("OTEL_EXPORTER_OTLP_ENDPOINT")),
20+
Config.option,
21+
),
22+
exporters: Config.Array(Schema.String, "OTEL_METRICS_EXPORTER"),
23+
}).pipe(
24+
Config.map(
25+
({ disabled, endpoint, exporters }) =>
26+
!disabled && Option.isSome(endpoint) && exporters.some((exporter) => exporter.trim().toLowerCase() === "otlp"),
27+
),
28+
);
29+
30+
const fiberMetrics = Layer.unwrap(
31+
Effect.gen(function* () {
32+
return (yield* metricsEnabled) ? Metric.enableRuntimeMetricsLayer : Layer.empty;
33+
}),
34+
);
35+
36+
// Only spans marked `level: "Info"` are exported; the rest, Effect's own `sql.*`
37+
// among them, default to `Debug`. Effect drops a span's children with it, so every
38+
// span on the path from a command to a model or tool call must be `Info`.
39+
const levels = { all: "All", trace: "Trace", debug: "Debug", info: "Info", warn: "Warn", error: "Error" } as const;
40+
41+
const traceLevels = Layer.unwrap(
42+
Effect.gen(function* () {
43+
const name = yield* Config.String("CODEWORK_TRACE_LEVEL").pipe(Config.withDefault("info"));
44+
const minimum = levels[name.trim().toLowerCase() as keyof typeof levels] ?? "Info";
45+
return Layer.mergeAll(
46+
Layer.succeed(References.CurrentTraceLevel, "Debug"),
47+
Layer.succeed(References.MinimumTraceLevel, minimum),
48+
);
49+
}),
50+
);
51+
52+
interface OtlpResource {
53+
readonly serviceName: string;
54+
readonly serviceVersion: string;
55+
readonly attributes: Record<string, string>;
56+
}
57+
58+
// Effect has no gRPC exporter: `grpc` exports nothing rather than sending HTTP to a gRPC port.
59+
const exporters = (resource: OtlpResource) =>
60+
Layer.unwrap(
61+
Effect.gen(function* () {
62+
const protocol = yield* Config.String("OTEL_EXPORTER_OTLP_PROTOCOL").pipe(Config.withDefault("http/protobuf"));
63+
const format = {
64+
"http/json": OtlpSerialization.layerJson,
65+
"http/protobuf": OtlpSerialization.layerProtobuf,
66+
}[protocol.trim().toLowerCase()];
67+
if (format === undefined) return Layer.empty;
68+
return Layer.mergeAll(
69+
OtlpTracer.layerFromConfig({ resource }),
70+
OtlpMetrics.layerFromConfig({ resource }),
71+
).pipe(Layer.provide(format));
72+
}),
73+
);
74+
75+
const channel = (version: string) =>
76+
version.includes("-canary.") ? "canary" : version.includes("-dev.") ? "dev" : "latest";
77+
78+
const instanceId = uuidv7();
79+
80+
export const layer = (options: { readonly version: string; readonly command: string }) => {
81+
const resource: OtlpResource = {
82+
serviceName: "codework",
83+
serviceVersion: options.version,
84+
attributes: {
85+
"service.instance.id": instanceId,
86+
"deployment.environment.name": channel(options.version),
87+
"codework.command": options.command,
88+
},
89+
};
90+
return Layer.mergeAll(exporters(resource), fiberMetrics, traceLevels).pipe(
91+
Layer.provide(FetchHttpClient.layer),
92+
Layer.provide(exporterDefaults),
93+
Layer.catchCause(() => Layer.empty),
94+
);
95+
};
96+
97+
export * as Otel from "./otel.ts";

‎packages/codework/src/framework/runtime.ts‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import { Effect } from "effect";
22
import { Command } from "effect/cli";
33
import type { CommandError } from "../cli/error.ts";
4+
import { Otel } from "./otel.ts";
45
import { Spec } from "./spec.ts";
56

67
/**
@@ -72,21 +73,33 @@ export function handlers<const Root extends Spec.Any>(
7273
}
7374

7475
export function run(root: Spec.Any, handlers: ReadonlyArray<LazyHandler>, options: { readonly version: string }) {
75-
return Command.run(provide(root, handlers), options);
76+
return Command.run(provide(root, handlers, { version: options.version, path: [] }), options);
7677
}
7778

78-
function provide(node: Spec.Any, handlers: ReadonlyArray<LazyHandler>): ProvidedCommand {
79+
function provide(
80+
node: Spec.Any,
81+
handlers: ReadonlyArray<LazyHandler>,
82+
context: { readonly version: string; readonly path: ReadonlyArray<string> },
83+
): ProvidedCommand {
7984
const found = handlers.find((entry) => entry.spec === node.spec);
85+
const command = context.path.length === 0 ? node.name : context.path.join(" ");
8086
const spec = found
8187
? node.spec.pipe(
8288
Command.withHandler((input) =>
83-
Effect.promise(found.load).pipe(Effect.flatMap((module) => module.default(input))),
89+
Effect.promise(found.load).pipe(
90+
Effect.flatMap((module) => module.default(input)),
91+
Effect.provide(Otel.layer({ version: context.version, command })),
92+
),
8493
),
8594
)
8695
: node.spec;
8796
const children = Object.values(node.commands);
8897
if (children.length === 0) return spec as ProvidedCommand;
89-
return spec.pipe(Command.withSubcommands(children.map((child) => provide(child, handlers)))) as ProvidedCommand;
98+
return spec.pipe(
99+
Command.withSubcommands(
100+
children.map((child) => provide(child, handlers, { ...context, path: [...context.path, child.name] })),
101+
),
102+
) as ProvidedCommand;
90103
}
91104

92105
export * as Runtime from "./runtime.ts";

‎packages/codework/src/server/handlers.ts‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,11 @@ const sessionInfo = Effect.fn("Server.sessionInfo")(function* (sessionId: Sessio
4040
return toSessionInfo(yield* found.value.info);
4141
});
4242

43+
// RpcServer's request span has no level, so it is Debug and drops every Info span under it.
44+
// Provisioning a sandbox is worth exporting, so it starts a trace of its own.
45+
const exported = <A, E, R>(name: string, self: Effect.Effect<A, E, R>) =>
46+
self.pipe(Effect.withSpan(name, { level: "Info", root: true }));
47+
4348
const releaseOnFailure = (selection: Effect.Success<ReturnType<typeof Sandbox.resolve>> | undefined) =>
4449
selection?.created && selection.info !== undefined
4550
? Sandbox.stop(selection.info.id).pipe(Effect.catchCause(Effect.logError))
@@ -61,7 +66,8 @@ export const layer = Contract.Api.toLayer(
6166
"auth.refresh": ({ provider }) => oauth.refresh(provider),
6267
"auth.logout": ({ provider }) => oauth.logout(provider),
6368
"session.create": Effect.fnUntraced(function* ({ title, directory, hostDir, sandbox, runtime }) {
64-
const selection = sandbox === undefined ? undefined : yield* Sandbox.resolve(sandbox);
69+
const selection =
70+
sandbox === undefined ? undefined : yield* exported("Server.session.create", Sandbox.resolve(sandbox));
6571
const selected = selection?.info;
6672
const handle = yield* Session.create({
6773
...(title === undefined ? {} : { title }),
@@ -93,7 +99,8 @@ export const layer = Contract.Api.toLayer(
9399
return toSessionInfo(yield* handle.info);
94100
}),
95101
"session.relink": Effect.fnUntraced(function* ({ sessionId, sandbox, directory }) {
96-
const selection = sandbox === undefined ? undefined : yield* Sandbox.resolve(sandbox);
102+
const selection =
103+
sandbox === undefined ? undefined : yield* exported("Server.session.relink", Sandbox.resolve(sandbox));
97104
const selected = selection?.info;
98105
const handle = yield* Session.relink({
99106
sessionId,
@@ -132,7 +139,7 @@ export const layer = Contract.Api.toLayer(
132139
return (yield* Sandbox.list()).map(toSandboxInfo);
133140
}),
134141
"sandbox.create": Effect.fnUntraced(function* ({ driver }) {
135-
return toSandboxInfo(yield* Sandbox.create({ driver }));
142+
return toSandboxInfo(yield* exported("Server.sandbox.create", Sandbox.create({ driver })));
136143
}),
137144
"sandbox.register": Effect.fnUntraced(function* ({ driver, providerResourceId }) {
138145
return toSandboxInfo(yield* Sandbox.register({ driver, providerResourceId }));

0 commit comments

Comments
 (0)