Skip to content

Generate KnownTags from a tag-conventions registry#11961

Draft
dougqh wants to merge 1 commit into
dougqh/dense-bloom-filterfrom
dougqh/tag-registry-generator
Draft

Generate KnownTags from a tag-conventions registry#11961
dougqh wants to merge 1 commit into
dougqh/dense-bloom-filterfrom
dougqh/tag-registry-generator

Conversation

@dougqh

@dougqh dougqh commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Code-generates the KnownTags registry from a language-agnostic tag-conventions.yaml (+ a Java
overlay), replacing the hand-maintained version. Based on dougqh/tag-id-api (the dense-store
branch). Draft — the generator works end to end (main compiles, tests pass); the committed-output
wiring is the remaining piece.

From Claude:

Pipeline (buildSrc task dd-trace-java.tag-registry-generator): parse → resolve per-type tag sets
(extends/include/applies) → assign ids → assign slots by deterministic graph coloring over
per-type layers + an explicit trace_level layer → emit KnownTags.java.

  • Ids encode serial | slot | intercepted (no name hash — known ids are dense-store addressed;
    the name hash is the custom/bucket path). Emitted as literals (javac constant-folds them) with a
    // tagId(...) derivation comment for auditability.
  • slot is a fast-path bitmask bit, not positional storage — co-occurrence is tolerated (the scan
    is authoritative). Coloring gives co-occurring tags distinct bits (fewer scans); colorability
    derives from required. The whole mask fits one long.
  • Trace-level is its own tier (trace_level section → its own TagMap layer); the per-span base
    type holds parent_id/_dd.integration/_dd.svc_src (core-set but per-span).
  • Virtual keys (service/resource.name/error/sampling.priority/…) get reserved ids; the
    generated id → handler dispatch table is the on-ramp to retiring the TagInterceptor.

Verified: internal-api main compiles; KnownTagsTest (property-based resolver round-trip over ~55
tags) passes; OptimizedTagMapDenseForkedTest passes; spotless clean. Note: literal ids fold, so a
constant reference no longer triggers <clinit> — registration is via KnownTags.init().

Remaining (why draft): the generated KnownTags.java is a manual copy in src/main right now;
wire the task to emit to a committed src/generated/java + srcDir + a verifyGenerated freshness
check + format-on-generate, and route the layout reports (folded-types/tag-assignment) there as
committed artifacts.

🤖 Generated with Claude Code

@dougqh dougqh added comp: core Tracer core comp: tooling Build & Tooling tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: refactoring labels Jul 15, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.05 s 13.98 s [-0.1%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent 12.94 s 13.09 s [-2.2%; -0.0%] (maybe better)
startup:petclinic:appsec:Agent 16.93 s 16.74 s [-0.2%; +2.4%] (no difference)
startup:petclinic:iast:Agent 16.86 s 16.96 s [-1.6%; +0.4%] (no difference)
startup:petclinic:profiling:Agent 16.21 s 16.70 s [-7.4%; +1.6%] (no difference)
startup:petclinic:sca:Agent 16.84 s 16.73 s [-0.4%; +1.7%] (no difference)
startup:petclinic:tracing:Agent 16.07 s 15.73 s [-2.2%; +6.6%] (no difference)

Commit: ad3c1f70 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@dougqh dougqh changed the base branch from dougqh/tag-id-api to dougqh/dense-bloom-filter July 15, 2026 21:07
@dougqh dougqh force-pushed the dougqh/tag-registry-generator branch from 6a83a99 to fdf3c4a Compare July 15, 2026 21:07
@dougqh dougqh force-pushed the dougqh/dense-bloom-filter branch from 267ebad to 56550cb Compare July 15, 2026 21:23
@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Jul 15, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 97.40%
Overall Coverage: 57.31% (-0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ad3c1f7 | Docs | Datadog PR Page | Give us feedback!

Replace the hand-written KnownTags with a build-time code generator: a
language-agnostic tag-conventions.yaml (span types via extends/include/applies)
+ a Java overlay (tag-conventions.java.yaml: intercepted hints + the virtual
registry) drive a buildSrc generator that emits KnownTags.java (id constants +
keyOf/nameOf resolver) to committed src/generated. A verifyKnownTags check
regenerates and byte-compares to catch stale commits; src/generated is excluded
from spotless (the emitter produces google-java-format-clean output).

- buildSrc: TagRegistryGeneratorPlugin + generate/verify tasks + parse/resolve
  (TagConventions) + id-assignment/coloring (TagRegistry) + emit (KnownTagsEmitter).
- KnownTagCodec gains the name-free 3-arg tagId(globalSerial, intercepted, slot)
  the generator uses.
- Generated per-type layout reports (resolved/tag-assignment/layout-by-type/
  folded-types) committed as auditable artifacts — the per-type counts are the
  substrate the (follow-on) per-type dense-array sizing reads.

Substrate only: this lands the generated ids/resolver + the codegen source of
truth; it does NOT yet change dense-array sizing (that per-type-sizing step is
the follow-on that climbs out of the alloc valley). Reconciled onto the dense +
bloom stack; generated ids cohere with the dense store (forked/fuzz green).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dougqh dougqh force-pushed the dougqh/tag-registry-generator branch from fdf3c4a to ad3c1f7 Compare July 15, 2026 21:25
dougqh added a commit that referenced this pull request Jul 15, 2026
set(long id, Object value) is the id-keyed insertion path: the caller passes a
resolved KnownTags id, so it skips the keyOf name resolution the set(String, ...)
methods pay and stores densely. The id must be a stored known-tag id; custom
names have no id and use the name-keyed setters. The name is resolved lazily only
to clear a read-through tombstone (rare).

Adds TagMapInsertionComparisonBenchmark (dense vs HashMap insertion, -prof gc) —
the isolated micro where the dense store + bloom insertion win shows, and the
home for the id-keyed vs string-keyed comparison.

Reconciled onto the tag registry (#11961).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core comp: tooling Build & Tooling tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant