feat(otel): initialize OpenTelemetry tracing and metrics in boot()

[fubezz]

Summary

• Adds otel_initialize() in a new src/aignostics_foundry_core/otel.py, following the exact same pattern as sentry_initialize(): pydantic-settings-driven enabled flag (env-prefixed per FoundryContext), graceful no-op when the SDK isn't available or isn't enabled.
• Sets up a process-wide TracerProvider/MeterProvider exporting via OTLP/gRPC, using the standard, unprefixed OpenTelemetry env vars (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME, etc.) rather than reinventing project-specific ones — OTEL_SERVICE_NAME defaults to the FoundryContext project name if not explicitly set.
• Logs: this project uses loguru, not stdlib logging — loguru has no first-party OTel integration. Added a loguru sink (_make_otel_log_sink) that converts each loguru record into a stdlib logging.LogRecord and forwards it to OTel's own LoggingHandler, reusing the SDK's own conversion/trace-correlation logic instead of reimplementing OTLP log export. Gated behind a separate logs_enabled setting, off by default even when enabled is true — log volume/cost characteristics differ from traces/metrics, so it shouldn't be forced on every service that turns on tracing.
• Wires otel_initialize() into boot(), right after sentry_initialize() — every Foundry service gets this via a single core version bump, no per-service code changes for process-level tracing/metrics(/logs, opt-in).
• Also exports instrument_fastapi(app) for request-level tracing, since boot() runs before any FastAPI app instance exists.

Related: OP-3108 (expose an OTel endpoint for Foundry Cloud Run services — this is the service-side counterpart, decided against pushing this logic into the foundry-python template since it's not an importable runtime dependency there).

Known follow-up (not done in this PR)

instrument_fastapi(app) needs to actually be called somewhere — but the FastAPI app instance is constructed in Service.api_serve(), which lives in the foundry-python template (not this repo), not in boot(). This PR ships the capability; wiring the one-line call into the template is a separate, small follow-up in foundry-python. Until that lands, request-level FastAPI auto-tracing (spans per HTTP request) isn't automatic — process-level tracing/metrics/logs (manually-created spans, outbound instrumentation, log records) work regardless.

Test plan

• mise run lint — ruff, pyright, deptry all pass.
• mise run test_unit — 13 passed, otel.py at 100% coverage.
• mise run test_integration — 100 passed, otel.py at 100% coverage.
• mise run pre_commit_run_all — all hooks pass.
• Unit tests for the loguru→OTel log sink (level mapping including loguru-only TRACE/SUCCESS, message/name/function conversion, exception forwarding) and for the logs_enabled on/off gating.
• Manual smoke test against a real OTLP endpoint (e.g. once aignx-otel-gateway.aignostics.ai from the gitops side of OP-3108 is live) — not done here, this PR is unit/integration-tested only.

Opened as draft pending review of the design (in particular: the loguru→OTel logs bridge approach and its opt-in gating, and confirmation on the instrument_fastapi follow-up plan).

🤖 Generated with Claude Code

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
441	1	440	0

View the top 1 failed test(s) by shortest run time

tests.aignostics_foundry_core.otel_test.TestInstrumentFastapi::test_instrument_fastapi_returns_true_and_instruments_app

Stack Traces | 0.119s run time

self = <tests.aignostics_foundry_core.otel_test.TestInstrumentFastapi object at 0x7f782d8964d0>

    def test_instrument_fastapi_returns_true_and_instruments_app(self) -> None:
        """Returns True and calls FastAPIInstrumentor.instrument_app with the given app."""
        mock_app = MagicMock()
>       with patch("opentelemetry.instrumentation.fastapi.FastAPIInstrumentor.instrument_app") as mock_instrument:

tests/aignostics_foundry_core/otel_test.py:268: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../..../uv/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/unittest/mock.py:1430: in __enter__
    self.target = self.getter()
                  ^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

name = 'opentelemetry.instrumentation.fastapi.FastAPIInstrumentor'

    def resolve_name(name):
        """
        Resolve a name to an object.
    
        It is expected that `name` will be a string in one of the following
        formats, where W is shorthand for a valid Python identifier and dot stands
        for a literal period in these pseudo-regexes:
    
        W(.W)*
        W(.W)*:(W(.W)*)?
    
        The first form is intended for backward compatibility only. It assumes that
        some part of the dotted name is a package, and the rest is an object
        somewhere within that package, possibly nested inside other objects.
        Because the place where the package stops and the object hierarchy starts
        can't be inferred by inspection, repeated attempts to import must be done
        with this form.
    
        In the second form, the caller makes the division point clear through the
        provision of a single colon: the dotted name to the left of the colon is a
        package to be imported, and the dotted name to the right is the object
        hierarchy within that package. Only one import is needed in this form. If
        it ends with the colon, then a module object is returned.
    
        The function will return an object (which might be a module), or raise one
        of the following exceptions:
    
        ValueError - if `name` isn't in a recognised format
        ImportError - if an import failed when it shouldn't have
        AttributeError - if a failure occurred when traversing the object hierarchy
                         within the imported package to get to the desired object.
        """
        global _NAME_PATTERN
        if _NAME_PATTERN is None:
            # Lazy import to speedup Python startup time
            import re
            dotted_words = r'(?!\d)(\w+)(\.(?!\d)(\w+))*'
            _NAME_PATTERN = re.compile(f'^(?P<pkg>{dotted_words})'
                                       f'(?P<cln>:(?P<obj>{dotted_words})?)?$',
                                       re.UNICODE)
    
        m = _NAME_PATTERN.match(name)
        if not m:
            raise ValueError(f'invalid format: {name!r}')
        gd = m.groupdict()
        if gd.get('cln'):
            # there is a colon - a one-step import is all that's needed
            mod = importlib.import_module(gd['pkg'])
            parts = gd.get('obj')
            parts = parts.split('.') if parts else []
        else:
            # no colon - have to iterate to find the package boundary
            parts = name.split('.')
            modname = parts.pop(0)
            # first part *must* be a module/package.
            mod = importlib.import_module(modname)
            while parts:
                p = parts[0]
                s = f'{modname}.{p}'
                try:
                    mod = importlib.import_module(s)
                    parts.pop(0)
                    modname = s
                except ImportError:
                    break
        # if we reach this point, mod is the module, already imported, and
        # parts is the list of parts in the object hierarchy to be traversed, or
        # an empty list if just the module is wanted.
        result = mod
        for p in parts:
>           result = getattr(result, p)
                     ^^^^^^^^^^^^^^^^^^
E           AttributeError: module 'opentelemetry.instrumentation' has no attribute 'fastapi'

../../../..../uv/python/cpython-3.11.15-linux-x86_64-gnu/lib/python3.11/pkgutil.py:715: AttributeError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[fubezz]

Suggested change

	otel_initialize(context=ctx)
	_otel_initialize(context=ctx)

[fubezz]

Suggested change

	def otel_initialize(
	def _otel_initialize(

[fubezz]

Not used?

[fubezz]

Notes from testing this against a locally-generated Foundry service (OP-3108, sandbox). A few design points worth considering before merge, none blocking:

1. otel_initialize() has no equivalent to SENTRY_INTEGRATIONS

sentry_initialize() takes an explicit integrations: list[Integration] | None built by the generated project's own constants.py (try/except-imported per optional dep, e.g. FastApiIntegration, TyperIntegration). otel_initialize() has no equivalent — it only sets up the bare TracerProvider/MeterProvider/LoggerProvider. Everything downstream of the inbound HTTP request (DB queries, outbound httpx calls, Redis, etc.) is invisible unless something calls the relevant opentelemetry-instrumentation-* package's .instrument().

Most of those packages subclass a common BaseInstrumentor with a no-argument .instrument() — genuinely Sentry-integration-shaped, so the same pattern would fit:

# otel.py
def otel_initialize(*, context=None, instrumentors: list[BaseInstrumentor] | None = None) -> bool:
    ...
    for instrumentor in instrumentors or []:
        instrumentor.instrument()

with a project's constants.py building OTEL_INSTRUMENTORS = [HTTPXClientInstrumentor(), SQLAlchemyInstrumentor()] the same way it builds SENTRY_INTEGRATIONS today.

Concretely, HTTPXClientInstrumentor is the one I'd prioritize: without it, an outbound httpx call to another Foundry service doesn't get the W3C traceparent header injected, so the callee's span starts a new disconnected trace instead of a child — which undercuts the actual point of this ADR (getting Cloud Run into the same distributed-tracing picture as everything else).

FastAPIInstrumentor stays exactly as it is (instrument_fastapi(app), called separately from Service.api()) — it's the odd one out because ASGI middleware has to wrap the live app instance, which doesn't exist yet at boot() time. That's a real API shape difference, not an oversight, so full parity with the Sentry list isn't achievable — only partial.

Separately, worth a doc note (in otel_initialize()'s docstring or the ADR's consequences section): manual/business-level spans and custom metrics already work today, no signature change needed — trace.get_tracer(__name__) / metrics.get_meter(__name__) just work anywhere once otel_initialize() has set the global providers.

2. boot() silently discards the init result

otel_initialize() and sentry_initialize() both return bool (whether they actually initialized vs. no-op'd because the master switch was off or the endpoint/DSN was missing), but boot() doesn't do anything with either return value. Today there's no log line telling you at runtime whether OTel/Sentry actually came up. Appending something like , otel: {on/off}, sentry: {on/off} to the existing _log_boot_message() call would close that gap cheaply, using data that's already computed.

3. Considered and rejected: master enabled switch as a boot() parameter

To make it "visible in code" rather than purely env-driven. Checked SentrySettings.enabled — it's the exact same env-only pattern as OTelSettings.enabled (no boot() param for it either), so promoting only OTel's switch to a boot() arg would break symmetry with Sentry rather than restore it, and risks a second source of truth (hardcode True in generated code but still gate on env, or duplicate the env read outside OTelSettings). Point 2 above (log the actual init result) gets the same visibility without the architecture change.