From 540475ac68adafcff976842ba960bcb18d9da159 Mon Sep 17 00:00:00 2001 From: m96-chan Date: Sun, 6 Sep 2026 15:53:35 +0900 Subject: [PATCH 1/3] ci: Let a re-run publish, and refuse to publish a partial set v3.0.2 is tagged, has a GitHub release, and is not on PyPI. `pip install pyetwkit` has been getting 3.0.1 since December. Two things went wrong, and the second is the dangerous one. The release run failed because the 3.14 wheel build failed, which correctly blocked publishing. Someone then re-ran the workflow manually -- and that run went green with `Publish to PyPI` skipped, because `workflow_dispatch` can never satisfy `if: github.event_name == 'release'`. The recovery path reports success while doing nothing, which is why this went unnoticed for nine months. Give `workflow_dispatch` an explicit `publish` input, default false, so a failed release can be finished without a dispatch ever uploading by accident. Also check the artifact set before uploading. PyPI does not allow a version to be re-uploaded, so publishing four wheels instead of five is not a mistake that can be corrected afterwards; count them and fail instead. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01HhPcm483PrDmuQEUSALBFn --- .github/workflows/release.yml | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 3caa71f..8b83438 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -4,6 +4,13 @@ on: release: types: [published] workflow_dispatch: + inputs: + publish: + description: >- + Upload the built artifacts to PyPI. Use this to finish a release whose + first attempt failed after the GitHub release was already published. + type: boolean + default: false env: CARGO_TERM_COLOR: always @@ -66,7 +73,13 @@ jobs: name: Publish to PyPI runs-on: ubuntu-latest needs: [build-wheels, build-sdist] - if: github.event_name == 'release' + # A re-run has to be able to publish. v3.0.2 was tagged and announced but + # never reached PyPI: one wheel build failed, which correctly blocked this + # job, and the manual re-run that followed went green while skipping it, + # because `workflow_dispatch` could never satisfy `event_name == 'release'`. + # The recovery path looked like it had worked. Opting in explicitly keeps a + # dispatch from publishing by accident while making recovery possible. + if: github.event_name == 'release' || inputs.publish environment: pypi permissions: id-token: write @@ -77,6 +90,24 @@ jobs: path: dist/ merge-multiple: true + # PyPI will not let a version be re-uploaded, so a partial set of wheels + # is not something that can be fixed afterwards. Count them first. + - name: Check the artifact set is complete + run: | + set -euo pipefail + wheels=$(find dist -name '*.whl' | wc -l) + sdists=$(find dist -name '*.tar.gz' | wc -l) + echo "wheels: $wheels, sdists: $sdists" + find dist -name '*.whl' -o -name '*.tar.gz' | sort + if [ "$wheels" -ne 5 ]; then + echo "::error::expected 5 wheels (3.10-3.14), found $wheels" + exit 1 + fi + if [ "$sdists" -ne 1 ]; then + echo "::error::expected 1 sdist, found $sdists" + exit 1 + fi + - name: Publish to PyPI uses: pypa/gh-action-pypi-publish@release/v1 with: From 852b7eeecba29091b0f4266828c8ecba3129b1e5 Mon Sep 17 00:00:00 2001 From: m96-chan Date: Sun, 6 Sep 2026 15:53:35 +0900 Subject: [PATCH 2/3] fix: Stop OtlpExporter reporting success while discarding every event `OtlpExporter.flush()` cleared the batch and returned True. There is no HTTP or gRPC client anywhere in the module, so every event was dropped and the caller was told it had been delivered. Against a closed port -- 127.0.0.1:1 -- every `export()` and `flush()` still returned True. README advertised it as a v3.0 feature, "Export events to OTLP (Jaeger, Grafana, Datadog)", with a worked example. #52 was closed as done on this code. Raise NotImplementedError from `flush()` when there is anything buffered. A monitoring pipeline that reports success while sending nothing is worse than one that stops: the failure is at least visible. `shutdown()` stays quiet, since callers reach it from `finally` and a teardown path is the wrong place to learn this. Correct the README and the docstrings to describe what exists. `OtlpFileExporter` does work and writes spans to JSON/JSONL, so the example now uses it. The changelog entry for v3.0.0 keeps its wording, with a note that the transport was never implemented -- rewriting what was announced would hide the same fact twice. The tests could not have caught this: `test_exporter_flush` asserted `hasattr(exporter, "flush")` without calling it, one of ten `hasattr`-only tests in the file. Replaced with tests that exercise the behaviour, plus a regression guard on OtlpFileExporter. Closes #88 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01HhPcm483PrDmuQEUSALBFn --- README.md | 26 +++++++-------- src/pyetwkit/exporters/otlp.py | 59 ++++++++++++++++++++++++++-------- tests/test_otlp_exporter.py | 56 +++++++++++++++++++++++++++++--- 3 files changed, 109 insertions(+), 32 deletions(-) diff --git a/README.md b/README.md index 25c3ba9..5462bd8 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ A modern, high-performance ETW (Event Tracing for Windows) toolkit for Python, p - **Live Dashboard**: Browser-based real-time visualization with Gradio - **Event Correlation Engine**: Auto-correlate events by PID/TID/Handle - **Recording & Replay**: Capture and replay ETW sessions (.etwpack format) -- **OpenTelemetry Exporter**: Export events to OTLP (Jaeger, Grafana, Datadog) +- **OpenTelemetry span mapping**: Map ETW events to OTLP spans and write them to a file. Sending to a collector is [not implemented yet](https://github.com/m96-chan/PyETWkit/issues/88) ### Export Formats - CSV, JSON, JSONL, Parquet, Arrow @@ -168,19 +168,16 @@ for event in player.events(): print(f"Event {event['event_id']}") ``` -### OpenTelemetry Export +### OpenTelemetry Span Export + +> **Sending to a collector is not implemented.** `OtlpExporter` maps events to +> spans but has no HTTP or gRPC transport, and its `flush()` raises +> `NotImplementedError` rather than reporting a delivery it cannot make. Use +> `OtlpFileExporter` to get spans out today; see +> [#88](https://github.com/m96-chan/PyETWkit/issues/88) for the transport. ```python -from pyetwkit import OtlpExporter, SpanMapper - -# Configure exporter -exporter = OtlpExporter( - endpoint="http://collector:4317", - service_name="my-service", - resource_attributes={ - "deployment.environment": "production", - }, -) +from pyetwkit import OtlpFileExporter, SpanMapper # Map ETW events to spans mapper = SpanMapper() @@ -191,7 +188,8 @@ mapper.add_rule( attributes=["ProcessId", "ImageFileName"], ) -# Export events +# Write spans to a file, ready to be shipped by a collector agent +exporter = OtlpFileExporter("traces.json", service_name="my-service") for event in events: exporter.export(event) exporter.flush() @@ -280,7 +278,7 @@ Windows ETW subsystem - **Live Dashboard**: Gradio-based real-time UI (`pyetwkit dashboard` CLI) - **Event Correlation Engine**: Link events by PID/TID/Handle with timeline export - **Recording & Replay**: Capture sessions to `.etwpack` format with compression -- **OpenTelemetry Exporter**: Export to OTLP endpoints (Jaeger, Grafana, etc.) +- **OpenTelemetry Exporter**: Export to OTLP endpoints (Jaeger, Grafana, etc.) — _announced, but the transport was never implemented; see [#88](https://github.com/m96-chan/PyETWkit/issues/88)_ ### v2.0.0 (2024-12) - **Multi-session support**: Run multiple ETW sessions simultaneously diff --git a/src/pyetwkit/exporters/otlp.py b/src/pyetwkit/exporters/otlp.py index f9ff746..e17e4c5 100644 --- a/src/pyetwkit/exporters/otlp.py +++ b/src/pyetwkit/exporters/otlp.py @@ -145,14 +145,27 @@ def extract_attributes(self, event: Any) -> dict[str, Any]: class OtlpExporter: - """Exports ETW events to OpenTelemetry Protocol (OTLP). + """Maps ETW events to OTLP spans. **Sending is not implemented.** + + Nothing in this class transmits anything: there is no HTTP or gRPC client + behind `endpoint`, and :meth:`flush` raises :class:`NotImplementedError` + rather than reporting a success it cannot deliver. Event to span mapping, + sampling and batching all work, so this remains useful for building the + payload, but it will not reach a collector. + + Use :class:`OtlpFileExporter` to write spans out today, and see #88 for the + transport. Example: >>> exporter = OtlpExporter( ... endpoint="http://collector:4317", ... service_name="windows-etw" ... ) - >>> exporter.export(event) + >>> exporter.export(event) # buffers the span + True + >>> exporter.flush() # raises NotImplementedError + Traceback (most recent call last): + NotImplementedError: ... """ def __init__( @@ -226,13 +239,17 @@ def sample_rate(self) -> float: return self._sample_rate def export(self, event: Any) -> bool: - """Export a single event. + """Buffer one event as a span. Args: event: ETW event to export. Returns: - True if exported successfully. + True once the span is buffered. + + Raises: + NotImplementedError: if this fills the batch, since that triggers a + :meth:`flush` and there is no transport to flush to. """ # Apply sampling if self._sample_rate < 1.0: @@ -267,23 +284,37 @@ def export_batch(self, events: list[Any]) -> bool: return self.flush() def flush(self) -> bool: - """Flush pending events to the collector. + """Send pending spans to the collector. **Not implemented.** - Returns: - True if flushed successfully. + Raises: + NotImplementedError: always, when there is anything to send. + + This used to clear the batch and return True, so every event was + discarded and the caller was told it had been delivered. Failing is the + only honest answer until there is a transport: a monitoring pipeline + that reports success while sending nothing is worse than one that stops. """ if not self._batch: + self._last_export = time.time() return True - # In production, would send to OTLP endpoint - # For now, just clear the batch - self._batch.clear() - self._last_export = time.time() - return True + pending = len(self._batch) + raise NotImplementedError( + f"OtlpExporter cannot send: no OTLP transport is implemented, so " + f"{pending} span(s) would be silently discarded. Use OtlpFileExporter " + f"to write spans to a file, or follow " + f"https://github.com/m96-chan/PyETWkit/issues/88 for the transport." + ) def shutdown(self) -> None: - """Shutdown the exporter.""" - self.flush() + """Shutdown the exporter. + + Discards anything still buffered rather than raising from a teardown + path, since callers reach this from ``finally`` blocks. `flush` is where + the missing transport is reported. + """ + self._batch.clear() + self._last_export = time.time() def attach_to_session(self, session: Any) -> None: """Attach the exporter to an ETW session. diff --git a/tests/test_otlp_exporter.py b/tests/test_otlp_exporter.py index e0e0312..43ba874 100644 --- a/tests/test_otlp_exporter.py +++ b/tests/test_otlp_exporter.py @@ -237,12 +237,60 @@ def test_exporter_shutdown(self) -> None: exporter = OtlpExporter(endpoint="http://localhost:4317") assert hasattr(exporter, "shutdown") - def test_exporter_flush(self) -> None: - """Test exporter flush.""" + def test_flush_does_not_report_success_without_sending(self) -> None: + """There is no transport, so flush must not claim delivery. + + It used to clear the batch and return True, which meant every event was + discarded while the caller was told it had been sent. Port 1 is closed, + so a real transport could not succeed here either. + """ from pyetwkit.exporters import OtlpExporter - exporter = OtlpExporter(endpoint="http://localhost:4317") - assert hasattr(exporter, "flush") + exporter = OtlpExporter(endpoint="http://127.0.0.1:1") + exporter.export({"provider_name": "P", "event_id": 1, "process_id": 1, "properties": {}}) + + with pytest.raises(NotImplementedError, match="no OTLP transport"): + exporter.flush() + + def test_export_batch_does_not_report_success_either(self) -> None: + """export_batch() ends in a flush, so it cannot claim delivery either.""" + from pyetwkit.exporters import OtlpExporter + + exporter = OtlpExporter(endpoint="http://127.0.0.1:1") + + with pytest.raises(NotImplementedError): + exporter.export_batch( + [{"provider_name": "P", "event_id": 1, "process_id": 1, "properties": {}}] + ) + + def test_flush_with_nothing_buffered_is_not_an_error(self) -> None: + """Nothing to send is not a failure to send.""" + from pyetwkit.exporters import OtlpExporter + + exporter = OtlpExporter(endpoint="http://127.0.0.1:1") + assert exporter.flush() is True + + def test_shutdown_does_not_raise(self) -> None: + """Callers reach shutdown from `finally`; it must not throw there.""" + from pyetwkit.exporters import OtlpExporter + + exporter = OtlpExporter(endpoint="http://127.0.0.1:1") + exporter.export({"provider_name": "P", "event_id": 1, "process_id": 1, "properties": {}}) + exporter.shutdown() + + def test_file_exporter_still_works(self, tmp_path) -> None: + """Regression guard: the exporter that does work must keep working.""" + import json + + from pyetwkit.exporters import OtlpFileExporter + + out = tmp_path / "traces.json" + exporter = OtlpFileExporter(str(out)) + exporter.export({"provider_name": "P", "event_id": 7, "process_id": 1, "properties": {}}) + + assert exporter.flush() is True + assert out.exists() + assert len(json.loads(out.read_text(encoding="utf-8"))["spans"]) == 1 class TestOtlpHeaders: From 7aeff5370384cbece1492699a3876e18c3beec17 Mon Sep 17 00:00:00 2001 From: m96-chan Date: Sun, 6 Sep 2026 15:55:25 +0900 Subject: [PATCH 3/3] chore: Release 3.1.0 Since v3.0.1, the last version PyPI actually has, event properties went from a hardcoded guess list of twelve names to whatever the provider's schema declares, which is the substance of this release. Arrays, nested structures and WPP came with it, and `pip install .` was fixed -- it had never worked. Minor rather than patch: the API gains `set_wpp_pdb_path`, `set_wpp_tmf_file`, `set_wpp_tmf_search_path`, `set_property_formatting`, `formatted_properties` and `raw_data`, all additive. 3.0.2 is skipped rather than reused. That version is tagged and has a GitHub release but never reached PyPI, and republishing a version number that already means something to anyone who read the release notes would be worse than leaving a gap. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01HhPcm483PrDmuQEUSALBFn --- README.md | 18 ++++++++++++++++++ pyproject.toml | 2 +- src/pyetwkit/__init__.py | 2 +- 3 files changed, 20 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5462bd8..5b923a0 100644 --- a/README.md +++ b/README.md @@ -274,6 +274,24 @@ Windows ETW subsystem ## Changelog +### v3.1.0 (2026-09) + +Event properties are now decoded from the schema via TDH, rather than guessed +from a list of twelve names. See [Event Properties](docs/advanced/event_properties.rst). + +- **All properties decoded**: whatever the provider declares, not a guess list (#72, #76) +- **Arrays and nested structures**: lists and dicts, instead of only the first element (#84) +- **WPP events**: decoded given a `.pdb` or `.tmf` — `set_wpp_pdb_path()` needs no SDK tooling (#72) +- **Undecodable events keep their payload** in `raw_data` instead of losing it +- **`formatted_properties`**: TDH's own display strings, opt-in, with value maps resolved +- Correct SIDs, `win:Boolean`, pointer widths from 32-bit processes, and counted strings +- **`pip install .` works** — it never had (#79) +- `OtlpExporter` no longer reports success while discarding events; its transport + was never implemented (#88). `OtlpFileExporter` is unaffected + +_Note: v3.0.2 was tagged and released on GitHub but never reached PyPI, so this +is the first release since v3.0.1 that PyPI users will see._ + ### v3.0.0 (2024-12) - **Live Dashboard**: Gradio-based real-time UI (`pyetwkit dashboard` CLI) - **Event Correlation Engine**: Link events by PID/TID/Handle with timeline export diff --git a/pyproject.toml b/pyproject.toml index aa4fd63..67f2f62 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "maturin" [project] name = "pyetwkit" -version = "3.0.2" +version = "3.1.0" description = "High-performance ETW (Event Tracing for Windows) consumer library for Python" readme = "README.md" license = { text = "MIT" } diff --git a/src/pyetwkit/__init__.py b/src/pyetwkit/__init__.py index f872b18..1cb51be 100644 --- a/src/pyetwkit/__init__.py +++ b/src/pyetwkit/__init__.py @@ -27,7 +27,7 @@ from typing import TYPE_CHECKING -__version__ = "3.0.2" +__version__ = "3.1.0" __author__ = "m96-chan" # Import core types from Rust extension