Skip to content

Add ResourceMonitor for the index builds of other engines and share the resource-usage plot code [SUBTRACT ONE SEC, THEN MERGE WHEN ALL GREEN] - #327

Open
tanmay-9 wants to merge 23 commits into
qlever-dev:mainfrom
tanmay-9:share-resource-usage-plot-and-monitor
Open

tanmay-9 wants to merge 23 commits into
qlever-dev:mainfrom
tanmay-9:share-resource-usage-plot-and-monitor

Conversation

@tanmay-9

@tanmay-9 tanmay-9 commented Aug 18, 2026 •

Copy link
Copy Markdown
Collaborator

Since #305, the QLever binaries sample their own RSS memory and CPU usage and write the resource-usage log themselves, so the Python-side sampler was removed. The other engines of qeval do not write such a log, so they need a sampler on the Python side. The plot code so far also only described a QLever index build, since the shaded phase bands and the subtitle come from the QLever index log.

With this change, qeval/resource_monitor.py provides that sampler. It wraps the index command of an engine and writes a log with the columns that the plot needs (elapsed_s, rss, cpu_percent), by following the index process natively or with docker stats or podman stats in a container. The plot code is shared by all engines. Only the shaded bands and the subtitle differ per engine, and an engine passes its own engine_overlay and engine_subtitle functions to render_usage_plot_if_installed in the new qlever/resource_usage/plot_util.py. Without them, the plot describes a QLever index build, so qlever index works as before. plot_util.py can be imported without matplotlib and numpy, so the plotting libraries stay optional for every engine.

The log and the plot files of a non-QLever engine are prefixed <dataset>.<engine>, so that it is clear which engine a file belongs to. QLever keeps the plain <dataset> prefix, since its own binaries write that log.

On the side, the two helpers in qlever/util.py that only the sampler uses moved along with it to qeval, and parse_git_hash moved next to its only caller in usage_plot.py. The shared test fixtures, including the guard against signals to unrelated processes, moved to test/conftest.py, so that they also apply to the tests of qeval.

NOTE: No engine uses the sampler yet. Oxigraph will be the first one, in #328.

@tanmay-9 tanmay-9 changed the title Share resource usage plot and monitor Monitor and plot RSS and CPU usage during index builds of non-qlever engines Aug 18, 2026
@tanmay-9
tanmay-9 marked this pull request as ready for review October 13, 2026 14:41
@hannahbast hannahbast changed the title Monitor and plot RSS and CPU usage during index builds of non-qlever engines Add ResourceMonitor for the index builds of other engines and share the resource-usage plot code Oct 22, 2026
@hannahbast hannahbast changed the title Add ResourceMonitor for the index builds of other engines and share the resource-usage plot code Add ResourceMonitor for the index builds of other engines and share the resource-usage plot code [ADD TEST, THEN MERGE] Oct 22, 2026
@hannahbast hannahbast changed the title Add ResourceMonitor for the index builds of other engines and share the resource-usage plot code [ADD TEST, THEN MERGE] Add ResourceMonitor for the index builds of other engines and share the resource-usage plot code [SUBTRACT ONE SEC, THEN MERGE] Oct 22, 2026
@hannahbast hannahbast changed the title Add ResourceMonitor for the index builds of other engines and share the resource-usage plot code [SUBTRACT ONE SEC, THEN MERGE] Add ResourceMonitor for the index builds of other engines and share the resource-usage plot code [SUBTRACT ONE SEC, THEN MERGE WHEN ALL GREEN] Oct 22, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant