Repository navigation
Add ResourceMonitor for the index builds of other engines and share the resource-usage plot code [SUBTRACT ONE SEC, THEN MERGE WHEN ALL GREEN] - #327
Open
tanmay-9 wants to merge 23 commits into
Conversation
…t new engines can reuse existing code
…o share-resource-usage-plot-and-monitor
…fixtures across `qlever` and `qeval`
…ss their `overlay` and `subtitle` directly
…nd test that engine hooks reach the plot
tanmay-9
marked this pull request as ready for review
October 13, 2026 14:41
…`find_process_by_binary`'s `parent_pid` as `int`
ResourceMonitor for the index builds of other engines and share the resource-usage plot code
ResourceMonitor for the index builds of other engines and share the resource-usage plot codeResourceMonitor for the index builds of other engines and share the resource-usage plot code [ADD TEST, THEN MERGE]
ResourceMonitor for the index builds of other engines and share the resource-usage plot code [ADD TEST, THEN MERGE]ResourceMonitor for the index builds of other engines and share the resource-usage plot code [SUBTRACT ONE SEC, THEN MERGE]
ResourceMonitor for the index builds of other engines and share the resource-usage plot code [SUBTRACT ONE SEC, THEN MERGE]ResourceMonitor for the index builds of other engines and share the resource-usage plot code [SUBTRACT ONE SEC, THEN MERGE WHEN ALL GREEN]
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since #305, the QLever binaries sample their own RSS memory and CPU usage and write the resource-usage log themselves, so the Python-side sampler was removed. The other engines of
qevaldo not write such a log, so they need a sampler on the Python side. The plot code so far also only described a QLever index build, since the shaded phase bands and the subtitle come from the QLever index log.With this change,
qeval/resource_monitor.pyprovides that sampler. It wraps the index command of an engine and writes a log with the columns that the plot needs (elapsed_s,rss,cpu_percent), by following the index process natively or withdocker statsorpodman statsin a container. The plot code is shared by all engines. Only the shaded bands and the subtitle differ per engine, and an engine passes its ownengine_overlayandengine_subtitlefunctions torender_usage_plot_if_installedin the newqlever/resource_usage/plot_util.py. Without them, the plot describes a QLever index build, soqlever indexworks as before.plot_util.pycan be imported withoutmatplotlibandnumpy, so the plotting libraries stay optional for every engine.The log and the plot files of a non-QLever engine are prefixed
<dataset>.<engine>, so that it is clear which engine a file belongs to. QLever keeps the plain<dataset>prefix, since its own binaries write that log.On the side, the two helpers in
qlever/util.pythat only the sampler uses moved along with it toqeval, andparse_git_hashmoved next to its only caller inusage_plot.py. The shared test fixtures, including the guard against signals to unrelated processes, moved totest/conftest.py, so that they also apply to the tests ofqeval.NOTE: No engine uses the sampler yet. Oxigraph will be the first one, in #328.