Skip to content

Keep now on the metrics axis while a job is reporting - #4132

Merged
peterschmidt85 merged 4 commits into
masterfrom
metrics-now-bold
Aug 12, 2026
Merged

Keep now on the metrics axis while a job is reporting#4132
peterschmidt85 merged 4 commits into
masterfrom
metrics-now-bold

Conversation

@peterschmidt85

@peterschmidt85 peterschmidt85 commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

On a healthy run the axis showed a timestamp instead of now about a quarter of the time, and alternated between the two under -w — so a job that was fine looked like it kept dying. The check was tighter than the collection interval it was reporting on.

now is also drawn bold, so a live run is distinguishable at a glance from one that stopped reporting.

$ dstack metrics llama-70b-sft

               UTILIZATION                     MEMORY
 job=0  cpu    ▁▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▂▂▂▂▂▂▂▂▂ 33%  ▃▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ 579GB/960GB
        gpu=0  ▁▄▆▆▆▆▆▆▆▆▆▆▆▆▆▆▁▆▆▆▆▆▆▆▆▆ 93%  ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB
        gpu=1  ▂▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▁▆▆▆▆▆▆▆▆▆ 93%  ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB

               12 Aug 13:06 ┄┄┄┄┄┄┄┄┄ now      12 Aug 13:06 ┄┄┄┄┄┄┄┄┄ now

A run whose instance goes unreachable still shows its last timestamp, unchanged.

@peterschmidt85 peterschmidt85 changed the title Emphasise the live edge of the metrics axis Fix and emphasise the now marker on the metrics axis Aug 12, 2026
@peterschmidt85 peterschmidt85 changed the title Fix and emphasise the now marker on the metrics axis Keep now on the metrics axis while a job is reporting Aug 12, 2026
The axis labelled its right edge `now` only when the newest sample was
under ten seconds old, borrowing `pretty_date`'s threshold. Collection
runs every ten seconds, so samples routinely arrive older than that:
polling a live job, three renders in ten showed an absolute time instead,
and under `--watch` the edge alternated between the two.

Thresholds on three collection intervals instead. A job whose instance
goes unreachable still shows its last timestamp -- that gap is minutes,
not seconds.

Draws `now` bold, in the grey `no data` already uses, so a live run is
distinguishable at a glance from one that stopped reporting.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Andrey Cheptsov and others added 3 commits August 12, 2026 14:12
Five of the six in this module restated what the function below already
said, or argued for the option taken over one that was not. The remaining
one records that the server sends samples newest-first, which is a fact
from outside this file and draws every chart backwards if missed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`_axis` and `_stamp` said nothing about building a timeline row or
formatting a timestamp for it, and `_span` sat next to `_window` meaning
two different windows. Names that carry their meaning at the call site
beat a docstring that only carries it at the definition:

  _axis        -> _time_axis        builds the timeline row
  _stamp       -> _time_label       formats one timestamp for it
  _span        -> _shared_window    the window all jobs are drawn against
  _window      -> _job_window       one job's own first and last sample
  _lead        -> _blank_cells      cells to blank before a job started
  _drawn       -> _cells_drawn      cells a job actually fills
  _cell        -> _chart_cell       a sparkline joined to its number
  _level_cell  -> _capacity_cell    memory as a fraction of capacity
  _latest      -> _latest_value

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Renaming alone did not carry it: `_time_axis` still did not say it builds
a row, `_time_label` hid that it returns `now` for a live job, and
`_samples_num` did not say it takes the longest series. Each now has one
line stating what it does.

`_chart_cell` is gone -- it wrapped a single `Text.assemble` at three
call sites and only added a name to look up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@peterschmidt85
peterschmidt85 merged commit 4ed4170 into master Aug 12, 2026
27 checks passed
@peterschmidt85
peterschmidt85 deleted the metrics-now-bold branch August 12, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant