Skip to content

Add fastermodels.measurement: latency rows and quality rows, validated #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nathanhubens merged 3 commits into from
Sep 29, 2026
Merged

Add fastermodels.measurement: latency rows and quality rows, validated #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
12889da
Add fastermodels.measurement: one on-device measurement row, validated
nathanhubens Sep 28, 2026
d26ccfc
Split measurement rows into latency rows and quality rows
nathanhubens Sep 29, 2026
9457f27
Write measurement Parquet files with an explicit Arrow schema
nathanhubens Sep 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions fastermodels/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@
params, macs, peak_activation_bytes)
from fastermodels.card import render_card, check_card, FORBIDDEN
from fastermodels.gate import run_gate, gate_passed, GateRow
from fastermodels import measurement as measurement
27 changes: 27 additions & 0 deletions fastermodels/_modidx.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,33 @@
'fastermodels.gate._reload': ('gate.html#_reload', 'fastermodels/gate.py'),
'fastermodels.gate.gate_passed': ('gate.html#gate_passed', 'fastermodels/gate.py'),
'fastermodels.gate.run_gate': ('gate.html#run_gate', 'fastermodels/gate.py')},
'fastermodels.measurement': { 'fastermodels.measurement._canonical': ( 'measurement.html#_canonical',
'fastermodels/measurement.py'),
'fastermodels.measurement._cross_problems': ( 'measurement.html#_cross_problems',
'fastermodels/measurement.py'),
'fastermodels.measurement._field_problems': ( 'measurement.html#_field_problems',
'fastermodels/measurement.py'),
'fastermodels.measurement._is_date': ('measurement.html#_is_date', 'fastermodels/measurement.py'),
'fastermodels.measurement._is_json': ('measurement.html#_is_json', 'fastermodels/measurement.py'),
'fastermodels.measurement._item': ('measurement.html#_item', 'fastermodels/measurement.py'),
'fastermodels.measurement._json': ('measurement.html#_json', 'fastermodels/measurement.py'),
'fastermodels.measurement._pair': ('measurement.html#_pair', 'fastermodels/measurement.py'),
'fastermodels.measurement._parse': ('measurement.html#_parse', 'fastermodels/measurement.py'),
'fastermodels.measurement._typed': ('measurement.html#_typed', 'fastermodels/measurement.py'),
'fastermodels.measurement.graph_hash': ( 'measurement.html#graph_hash',
'fastermodels/measurement.py'),
'fastermodels.measurement.join': ('measurement.html#join', 'fastermodels/measurement.py'),
'fastermodels.measurement.key': ('measurement.html#key', 'fastermodels/measurement.py'),
'fastermodels.measurement.read_rows': ( 'measurement.html#read_rows',
'fastermodels/measurement.py'),
'fastermodels.measurement.status': ('measurement.html#status', 'fastermodels/measurement.py'),
'fastermodels.measurement.throughput': ( 'measurement.html#throughput',
'fastermodels/measurement.py'),
'fastermodels.measurement.to_parquet': ( 'measurement.html#to_parquet',
'fastermodels/measurement.py'),
'fastermodels.measurement.validate': ('measurement.html#validate', 'fastermodels/measurement.py'),
'fastermodels.measurement.write_row': ( 'measurement.html#write_row',
'fastermodels/measurement.py')},
'fastermodels.model': { 'fastermodels.model.FasterModel': ('model.html#fastermodel', 'fastermodels/model.py'),
'fastermodels.model.FasterModel.__init__': ('model.html#fastermodel.__init__', 'fastermodels/model.py'),
'fastermodels.model.FasterModel._from_pretrained': ( 'model.html#fastermodel._from_pretrained',
Expand Down
346 changes: 346 additions & 0 deletions fastermodels/measurement.py
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions nbs/00_model.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -374,6 +374,7 @@
"- [Eval](01_eval.html) - the per-image harness the published numbers come from\n",
"- [Card](02_card.html) - what the artifact says about itself\n",
"- [Gate](03_gate.html) - the conditions an artifact meets before publication\n",
"- [Measurement](04_measurement.html) - one on-device measurement row of a published model\n",
"\n",
"Tests live in `nbs/tests/test_model.ipynb`."
]
Expand Down
1 change: 1 addition & 0 deletions nbs/01_eval.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -436,6 +436,7 @@
"- [Model](00_model.html) - the model these numbers are measured on\n",
"- [Card](02_card.html) - where `k`, `n`, the interval and the delta are written down\n",
"- [Gate](03_gate.html) - the condition that reads `lo` against the floor\n",
"- [Measurement](04_measurement.html) - where an on-device accuracy records its interval and `ci_method`\n",
"\n",
"Tests live in `nbs/tests/test_eval.ipynb`."
]
Expand Down
1 change: 1 addition & 0 deletions nbs/02_card.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,7 @@
"- [Eval](01_eval.html) - where `k`, `n`, the delta and the agreement come from\n",
"- [Gate](03_gate.html) - condition 7 refuses to publish a card `check_card` flags\n",
"- [Model](00_model.html) - the artifact the card describes\n",
"- [Measurement](04_measurement.html) - the on-device measurement rows a latency comes from\n",
"\n",
"Tests live in `nbs/tests/test_card.ipynb`."
]
Expand Down
1 change: 1 addition & 0 deletions nbs/03_gate.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,7 @@
"- [Model](00_model.html) - `state_hash`, the digest condition 1 compares\n",
"- [Card](02_card.html) - `check_card`, which condition 7 runs\n",
"- [Eval](01_eval.html) - the paired interval condition 3 reads\n",
"- [Measurement](04_measurement.html) - the validity checks of one on-device measurement row\n",
"\n",
"Tests live in `nbs/tests/test_gate.ipynb`."
]
Expand Down
Loading
Loading