Repository navigation
Show your shelf: what hardware are you running this on? #17
addisdev
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
This is the thread for what you are running Fleet Runner on.
The whole point of the project is comparable numbers across hardware nobody has
a lab full of. Every device that is not on this shelf is data this repository
does not have — including the ones that refused to work, which are often
more useful than the ones that did.
What this shelf is
Old hardware is deliberate. A phone too slow to hold a model still runs the
synthetic backend and still produces a comparable number, which is what makes a
drawer of handsets a test matrix instead of e-waste.
What to post
Anything. A one-line "it registered and ran a synthetic benchmark on a Pixel 4a"
is genuinely useful. So is "it would not build", and so is a number that looks
wrong.
If you want structure, the
device report template
has the fields that matter — model, SoC, OS, RAM, which runner, which backend,
the job spec and the result row.
The thing most worth reporting
A second device disagreeing with the first.
The iOS Simulator's emulated GPU returned an all-zero logits tensor for the
plant-ID model. Silently: no error, no warning, just zeros, while
.cpuOnlygave logits identical to the Mac. The only reason that did not ship as a real
accuracy number is that the same weights were run somewhere else and the answers
did not match.
If your device produces an accuracy or a throughput that disagrees with what is
in the eval,
that is the most valuable thing you can post here.
Getting started
Fifteen minutes, Node and nothing else
— the first agent is your own laptop, so you can see the loop work before
touching a phone.
All reactions