Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions .github/ISSUE_TEMPLATE/bug.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: Bug
description: Something behaves differently from what the docs say.
labels: ["bug"]
body:
- type: dropdown
id: component
attributes:
label: Component
options:
- collector
- runner-android
- runner-ios
- runner-machine
- host executor
- dashboard
- docs
validations:
required: true

- type: textarea
id: expected
attributes:
label: What the docs say, and what happened
description: >
A link to the page or the README line, if there is one. If the
documentation is what is wrong, that is a bug too and it is the easiest
kind to fix.
validations:
required: true

- type: textarea
id: repro
attributes:
label: How to reproduce
description: >
The commands, and the job spec if one is involved. `npm test` in
`collector/` starts its own throwaway collector, so a repro that runs
there is worth more than one that needs your fleet.
validations:
required: true

- type: textarea
id: env
attributes:
label: Versions
placeholder: |
fleet-runner: v0.2.0 (or a commit sha)
node: v22.11.0
os: macOS 26.1
validations:
required: true
11 changes: 11 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
blank_issues_enabled: true
contact_links:
- name: Security report
url: https://github.com/addisdev/fleet-runner/security/advisories/new
about: >
Privately, via GitHub's advisory form. Read SECURITY.md first — the
collector has no authentication by design, and that on its own is not a
finding.
- name: Question, or show your shelf
url: https://github.com/addisdev/fleet-runner/discussions
about: What hardware you are running this on, and anything that is not a bug.
76 changes: 76 additions & 0 deletions .github/ISSUE_TEMPLATE/device-report.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
name: Device report
description: What your hardware measured, refused, or broke on. The most useful thing you can send.
title: "[device] "
labels: ["device-report"]
body:
- type: markdown
attributes:
value: |
The whole point of this project is comparable numbers across hardware
nobody has a lab full of. Every phone that is not on this shelf is data
this repository does not have — including the ones that refused to work.

A report where something failed is as useful as one where it worked.

- type: input
id: device
attributes:
label: Device
description: Model and SoC, as the runner reports them if you have it.
placeholder: "SM-X930 (Dimensity 9400) / iPhone 16 simulator / ThinkPad X1, i7-1260P"
validations:
required: true

- type: input
id: os
attributes:
label: OS and RAM
placeholder: "android-15, 12288 MB"
validations:
required: true

- type: dropdown
id: runner
attributes:
label: Which runner
options:
- runner-android
- runner-ios
- runner-machine
- host executor
validations:
required: true

- type: input
id: backend
attributes:
label: Workload and backend
placeholder: "benchmark / synthetic, or vision-eval / litert"
validations:
required: true

- type: textarea
id: spec
attributes:
label: The job spec
description: The JSON you enqueued. Redact hostnames and hashes if you like.
render: json

- type: textarea
id: result
attributes:
label: The result row
description: >
From `GET /api/results?job=<id>`, or the error if it failed. The
`metrics` object is the part that matters.
render: json

- type: textarea
id: notes
attributes:
label: Anything surprising
description: >
Numbers that look wrong, a capability that was declared but bounced, a
device left in a state the journals did not record. Especially if a
second device disagreed with this one — that is how the all-zero logits
tensor was caught.
23 changes: 23 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
<!--
Branch from `develop`. `ci-ok` is the only required check; it passes when every
component suite is green or skipped.
-->

## What and why



## Verified

<!-- Which suites you actually ran, and what they said. Not which you believe
would pass. -->

- [ ] Tests run for what I touched, and named above
- [ ] Docs updated in this change (the site builds `--strict`, so a broken link fails CI)
- [ ] `CHANGELOG.md` entry under `## [Unreleased]`, if this is user-visible

## What is still not true

<!-- Anything this leaves unproven or unimplemented. Say it here rather than
letting somebody find it. This section is the one that makes a review
fast. -->
86 changes: 86 additions & 0 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@

# Contributor Covenant Code of Conduct

## Our Pledge

We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.

We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.

## Our Standards

Examples of behavior that contributes to a positive environment for our community include:

* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall community

Examples of unacceptable behavior include:

* The use of sexualized language or imagery, and sexual attention or advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a professional setting

## Enforcement Responsibilities

Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.

Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.

## Scope

This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.

## Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at [@addisdev on GitHub](https://github.com/addisdev), by direct message or by
opening a private report on this repository. All complaints will be reviewed and investigated promptly and fairly.

All community leaders are obligated to respect the privacy and security of the reporter of any incident.

## Enforcement Guidelines

Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:

### 1. Correction

**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.

**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.

### 2. Warning

**Community Impact**: A violation through a single incident or series of actions.

**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.

### 3. Temporary Ban

**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior.

**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.

### 4. Permanent Ban

**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.

**Consequence**: A permanent ban from any sort of public interaction within the community.

## Attribution

This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].

Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder][Mozilla CoC].

For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at [https://www.contributor-covenant.org/translations][translations].

[homepage]: https://www.contributor-covenant.org
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations

147 changes: 147 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Contributing

This is a personal project that was made public because the write-up was worth
sharing. Contributions are welcome and there is no obligation for anyone to
send any.

The most useful thing you can send is **a device report**: what your hardware
measured, what it refused, and what broke. The whole point of the project is
comparable numbers across hardware nobody has a lab full of, and every phone
that is not on this shelf is data this repository does not have.

## Running the suites

Four components, four suites. Only run the one you touched.

=== "collector"

```bash
cd collector
npm install
npm run dash:install # once, for the dashboard build step
npm test
```

`npm test` typechecks, builds the dashboard, checks metric names against
the schema, validates the example job specs, then **starts its own
collector on a kernel-assigned port with a temporary database** and runs
the smoke suite against it. It never touches a real fleet's history, which
is why it can be wired to `npm test` at all.

It takes about four minutes.

=== "runner-android"

```bash
cd runner-android
./gradlew :app:testDebugUnitTest -Pfleet.skipNative
```

**`-Pfleet.skipNative` is what lets you run this with no NDK.** The arm64
llama.cpp build takes about fifteen minutes and needs a pinned NDK, and the
JVM tests never call into it. CI uses the same flag.

For the native build you need the submodule and NDK 27.2:

```bash
git submodule update --init --recursive
./gradlew :app:assembleDebug
```

=== "runner-ios"

```bash
cd runner-ios
brew install xcodegen
./generate.sh
xcodebuild build -project FleetRunner.xcodeproj -scheme FleetRunner \
-destination 'generic/platform=iOS Simulator' \
-derivedDataPath build CODE_SIGNING_ALLOWED=NO
```

=== "runner-machine"

```bash
cd runner-machine
npm install && npm test
```

122 tests, about a second. This is the easiest component to contribute to
and the reference implementation of the protocol.

## Things this project learned the hard way

Each of these cost somebody an evening.

**Run one `xcodebuild` per simulator.** Two concurrent test sessions against
the same simulator send each other `SIGTERM` and wedge it. A "crash" with no
`.ips` file is almost always a collision, not a bug in the app.

**Call `xcodegen` by its full path in anything non-interactive.** Homebrew's
`bin` is not on a non-interactive `PATH`, so a bare `xcodegen` over SSH or under
launchd fails in a way that looks like the project generated fine and your new
files simply did not compile. `generate.sh` already handles this.

**Regenerate the Xcode project after syncing files to another machine.** An
`rsync` that carries `*.xcodeproj` across overwrites a project generated for
that machine. Exclude it and run `./generate.sh` on the far end.

**The committed `FleetRunner.xcodeproj` is deliberately the one without the
llama.cpp framework.** Xcode treats a declared-but-missing XCFramework as a hard
error rather than something to skip, so a spec naming it unconditionally cannot
be built from a fresh clone. If you regenerate with the framework present, the
project will show as modified — do not commit that.

**A metric name that is not in `result.schema.json` is stored and unqueryable.**
Add it to the schema in the same change that emits it. This has already cost
one write-up its reproducibility: an eval's accuracy rode in `decode_tok_s`
because vision had no field of its own, and no query can reproduce those numbers
today. `npm test` fails if the schema and its mirror disagree.

**Declare a capability only when the machine can honour it.** A capability
claimed but not implemented is a job the queue hands over and the agent bounces,
which from the dashboard looks exactly like a broken workload.

## Adding a workload

You do not need a collector release. The collector accepts any workload a
registered agent declares in its capabilities. See [Add a
workload](https://addisdev.github.io/fleet-runner/integration/add-a-workload/)
and [Writing a runner](https://addisdev.github.io/fleet-runner/writing-a-runner/).

## The one thing to get right

**Say what is not true yet.**

Every workload page ends with what that workload refuses to do. Every README has
a "what works, and what does not". The integration guide says on its first
screen that the GitHub Actions path has never run on a real runner.

When you cannot measure the thing asked for, fail with a message naming what is
missing. A wrong number recorded as a result is worse than a failure, because a
failure gets investigated and a number gets believed. The iOS Simulator's GPU
returning an all-zero logits tensor — silently, with no error — is the reason
this is the first rule rather than a nice sentiment.

## Pull requests

- Branch from `develop`. `main` is the released trunk.
- Run the suite for what you touched, and say in the description what you ran.
- Update the docs in the same change. The site builds with `--strict`, so a
broken link fails CI.
- Add a `CHANGELOG.md` entry under `## [Unreleased]` for anything user-visible.

`ci-ok` is the only required check. It always reports, and it passes when every
component suite is green **or skipped** — a change to one runner does not build
the others.

## Dependencies

**Dependabot is deliberately limited to security alerts and security fixes.**
There is no `dependabot.yml`, which is what keeps version-bump pull requests
from burying the ones that matter. If you want to upgrade something, do it in a
change that says why.

## Code of conduct

[Contributor Covenant 2.1](CODE_OF_CONDUCT.md).
Loading
Loading