DeviceTypeSources

Selecting Device Types (DeviceTypeSources.json)

Device type records are kept one file per record: RaftCore's are in devtypes/records/, as <Key>.json, for example MCP9808.json. Each file holds the record exactly as it appears under "devTypes" in a catalogue file (see Device Type Record Format). The file name is the device type key, and a deviceType field, if present, must equal it.

A project chooses the records compiled into its firmware with a DeviceTypeSources.json file in its SysType folders.

Existing projects

Nothing changes for a project without a DeviceTypeSources.json:

• DEV_TYPE_JSON_FILES in features.cmake and systypes/Common/DevTypes.json / systypes/<SysType>/DevTypes.json work as before;
• /devtypes/DeviceTypeRecords.json still means "all RaftCore's records": with a RaftCore whose records are one file per record, it reads the devtypes/records folder;
• A DEV_TYPE_JSON_FILES entry can also be a folder of record files.

DeviceTypeSources.json

Put it in systypes/Common/ and/or systypes/<SysType>/. The SysType's file adds to Common's.

{
    "sources": {
        "raftcore": { "bundled": true },
        "local": { "path": "devtypes" }
    },
    "builtin": {
        "include": ["raftcore:VL53L0X", "raftcore:MCP9808", "raftcore:TrillRing", "local:*"],
        "exclude": [],
        "policy": { "TrillRing": "disabled" }
    }
}

sources

Each source is named, and the name is used in include:

Source	Means
{ "bundled": true }	RaftCore's records, at the RaftCore version the build uses ("builtin": true, its earlier name, still works)
{ "path": "devtypes" }	A folder of record files, a single record file (devtypes/MySensor.json), or a catalogue file ({"devTypes": {...}}). The path is relative to the DeviceTypeSources.json that names it
{ "git": "<url>", "ref": "<ref>", "path": "records" }	Records in a git repo - see Git sources

builtin

The records compiled into the firmware:

• include: "*" (everything from every source), "<source>:*", "<source>:<key>", a bare "<key>" (only when one source has it), or "<source>:@<set>" - a named set of RaftCore's records (below). The default is everything.
• exclude: the same forms, removed from what include selected.
• policy: "enabled", "disabled" or "disabledClaim" for a selected record (as typePolicy).

Named sets

RaftCore's devtypes/sets/ holds named selections of its records: common (about 10 popular sensors) and extended (about 25). Select one with the bundled source's name, e.g. "include": ["raftcore:@common"], in builtin or typeFile, and in exclude too. Records a set selects are treated like "raftcore:*": a local copy still replaces one. Sets are only for RaftCore's records (the bundled source); a set naming a record RaftCore doesn't have stops the build. Apps made by raft new use them for the "I2C and sensors" question (RaftCore 1.64.0 and later).

Choosing between sources

• A path source replaces the same key from another source, with a build message, so a project changes a RaftCore record by putting its own copy in a local folder. Of two path sources, the later one wins.
• The same key from two other sources is an error, unless include names the source to use (e.g. "raftcore:LSM6DS3").
• Naming a source in include ("raftcore:LSM6DS3") takes the record from that source even if a local folder has one too.

Records with a warning (_warning, acknowledge)

A record's _warning says what selecting it entails beyond the device itself - e.g. the CaliPile's generalCall, which makes the bus send an I2C general call on every slot. Such a record has to be chosen deliberately:

• A wildcard ("*", "<source>:*") doesn't select it; a build message says so. Named sets don't hold such records.
• Named in include, it's selected, and the build reports DEVICE TYPE WARNING: <key>: <text> (a CMake warning) while it's enabled.
• Listing its key in a top-level acknowledge accepts the warning: the wildcard then selects it and nothing is reported.

{
    "sources": { "raftcore": { "bundled": true } },
    "builtin": { "include": ["raftcore:@common", "raftcore:CaliPile"] },
    "acknowledge": ["CaliPile"]
}

acknowledge in Common's and the SysType's files is joined. The same applies to typeFile (and the Device Types app asks before adding such a record, adding it to acknowledge if you go ahead; raft devtypes push asks before pushing one). A record's _warning isn't compiled in or put in type files. (RaftCore versions after 1.64.0)

Board choices

A record describes the device, and every board uses the same one. Choices for the board or product - the poll bus speed and the slots it applies to, the poll interval, samples kept, sample rate and scan priority - are set for the type in the SysType's DevMan typeSettings, not by changing a copy of the record.

Order

Records are compiled in device type key order, whatever order they are listed in. At an address, records are tried in descending identPriority, then descending detection strength, then key order. So where the order between two records matters, give one of them an identPriority.

DEV_TYPE_JSON_FILES and DevTypes.json records still come after the selected records, in their files' order, and replace selected records with the same key, as before. With a DeviceTypeSources.json, select RaftCore's records there rather than with /devtypes/... in DEV_TYPE_JSON_FILES (the build stops if both are used).

Git sources

"sources": {
    "acme":   { "git": "https://github.com/acme/raft-devtypes.git", "ref": "v1.2.0" },
    "widget": { "git": "git@github.com:bob/widget-fw.git", "ref": "3f2a9c1e7b0d...", "path": "devtypes/Widget.json" }
}

• ref: a tag, a branch, or a commit given as its full 40-character hash.
• path: a folder of record files or one record file in the repo. The default is records.
• Fetching: the source is fetched shallowly with only path checked out, into build/<SysType>/raft/devtypes/. A commit already fetched isn't fetched again, so builds work offline once the cache is filled.
• The lock file, systypes/<SysType>/DeviceTypeSources.lock, records the commit each source's ref resolved to, and the SHA-256 of each record used. Commit it with the project.
  • While a source's git and ref are unchanged, the build uses the locked commit, so a moved tag or a new commit on a branch doesn't change a build.
  • A new or changed source is resolved and the lock updated, with a build message.
  • To take new commits, run raft devtypes update (or python3 RaftCore/scripts/AssembleDeviceTypes.py ... --update).
  • A cached record that doesn't match the lock is an error.
• Local copy: a folder (or link) raftdevtypes/<source> at the project root replaces that git source, for editing records against a real build before committing them. A broken link is an error. Adding or removing one re-runs the selection.
• Source manifest: a repo may have raftdevtypes.json at its root, {"name": "acme", "formatVersion": 1, "minRaftCore": "1.60.0"}. A format this RaftCore doesn't know, or a minRaftCore newer than the build's, is an error with a clear message.
• Decode code: a record's decode code (resp.c) is compiled into the firmware. A record from a git source whose code uses anything outside the decode language (function calls, other names, other characters) is rejected.
• Same key in two sources: a key that a git source and RaftCore (or two git sources) both have must be named in include, e.g. "acme:LSM6DS3".
• Credentials: builds use the git on the build machine, with its credentials. A git@github.com: URL needs an SSH key that the build can use. In the Docker build container that means mounting the key, so a public https:// URL is simplest.

Build

The selection is resolved when the build is configured:

• An error, such as a key no source has, stops the build with a message naming it.
• Changing a selection file, or adding or changing a record file in a selected folder, re-runs it.

The build writes:

File	Contents
build/<SysType>/raft/DeviceTypeRecords.json	The selected records, as one catalogue file (what the generator reads)
build/<SysType>/raft/DeviceTypeSources.manifest.json	For each record, its source, file, SHA-256, and for a git source the commit
systypes/<SysType>/DeviceTypeSources.lock	The commits git sources are pinned to (commit it)
build/<SysType>/raft/devtypes/	Fetched git sources, by commit

To check a selection without a full build:

python3 RaftCore/scripts/AssembleDeviceTypes.py --raftcore RaftCore \
    --select systypes/Common/DeviceTypeSources.json,systypes/Main/DeviceTypeSources.json \
    --out /tmp/DeviceTypeRecords.json --manifest /tmp/manifest.json

--all writes every RaftCore record as one catalogue file (also attached to each RaftCore release as DeviceTypeRecords.json). --split <catalogue.json> --out-dir <folder> splits a catalogue file into one file per record, keeping each record as written.

Runtime type files

The typeFile section of DeviceTypeSources.json selects records for a device type file - records loaded at startup from the device's file system, so they can be added or changed without rebuilding the firmware:

"typeFile": {
    "include": ["acme:Gadget", "local:MPU6050"],
    "policy": { "TrillRing": "disabled" }
}

It takes the same include, exclude and policy as builtin, but includes nothing unless it says so. A policy for a type that isn't in the file applies to the compiled record (disabled or disabledClaim - a type file can't enable a compiled type). To build the file:

python3 RaftCore/scripts/AssembleDeviceTypes.py --raftcore RaftCore --select systypes/Main/DeviceTypeSources.json \
    --typefile --out devlib.json [--compiled build/Main/raft/DeviceTypeRecords.json] [--report report.json]

• The file is minified, without _notes, within the device's size limit: by default 256 KB if the SysType's sdkconfig.defaults (next to its DeviceTypeSources.json) enables PSRAM (CONFIG_SPIRAM=y), otherwise 64 KB - as the firmware's default typeFileLimits.maxFileBytes. --max-bytes sets another. raft devtypes push uses the limit the device reports.
• The file says what it was made from, in a top-level _sources entry that the device ignores: the sources that supplied a record (a git source with the commit its ref resolved to, RaftCore's records with RaftCore's version, a path source with its path) and the typeFile selection. So a file read back from a device (raft devtypes pull, or the Device Types app) shows how to make it again. devman/typepolicy ... save=1 keeps it as it is, so after such a change the file's records can differ from what _sources describes.
• Each record is reported as new, an override of a compiled record, or unchanged - identical to the compiled record, so left out of the file. Policies are reported as policy.
• Records with data but no compiled decoder are listed: new records, and overrides whose response layout differs from the compiled record's. Firmware with runtime decoding (RAFT_RUNTIME_DECODE, the default) decodes them on the device from their attributes; a record with custom decode code (resp.c) is decoded on the device only by firmware with the code interpreter (RAFT_RUNTIME_DECODE_CODE); otherwise only raftjs and the WebUI decode it. The report's entries give noDecoder: true and decoder: "runtime" or "none" (custom code); the device reports runtimePending for a record whose decoder it will make when a device of the type is identified. A record with its own initCode is reported with initCode: "runtime" (only firmware with the interpreter runs it) or "runtime or compiled" (firmware without it uses the compiled record's).
• The compiled records are those of the builtin selection of the same files, unless --compiled gives a catalogue file. For a project that adds records with DEV_TYPE_JSON_FILES or DevTypes.json, use --compiled build/<SysType>/raft/DeviceTypeRecords.compiled.json: the build writes every record it compiles there.

In the file system image

With "fsImage": true in typeFile, the build also puts the type file in the SysType's file system image, so flashing the board (raft flash, without --no-fs) installs it and the device applies it at its first boot:

"typeFile": {
    "include": ["local:Gadget"],
    "fsImage": true
}

• It is built against every record the build compiles (the selection, DEV_TYPE_JSON_FILES and DevTypes.json).
• It goes where the device loads it from: DevMan typeFile in SysTypes.json (default /local/devlib.json), which must be on /local/. DevMan typeFileLimits.maxFileBytes is the size limit (default 256 KB if the SysType's merged sdkconfig.defaults enables PSRAM, otherwise 64 KB).
• A file at the same path in the SysType's FSImage folder is replaced, with a build warning. The SysType needs a file system image (FS_TYPE and FS_IMAGE_PATH in features.cmake).
• Only a full flash writes it: an OTA update, or raft flash --no-fs, leaves the device's file as it is. And flashing the file system image replaces a type file pushed or saved on the device since.

Then upload it as /local/devlib.new.json, check it on the device (devman/typefile/check?file=/local/devlib.new.json - the device's own rules decide) and apply it (devman/typefile/apply?file=/local/devlib.new.json). raft devtypes push does these steps.

In a browser: the catalogue and the Device Types app

RaftCore's GitHub Pages site publishes the records of each release (from v1.59.0) and of main, built by the pages.yml workflow with AssembleDeviceTypes.py --pages:

Path	Contents
devtypes/index.json	The versions published (main first, then releases, newest first) and the latest release
devtypes/<version>/catalogue.json	Each record's key, file, SHA-256 and a summary (name, manufacturer, class, addresses, hardware type ID, whether it has data, custom decode code or init code, and its _warning), and the RaftCore version and record format
devtypes/<version>/records/<Key>.json	Each record file, as it is in RaftCore
devtypes/<version>/DeviceTypeRecords.json	All of them in one file

The Device Types app (raftjs apps/devtypes, see also Raft Web Apps) reads it. In the app you can:

• browse and search the records (by name, manufacturer, class or I2C address); a record with a warning is marked, and adding it asks first;
• connect to a device by Bluetooth or USB serial (Chrome or Edge), and see the devices it has found (with catalogue records for addresses it couldn't identify), the types it knows and its type file;
• build a type file from RaftCore's records, git sources on GitHub (read at the commit their ref resolves to) and record files on your computer, as a DeviceTypeSources.json describes them. A project's DeviceTypeSources.json and lock can be imported, and give the same type file as raft devtypes typefile;
• push the file to the device (upload, the device's check, apply, restart, verify), as raft devtypes push does, and read the device's file back;
• download the setup as a project: DeviceTypeSources.json (as typeFile, or builtin to compile the records in), its lock, local record files and devlib.json.

Records from a git source whose decode or init code uses anything outside the decode language are refused in the app, as in a build; such code in local record files is shown as a problem.

A page on https can't open a WebSocket to a device on your network, so the hosted app has no WiFi connection. Run the app on your computer (npm start in apps/devtypes, served on localhost) to use WiFi. The same logic is in raftjs for other apps (buildTypeFile, resolveDeviceTypeSelection, RaftCoreCatalogue, readGitHubSource, DeviceTypeFileManager), and is kept the same as the Python by shared test vectors (scripts/tests/typefile_vectors.json, selection_vectors.json).

raft CLI

raft devtypes (alias raft dt) works on a project's selection files (-s <SysType>), or on one DeviceTypeSources.json with --config <file>. It runs scripts/DeviceTypesTool.py from the project's RaftCore (raftdevlibs/RaftCore, the build's copy, or --raftcore), so the rules are those of the firmware being built. Once the project has been built, type files are built against the records that build compiled (build/<SysType>/raft/DeviceTypeRecords.compiled.json - the firmware the device runs); before that, against the builtin selection.

Command	Does
list	The records for builtin and typeFile, with source, commit, policy, and for the type file new / override / unchanged / policy and records with no on-device decoder
init	Create the SysType's DeviceTypeSources.json (Common's with --common) selecting raftcore:*
add <pattern>...	Add to builtin.include (--typefile: typeFile.include). A pattern that doesn't resolve isn't added
remove <pattern>...	Remove from include; a record selected by a wildcard is added to exclude
acknowledge <key>...	Accept a record's warning (adds it to acknowledge). add, list (! before the key), check and typefile report the warnings of selected records not acknowledged
new <key> [--source <name>] [--address 0xHH]	A new record file in a path source, with the fields to fill in
check	The generator's checks on the compiled selection, and grammar and decode code checks on type file records
update	Resolve git sources' refs again and update the lock file
diff [<key>]	Local copies against the records they replace, and type file overrides against the compiled record
typefile	Build the type file (build/<SysType>/raft/devlib.json, or --out), reporting size, CRC and kinds
push <device>	Build the type file, upload it as devlib.new.json, show the device's check, ask, apply, wait for the restart and confirm the device loaded it. --dry-run stops after the check; --yes doesn't ask; warns if the device's file was changed since the last push from the project, and shows records with a warning and asks
pull <device>	Download the device's type file and list the records that differ from the project's
status <device>	What the device loaded at startup (per entry), its limits, and whether it is the project's file

<device> is the device as for raft fs: an IP address or hostname (HTTP; -p <port> for a port other than 80), ws://host (the device's WebSocket), serial:<port> (its USB serial console) or ble:<name> - see Raft CLI. It can also come from -d/--device or the RAFT_DEVICE environment variable. Name a device by hostname rather than IP address for push, which restarts it: it may come back with another address. The same steps are available from raftjs (checkTypeFile, applyTypeFile, getTypeFileStatus).

Plan

See devdocs/device-type-record-management-plan.md for the whole design, including:

• builds with no compiled records.