FileDownloadProtocol

File Download Protocol (OKTO)

The File Download OKTO Protocol handles downloading files from a Raft device to a connected client (e.g. via BLE or WebSocket). It uses a batch-acknowledged block transfer with flow control and CRC integrity checking. Uploads to the device use the same scheme in the other direction — see Uploads (ufStart).

Protocol Flow

Start (dfStart)

The client initiates a download by sending a dfStart command:

{
  "cmdName": "dfStart",
  "reqStr": "getFile",
  "fileType": "fs",
  "fileName": "data/log.csv",
  "batchMsgSize": 5000,
  "batchAckSize": 4,
  "crcAt": "end"
}

Field	Description
fileName	Path to the file on the device filesystem; /local/... or /sd/... names the file system, otherwise the default one is used
fileType	Source type, typically "fs" for filesystem
batchMsgSize	Requested block size in bytes (default 5000)
batchAckSize	Number of blocks per batch before requiring an ack (default 40)
crcAt	Optional. Set to "end" to defer CRC computation. Omit for legacy behaviour (CRC at start).

The server responds with:

{
  "req": "dfStart",
  "rslt": "ok",
  "batchMsgSize": 5000,
  "batchAckSize": 4,
  "streamID": 1,
  "fileLen": 45678,
  "crc16": "a1f2"
}

Field	Description
batchMsgSize	Actual block size: the requested size limited to between 20 bytes and the channel's maximum outbound block size
batchAckSize	Actual batch ack size
streamID	Non-zero session identifier for this transfer. The FILEBLOCK field can represent 1..255; 0 is reserved internally as FILE_STREAM_ID_ANY and is not a valid data-block session ID.
fileLen	Total file size in bytes
crc16	Present only when CRC is computed at start (legacy mode or old firmware). CRC-16/CCITT of the entire file, hex-encoded.

If the download can't start — for example the file doesn't exist — the reply is a failure, so the client needn't wait for a timeout:

{"req": "dfStart", "reason": "sessionNotStarted", "rslt": "fail"}

Data Blocks

The server sends file data as binary RICREST file-block frames. Blocks are sent in batches of batchAckSize. Each block contains:

• The allocated streamID from dfStart
• File position (24-bit byte offset)
• Block data

The FILEBLOCK header encodes the stream ID and file position as:

[streamID:1 byte][filePos:3 bytes big-endian][payload:N bytes]

Equivalently, the implementation stores the header position word as:

(streamID << 24) | (filePos & 0x00ffffff)

Download FILEBLOCKs for a dfStart session must carry the same non-zero streamID returned in the dfStart response. streamID 0 means "unspecified / any" inside the session lookup code and must not be emitted for normal transfer data.

Acknowledgement (dfAck)

After receiving a batch of blocks, the client sends an acknowledgement:

{
  "cmdName": "dfAck",
  "okto": 20000,
  "streamID": 1,
  "rslt": "ok"
}

The okto field tells the server how many bytes have been successfully received. The server uses this to advance its send window. If no ack is received within the timeout, the server retries from the last acknowledged position.

The streamID field must match the active dfStart session being acknowledged.

End (dfEnd)

After receiving all data, the client sends a dfEnd message and waits for the response:

{
  "cmdName": "dfEnd",
  "reqStr": "getFile",
  "fileName": "data/log.csv",
  "fileLen": 45678,
  "streamID": 1
}

The server responds with:

{
  "req": "dfEnd",
  "rslt": "ok",
  "crc16": "a1f2"
}

The crc16 field is present when CRC was deferred to the end ("crcAt": "end" in dfStart). The client verifies the file integrity by computing CRC-16/CCITT over the received data and comparing.

With the CRC at the end, the session waits for dfEnd for up to 2 s (END_MSG_WAIT_MS) after the last data has been acknowledged. A dfEnd sent later finds no session and gets a plain {"rslt":"ok"} without crc16, so send it straight after the final dfAck. (Without crcAt, the session ends as soon as the last data is acknowledged.)

Cancel (dfCancel)

Either side can cancel the transfer:

{
  "cmdName": "dfCancel",
  "reqStr": "getFile",
  "streamID": 1
}

The server also sends a cancel message if it detects a timeout or error, including a reason field.

CRC Computation Modes

The protocol supports two modes for CRC-16/CCITT integrity checking:

Legacy Mode (CRC at Start)

When crcAt is omitted or set to "start" in the dfStart request:

1. Server reads the entire file to compute CRC before responding to dfStart
2. CRC is included in the dfStart response
3. Client stores CRC and verifies after all data is received
4. dfEnd response does not include CRC

This mode causes a blocking delay at the start of transfer proportional to file size, as the file must be read twice (once for CRC, once for sending).

Deferred Mode (CRC at End)

When crcAt is set to "end" in the dfStart request:

1. Server responds to dfStart immediately without reading the file for CRC
2. CRC is computed incrementally as each block is sent during the transfer
3. CRC is included in the dfEnd response
4. Client verifies CRC after receiving the dfEnd response

This mode eliminates the blocking delay at transfer start and avoids reading the file twice.

Backward Compatibility

The crcAt field provides seamless backward compatibility between old and new versions of firmware and client libraries:

Firmware	Client	Behaviour
Old	Old	Legacy mode. CRC in dfStart response.
Old	New	New client sends crcAt:"end", old firmware ignores unknown field. CRC still in dfStart response. Client detects CRC in dfStart and uses it.
New	Old	Old client doesn't send crcAt. New firmware defaults to legacy mode. CRC in dfStart response.
New	New	Deferred mode. No CRC in dfStart. Incremental CRC in dfEnd response. Non-blocking start.

Flow Control

• Block timing: Minimum MIN_TIME_BETWEEN_BLOCKS_MS (100ms) between blocks
• Batch acknowledgement: Server sends batchAckSize blocks then waits for a dfAck
• Ack timeout: If no ack within BLOCK_MSGS_TIMEOUT_MS (3s), server retries from last acked position
• Max retries: MAX_BATCH_BLOCK_ACK_RETRIES (10) before cancelling
• Heap throttling: Blocks are paused when free internal heap drops below HEAP_LOW_WATER_MARK_BYTES (35KB)
• Overall timeout: DOWNLOAD_FAIL_TIMEOUT_MS (2 hours) to accommodate slow BLE transfers
• Reads off the main loop: each block is read on the file system worker, and the next block is read ahead while the current one is sent. While a read is in progress the block read returns RAFT_BUSY and the block is sent on a later pass, without waiting another MIN_TIME_BETWEEN_BLOCKS_MS.
• Read errors: a block that can't be read cancels the transfer (dfCancel with reason readError).

Uploads (ufStart)

A file (or firmware) upload over RICREST uses ufStart, file blocks from the client, okto acks from the device and ufEnd. (HTTP uploads use fileupload instead — see FileManager.)

{
  "cmdName": "ufStart",
  "reqStr": "fileupload",
  "fileType": "fs",
  "fileName": "/sd/data/log.csv",
  "fileLen": 45678,
  "batchMsgSize": 800,
  "batchAckSize": 20,
  "CRC16": "0xa1f2"
}

Field	Description
fileType	"fs" (or "file") for a file; "fw" (or "ricfw") for a firmware update (see OTA Update Flow); "rtstream" for a real-time stream
fileName	Where to write the file. A name starting /local/ or /sd/ names the file system; otherwise the default file system is used. Send /<fs>/<path> to be sure which file system is used (see File System on how names are resolved).
fileLen	File length in bytes
batchMsgSize	Requested block size (default 5000)
batchAckSize	Requested blocks per ack (default 40)
CRC16	Optional: the CRC-16/CCITT of the whole file (as filehash gives it), as "0x...." hex or decimal. The file is checked against it once written - see Checking the file

The reply gives the sizes actually used, and the streamID, which the client must use:

{"req": "ufStart", "batchMsgSize": 800, "batchAckSize": 20, "streamID": 2, "rslt": "ok"}

The device reduces the requested sizes:

• The block size is limited to between 20 bytes and 2/3 of the channel's maximum inbound block size (e.g. 800 bytes for a channel whose inbound block is 1200 bytes).
• Block size × batch size is limited to 50000 bytes (MAX_TOTAL_BYTES_IN_BATCH); the batch size is reduced.
• The batch must fit in memory: each block is held about four times over while it is received, and half the largest free memory block (after a 16 KB margin) may be used. When memory is short the batch size, and then the block size, are reduced.

A repeat ufStart for an upload that has already received data fails with transferInProgress.

Blocks and acks

Blocks are FILEBLOCK frames carrying the streamID and file position, as for downloads. The device acks with {"req":"ufBlock","okto":<position>,"rslt":"ok"}:

• after the first block, after every batchAckSize blocks, and after the final block;
• at once, with the position wanted, on the first block out of sequence (blocks out of sequence are dropped);
• on a timer if blocks stop arriving: after 5 s while fewer than two blocks have been received, then after 1 s. After 4 timer acks with no blocks in between, the upload is cancelled (ufCancel, reason failRetries).

The client resends from the okto position (go-back-N).

Writes in the background

File uploads are written on the file system worker, not in the main loop, and a firmware upload is written on it by ESPOTAUpdate. An ack is held back while blocks received are still being written, and sent once they are, so the client is never more than a batch ahead of what is on the file system (or in flash). While writes are pending no timer acks are sent.

If a background write fails, the error sticks: later blocks aren't written, and the upload is cancelled with ufCancel and reason failFileWrite (failOTAWrite for firmware; failCRC when the file doesn't match its CRC16). When write pacing is holding off writes, the device leaves received RICREST messages queued (on every channel) until the hold-off ends, so the acks — and the client — slow down.

The upload's session ends when the final block has been written. ufEnd replies ok, including when the session has already ended.

Checking the file (CRC16)

When ufStart gives a CRC16, the device works out the CRC-16/CCITT of the file as it writes it (on the file system worker) and, once the final block is written, compares the two:

• They match: the final ack is sent as usual, and the ufEnd reply says the file was checked: {"req":"ufEnd","crc16":"a1f2","rslt":"ok"}. The device keeps this after the session ends, so the ufEnd that follows the last ack gets it. A client that gets crc16 back (equal to what it sent) needn't check the file with filehash.
• They don't: the file is deleted, and in place of the final ack the device sends ufCancel with reason failCRC (or replies to the block with "reason":"CRCMismatch" when blocks are written as they arrive). The ufEnd reply has no crc16.

Firmware with RaftCore 1.63.0 or earlier ignores CRC16 and its ufEnd reply never has crc16: a client that doesn't get one back should check the file with filehash, as RaftCLI does. Firmware updates aren't checked against CRC16 (ESP-IDF validates the image).

For real-time streams that use ufStart, see Real-Time Streams and ProtocolExchange.

Implementation

Server-side files in RaftCore/components/comms/:

File	Purpose
FileStreamProtocols/FileDownloadOKTOProtocol.h/.cpp	Protocol state machine, block sending, ack handling, CRC modes
FileStreamProtocols/FileUploadOKTOProtocol.h/.cpp	Upload protocol: block and batch sizing, acks (held while blocks are written), timer acks
ProtocolExchange/FileStreamSession.h/.cpp	Session management, file chunker access, CRC callback, reads and writes on the file system worker
FileStreamProtocols/FileStreamBase.h	Base class with callback type definitions

Client-side files in raftjs/src/:

File	Purpose
RaftFileHandler.ts	File download orchestration, CRC verification
RaftTypes.ts	Response type definitions, including RaftFileStartResp and RaftFileDownloadResult