ProtocolExchange

ProtocolExchange

ProtocolExchange is the SysMod that decodes incoming framed messages, dispatches REST requests to the RestAPIEndpointManager, and owns the file/stream sessions used for uploads, downloads and OTA. It sits between the Comms Channels layer and the application's REST endpoints.

For the wider picture see Communications Stack Overview.

Responsibilities

1. Register the standard protocol codecs (RICSerial, RICFrame, RICJSON) with the Comms Channel Manager so any channel that names one of these protocols gets a working codec.
2. Receive every framed CommsChannelMsg that comes back from those codecs.
3. Decode the inner RICREST message when the protocol is MSG_PROTOCOL_RICREST and switch on its element code.
4. Dispatch URL-style requests to RestAPIEndpointManager::handleApiRequest() and wrap the response into a CmdRespJSON element to return on the originating channel.
5. Own file/stream sessions for chunked uploads, downloads (see File Download Protocol), and firmware updates.
6. Handle bridge envelopes (MSG_PROTOCOL_BRIDGE_RICREST) by unwrapping and forwarding to the appropriate bridge.
7. Notify SysManager about file/stream/firmware activity via a hook function.

Lifecycle

ProtocolExchange is itself a SysMod and is created early by RaftCoreApp (or whatever code creates SysManager). At construction it does very little; the protocol set-up happens in addCommsChannels():

void ProtocolExchange::addCommsChannels(CommsCoreIF& commsCore)
{
    // Each ProtocolCodecFactoryHelper carries:
    //  - protocol name (matches the name used in registerChannel())
    //  - factory function
    //  - the SysMod's config + a prefix
    //  - inbound rx callback (= processEndpointMsg)
    //  - inbound canAccept callback (= canProcessEndpointMsg)
    commsCore.addProtocol(/* RICSerial helper */);
    commsCore.addProtocol(/* RICFrame  helper */);
    commsCore.addProtocol(/* RICJSON   helper */);
}

Because all three codecs share the same rx callback (processEndpointMsg), every framed message — from any transport using any of these codecs — is funnelled through one method.

processEndpointMsg

This is the central dispatch:

CommsChannelMsg → switch(getProtocol())
   ├─ MSG_PROTOCOL_RICREST          → decode RICRESTMsg, switch on RICRESTElemCode
   │     ├─ URL          → processRICRESTURL          → RestAPIEndpointManager::handleApiRequest
   │     ├─ Body         → processRICRESTBody         (not implemented - no response)
   │     ├─ CmdRespJSON  → processRICRESTCmdRespJSON  (not used inbound - no response)
   │     ├─ CommandFrame → processRICRESTCmdFrame     → cmdName + JSON args → handleApiRequest
   │     └─ FileBlock    → processRICRESTFileStreamBlock → FileStreamSession
   ├─ MSG_PROTOCOL_BRIDGE_RICREST   → unwrap + bridgeHandleInboundMsg
   ├─ MSG_PROTOCOL_RAWCMDFRAME      → JSON cmdName + query string → handleApiRequest
   └─ MSG_PROTOCOL_ROSSERIAL        → not used in this direction

For successful URL/CommandFrame requests the response string is wrapped in a CmdRespJSON RICREST message, marked as a response (endpointMsg.setAsResponse(cmdMsg) copies channelID and msgNumber), and sent via commsCore.outboundHandleMsg(). If the handler returns an empty response nothing is sent, and the client only sees a timeout.

If processing exceeds MSG_PROC_SLOW_PROC_THRESH_MS (50 ms) a warning is logged (processEndpointMsg SLOW took ...) — handlers should return quickly and offload long work to their own loop, a task, or the file system worker, deferring their response if need be.

Deferred responses

A RICREST request with a non-zero message number can have its response deferred, because the client matches a response to its request by message number. For such a request ProtocolExchange sets canDefer, deferToken, msgProtocol and msgNum in the APISourceInfo it passes to the endpoint (for URL and CommandFrame elements).

• If the handler returns RAFT_RESP_DEFERRED, no response is sent now. When the handler calls completeDeferred() (or the deferral times out), the response is sent as a CmdRespJSON response on the same channel with the request's message number.
• A request with the same channel and message number as one whose response is still deferred is ignored, so a client that resends a request while it waits doesn't start it twice.
• Message number 0 (unnumbered) can't be deferred: the handler must respond at once.

See Deferred Responses for the handler side.

RICREST element codes (summary)

Code	Direction	Purpose
URL	Inbound	A REST request as a URL string (endpoint/path?query=...). Mapped 1:1 to RestAPIEndpointManager::handleApiRequest.
Body	Inbound	Body data for a POST-style request. Decoded, but POST over RICREST isn't implemented, so it gets no response.
CmdRespJSON	Outbound	JSON response payload (NUL-terminated). An inbound CmdRespJSON is decoded but not acted on and gets no response.
CommandFrame	Inbound	A JSON object with cmdName and arbitrary fields, treated as cmdName?key=value&....
FileBlock	Inbound	A binary chunk of an active file/stream session (streamID + 24-bit file position + payload).

For the wire layout of each element see RICREST Protocol.

File and stream sessions

ProtocolExchange owns the FileStreamSession objects. MAX_SIMULTANEOUS_FILE_STREAM_SESSIONS is 3, but a new session is only refused when more than that are open, so up to four can be open at once. A start message (ufStart, dfStart) for a file name that already has a session on the same channel goes to that session rather than opening another (a ufStart for an upload that has already received data fails with transferInProgress). A session with no activity for 10 s is closed. A session is created on demand when:

• A dfStart command (file download — see File Download Protocol) is received.
• An upload is started with ufStart.
• A real-time stream is started with ufStart and fileType: "rtstream" (see Real-Time Streams).
• A firmware update is initiated.

Each session is identified by a unique streamID, a channel ID (so concurrent sessions on different channels don't collide), and a content type / flow type. Stream IDs on the wire are non-zero values; 0 is reserved as FILE_STREAM_ID_ANY for internal lookup cases where the stream ID is unspecified. Normal FILEBLOCK data for an allocated upload, download, or real-time stream must carry the non-zero session ID in the FILEBLOCK header.

Inbound FileBlock RICREST messages are routed to the matching session by stream ID. The FILEBLOCK header encodes stream identity and position as one big-endian 32-bit word:

(streamID << 24) | (filePos & 0x00ffffff)

The low 24 bits are the position within the file/stream. The high byte is not available for file position data.

For the wire format of the upload and download messages see RICREST Protocol → File transfers.

File system work off the main loop

• Uploads (RICREST and HTTP) are written to the file system on the file system worker, not in the main loop. For RICREST uploads the okto acknowledgement of a batch is held back until its blocks are written, so the host is never more than a batch ahead of what is on the file system. A firmware update is written in the background by the firmware updater (e.g. ESPOTAUpdate) in the same way.
• Downloads read their blocks on the file system worker.
• A session is only removed once its background writes have finished.

ProtocolExchange::loop() services every active session and removes inactive ones. When file/stream/firmware-update activity changes, the registered fileStreamActivityHookFn is invoked so SysManager can update its busy-state indicator.

Hooks

void setFileStreamActivityHook(FileStreamActivityHookFnType cb);
void setFWUpdateHandler(RaftSysMod* pFirmwareUpdater);

// HTTP file uploads use this entry point directly
RaftRetCode handleFileUploadBlock(const String& req,
                                  FileStreamBlock& block,
                                  const APISourceInfo& sourceInfo,
                                  FileStreamBase::FileStreamContentType contentType,
                                  const char* restAPIEndpointName);

// HTTP uploads are written in the background
bool fileUploadCanAccept(uint32_t channelID);       // false while 16 KB waits to be written (the web server stops reading)
bool isFileUploadWriting(uint32_t channelID);
RaftRetCode getFileUploadWriteStatus(uint32_t channelID);  // RAFT_BUSY while still being written

FileManager's fileupload endpoint uses these: its ready callback returns fileUploadCanAccept(), so the web server stops reading the request body (and TCP flow control slows the client) while HTTP_UPLOAD_PENDING_MAX_BYTES (16384) are waiting to be written, and its reply is deferred until the file is written (getFileUploadWriteStatus()), reporting FileWriteFailed if writing failed.

setFWUpdateHandler lets the application plug in a firmware-update SysMod (e.g. ESPOTAUpdate from RaftSysMods); ProtocolExchange routes firmware streams to it.

Configuration

ProtocolExchange is created by RaftCoreApp with the SysMod name ProtExchg. Its own behaviour is determined by the standard protocol set it registers and the channels created by other SysMods. The only settings are the RICSerial codec's, read from ProtExchg/RICSerial in the SysType:

Key	Default	Meaning
MaxRxMsgLen	5000, or 200000 with PSRAM	Longest HDLC frame received; longer frames are dropped
MaxTxMsgLen	5000, or 200000 with PSRAM	Size limit of MiniHDLC's transmit buffer. Not used by the one-pass encoder ProtocolRICSerial normally uses
FrameBound	0xE7	HDLC frame boundary octet
CtrlEscape	0xD7	HDLC escape octet

"ProtExchg": {
    "RICSerial": { "MaxRxMsgLen": 20000 }
}

Diagnostics

getDebugJSON() returns a JSON array describing each active file/stream session — name, channel ID, stream ID, current state. Visible via sysmoddebug/<protoExchSysModName> if reporting is enabled.

Related

• Communications Stack Overview
• Comms Channels
• RICREST Protocol
• Real-Time Streams
• Adding REST API Endpoints
• File Download Protocol (OKTO)