FileSystem

File System

Raft mounts a flash-resident file system (LittleFS or SPIFFS) and, optionally, an SD card. The same FileSystem object is used by every SysMod, by the WebServer for static-asset delivery, by the DataLogger for autonomous CSV/JSONL capture and by FileManager to expose REST endpoints. This page covers how data gets into a build, how it is mounted at runtime, and which API to call from C++.

The code lives at components/core/FileSystem/ in RaftCore. The single global FileSystem fileSystem; instance is created by RaftCoreApp and configured by the FileManager SysMod using values from your SysType JSON.

Where the file system lives

A Raft device exposes up to two file systems, identified by name:

FS name	Mount point	Backing storage	Notes
local	/local	LittleFS or SPIFFS in the fs partition	Always present unless explicitly disabled. Alias spiffs accepted for legacy callers.
sd	/sd	FAT-on-SPI SD card	Optional; requires RAFT_ENABLE_SD and pin configuration.

When a REST API or C++ call passes an empty file-system name, getDefaultFSRoot() picks one. The default is local, but if DefaultSD is set in FileManager configuration and an SD card is present at boot, default operations resolve to sd instead.

The local partition is always labelled fs (with spiffs as a fallback label for older partition tables) — see Partitions and Flash Layout for the layout details.

LittleFS vs SPIFFS

Raft supports both, selected at build time by the SysType's FS_TYPE value (in features.cmake):

set(FS_TYPE "littlefs")     # or "spiffs"

This single value drives three things in the build:

1. Which IDF component is linked. RaftCore/CMakeLists.txt requires littlefs (and defines FILE_SYSTEM_SUPPORTS_LITTLEFS) for littlefs, or spiffs (and defines RAFT_FILESYSTEM_HAS_SPIFFS) otherwise.
2. Which image generator is used. RaftGenFSImage.cmake calls littlefs_create_partition_image(fs … FLASH_IN_PROJECT) or spiffs_create_partition_image(fs … FLASH_IN_PROJECT) accordingly.
3. Which subtype your partitions.csv must declare for the fs partition: 0x83 for LittleFS, spiffs for SPIFFS. Mismatches cause mount failure on first boot — and a re-format if LocalFsFormatIfCorrupt is set.

Recommendation: prefer LittleFS for new projects. It tolerates power loss, supports nested directories and is the default in current Raft examples. Keep SPIFFS only for legacy partition layouts that cannot be reformatted in the field.

At runtime, FileSystem::localFileSystemSetup() actually probes both — if LittleFS support is compiled in, it tries to mount LittleFS first regardless of LocalFsDefault, and only falls back to SPIFFS if the partition does not look like LittleFS. This means a device that originally shipped with SPIFFS can be migrated to LittleFS by switching FS_TYPE and reflashing.

Including files in the build (FS image)

Anything you want pre-loaded into local at first boot goes into a file-system image that is built and flashed alongside the firmware.

The Raft pipeline accepts content from two places, merged together by CopyFSAndWebUI:

+--------------------------+      +------------------------------+
| FS_IMAGE_PATH (your data |      | UI_SOURCE_PATH (Web UI src)  |
| files: configs, scripts, |      | parcel/npm bundle output     |
| audio, calibration .json |      | gzipped .html/.js/.css       |
| …)                       |      |                              |
+------------+-------------+      +--------------+---------------+
             \\                                  /
              \\        CopyFSAndWebUI         /
               \\______________________________/
                          |
                          v
            build/<sys>/raft/FSImage/
                          |
                          v
       littlefs_create_partition_image(fs FSImage/ FLASH_IN_PROJECT)
                          |
                          v
                    fs.bin (flashed with the rest of the project)

Adding a data folder

In your SysType's features.cmake:

set(FS_TYPE         "littlefs")
set(FS_IMAGE_PATH   "../Common/FSImage")        # source folder, relative to SysType
set(UI_SOURCE_PATH  "../Common/WebUI")          # optional; see WebUIBuildPipeline

Anything under systypes/Common/FSImage/ (or whatever path you set) is copied verbatim into the file-system image. Typical contents:

systypes/Common/FSImage/
├── config/
│   └── default-pipeline.json     ← read by a SysMod at boot
├── certs/
│   └── ca.pem                    ← MQTT TLS root cert
└── tunes/
    ├── startup.wav
    └── alarm.wav

After flashing, those appear at /local/config/default-pipeline.json, /local/certs/ca.pem, etc.

Skipping the FS image

If a SysType has no Web UI and no data files, leave FS_IMAGE_PATH unset. RaftGenFSImage.cmake is still useful (the partition is created empty) but the fs.bin will be tiny.

To skip image creation entirely, drop the include(${raftcore_SOURCE_DIR}/scripts/RaftGenFSImage.cmake) line from your project's top-level CMakeLists.txt — the partition will be left blank on flash, and the device will format it on first boot if LocalFsFormatIfCorrupt is set.

Sizing

The FS image must fit the fs partition or the build fails. Typical sizes:

• 256 KB — a few configs, no Web UI.
• 512 KB — the parcel-built React/MUI starter Web UI.
• 1–4 MB — Web UI plus audio/image assets.

See Partitions and Flash Layout to grow the partition.

Adding an SD card

Set RAFT_ENABLE_SD (a project-level CMake variable, often added to your top-level CMakeLists.txt or features.cmake) to pull in the fatfs and sdmmc ESP-IDF components and define RAFT_FILESYSTEM_HAS_FATSD:

set(RAFT_ENABLE_SD ON CACHE BOOL "" FORCE)

Then enable and configure SD pins in your SysType's SysTypes.json:

{
    "name": "FileManager",
    "LocalFsDefault": "littlefs",
    "SDEnabled": 1,
    "SDMOSI": "23",
    "SDMISO": "19",
    "SDCLK":  "18",
    "SDCS":   "5",
    "DefaultSD": 0,
    "CacheFileSysInfo": 1
}

Pin values can be GPIO numbers ("5") or symbolic pin names resolved by ConfigPinMap (e.g. "SD_CS"). With SDEnabled set the FileManager calls FileSystem::setup(...) with SD parameters; the FileSystem then mounts FAT on the card and exposes it as /sd. Set DefaultSD to make file ops with no FS-name argument resolve to the SD card.

Mounting at runtime

The FileSystem mount sequence happens during the FileManager SysMod's setup(). The sequence is:

1. FileSystem::setup(localFsType, formatIfCorrupt, enableSD, …) is called with SysType-derived parameters.
2. Local FS is probed:
   • LittleFS first (if compiled in). On success, _localFsType = LOCAL_FS_LITTLEFS and the partition is mounted at /local.
   • SPIFFS next (if compiled in and LittleFS failed).
   • If both fail, the FS is formatted (if LocalFsFormatIfCorrupt is true) and re-tried.
   • If everything fails, _localFsType = LOCAL_FS_DISABLE and local becomes unavailable.
3. SD is initialised if SDEnabled. Mount failure is logged but never fatal.
4. The cache is primed (10 service iterations) so the first filelist call returns immediately.

The cache (enabled with CacheFileSysInfo) is critical for serving Web UI listings on resource-constrained boards — getFilesJSON() uses it directly when the request is for the FS root. After boot, a cache made out of date (e.g. by closing a file that was written) is refreshed on the file system worker, not in the main loop.

C++ API

All APIs go through the global fileSystem object. The most useful entry points:

#include "FileSystem.h"

// Pass "" as fileSystemStr to use the default FS

// One-shot read (returns a heap-allocated, NUL-terminated buffer; caller frees)
uint8_t* pBuf = fileSystem.getFileContents("", "config/default.json");
if (pBuf) {
    // … use as char* …
    free(pBuf);
}

// One-shot write
String contents = "key=value";
fileSystem.setFileContents("local", "settings.txt", contents);

// Existence and metadata
uint32_t size = 0;
if (fileSystem.getFileInfo("", "tunes/startup.wav", size)) { … }

// Streamed I/O — preferred for anything that might be large
FILE* pFile = fileSystem.fileOpen("", "log.csv", /*write=*/true, /*seekToPos=*/0, /*seekFromEnd=*/true);
if (pFile) {
    fileSystem.fileWrite(pFile, (const uint8_t*)line, lineLen);
    fileSystem.fileClose(pFile, "", "log.csv", /*fileModified=*/true);
}

// Rename (atomic on the same file system - used to install an uploaded file in place of another, e.g.
// devman/typefile/apply; an existing file of the new name is replaced)
fileSystem.renameFile("local", "devlib.new.json", "devlib.json");

// Size and space used (cached; false if not known - e.g. an SD card's used bytes are still being computed)
uint64_t sizeBytes = 0, usedBytes = 0;
if (fileSystem.getFileSystemSize("local", sizeBytes, usedBytes)) { … }

// Reformat (destructive)
String resp;
fileSystem.reformat("local", resp, /*force=*/false);

Use FileSystemChunker when reading or writing in fixed-size blocks — the file upload and download protocols use it. The chunker keeps the FILE* open across calls and serialises mutex acquisition on its own.

A few invariants worth knowing:

• A lock for each file system. FileSystem serialises operations with one lock for local and another for sd, so a slow write to one doesn't hold up the other. Every call waits for its file system's lock with no timeout, so a long run of reads or writes holds up every other user of that file system: work in pieces (about 4 KB) rather than in one long call. An open FILE* uses the lock of the file system it was opened on (FileSystem keeps a list of the files open on the SD card for this, so open and close files with fileOpen() and fileClose()).
• Writes are slow and can stop everything. See Keeping file system work off the main loop: do slow file system work on the file system worker, not in loop().
• getFileContents allocates from PSRAM if available. It returns nullptr if the file is larger than ~⅓ of the largest available allocation block. For files larger than that, use the streamed API or a FileSystemChunker.
• Filenames may include the FS name. "/local/foo.txt", "local/foo.txt" and "foo.txt" (with fileSystemStr="local") all resolve to the same path. A name names its file system only at the start (local/..., /sd/..., or just local or sd): logs/sd/x.csv is a file in a folder called sd on the file system given. (In RaftCore 1.63.0 and earlier a local/ or sd/ anywhere in the name was taken as naming the file system, and could pick the wrong file system's lock.)

Keeping file system work off the main loop

The main loop should not be held up for more than about 10 ms. File system work often takes longer: a LittleFS write that needs a 4 KB flash block erased can take 50-160 ms, an SD write of a few KB tens of ms, and reading a 200 KB file to work out its hash took 469 ms (LittleFS) and 906 ms (SD) when it ran in the main loop.

The file system worker

FileSystemWorker (FileSystemWorker.h) is one FreeRTOS task shared by everything that does slow file system work. The global instance is fileSystemWorker.

Property	Value
Stack	4 KB (TASK_STACK_BYTES), in internal RAM — a task that writes to flash can't have its stack in PSRAM
Priority	1 (TASK_PRIORITY), below the main task
Core	The core the main task doesn't use (core 0 unless the main task is on core 0)
Queue	8 jobs (QUEUE_LEN), run one at a time in the order submitted
Created	On first use, and kept for the life of the program

#include "FileSystemWorker.h"

// Start the work (_jobDone is a std::atomic<bool> member)
_jobDone = false;
auto job = [this]() {
    // Runs on the worker: file system calls only - nothing that expects the main task
    // (REST replies, SysMod calls, publishing)
    writeTheData();
    _jobDone = true;
};
if (!fileSystemWorker.submit(job))
    job();      // Queue full (or the task couldn't be created): do the work here, or fail

// Later, in loop() on the main task
if (_jobDone.load()) { /* use the result, reply, etc. */ }

submit() returns false if the job can't be queued; the caller must then do the work itself or fail. Completion is checked from the main task (typically a flag the job sets), and anything that must run on the main task, such as completing a deferred REST reply, is done from there. jobsOutstanding() gives the number of jobs submitted and not yet finished.

The framework uses the worker for:

Work	Where
File upload writes (HTTP, and RICREST on any channel)	FileStreamSession — see uploads
File download reads, a block ahead	FileStreamSession — see File Download Protocol
filelist, fileread, filesection, filedelete, filehash	FileManager (with a deferred reply)
Firmware writes (esp_ota_*)	ESPOTAUpdate
Static files served by the web server, a chunk ahead	WebServer (RaftWebResponderFile)
File system info cache refreshes	FileSystem

Uploads are written in pieces of up to 4 KB so the file system lock is released between them, and a reader in the main loop (a static web file, a config file) only waits for one piece.

Flash writes stop the whole chip

Moving a write to the worker takes the file system's own processing out of the main loop, but not the flash hardware time. On the ESP32-S3 (and other ESP32 chips), erasing or programming the flash turns off the flash cache on both cores while it runs, so every task running code from flash stops, wherever the write was made. Only code in IRAM keeps running. An SD card is on its own SPI bus, so SD writes don't do this.

Measurements from flash-write-blocking-plan.md (writing a 200 KB file from a task; longest main loop pass):

Board	File system	Code from flash	Code from PSRAM (XIP)
ESP32-S3, 2 MB PSRAM	SD	6 ms	6 ms
ESP32-S3, 2 MB PSRAM	LittleFS	93-154 ms (about once per 4 KB block)	6 ms
ESP32-S3-WROOM-1-N16R8	LittleFS	about 12 ms	4 ms

Writing from the main loop instead gave passes of 19-367 ms (depending on board, chunk size and file system), with code from flash or from PSRAM: the main loop waits for its own write.

The flash stall goes away when:

• The app runs from PSRAM (CONFIG_SPIRAM_XIP_FROM_PSRAM=y in sdkconfig.defaults). The app's code and read-only data are copied to PSRAM at boot, so the cache stays on during flash operations. It costs about the size of the app image in PSRAM (1.9 MB of 8 MB on the N16R8 board; on the 2 MB board free memory fell from 2.2 MB to 0.6 MB). Boot was no slower, and on the N16R8 board the main loop and I2C polling ran faster, since code came from octal PSRAM rather than the flash.
• The flash chip supports auto-suspend (CONFIG_SPI_FLASH_AUTO_SUSPEND; ESP32-S3 and later, only with the flash chips ESP-IDF lists for it). An erase or program is suspended when the cache needs the flash. Not measured.

Practical guidance:

• Do slow file system work on the worker, not in loop().
• On a board with enough PSRAM, run the app from PSRAM. Otherwise expect pauses of tens of ms or more (depending on the flash chip) whenever a LittleFS write needs a block erased; write pacing limits how much of the time an upload stalls the system.
• For heavy logging, prefer an SD card.
• An OTA update erases and writes the flash too — see OTA Update Flow.

Slow writes and write pacing

fileWrite() counts each write that takes longer than SLOW_WRITE_THRESHOLD_US (10 ms) — usually a flash block erase — in getSlowWriteCount() and getSlowWriteMs() (totals since startup). SysManager uses them to note when a slow SysMod loop coincided with a slow file system write, and reports the totals in sysmoddebug/SysMan (fsSlowWrites, fsSlowWriteMs).

With write pacing on, a slow write also starts a hold-off long enough that slow writes take at most a set percentage of the time. Writers that can wait check isWriteHoldoffActive(); file uploads then stop taking data (see write pacing).

fileSystem.setWritePacing(30);   // at most 30% of the time in slow writes; 0 = no pacing

The default (WRITE_PACING_MAX_STALL_PCT_DEFAULT) is 0 (no pacing) when the app runs from PSRAM (CONFIG_SPIRAM_XIP_FROM_PSRAM) or the flash supports auto-suspend (CONFIG_SPI_FLASH_AUTO_SUSPEND), since a write then doesn't stop other code, and 30 otherwise. The FileManager sets it from its WritePacingPct setting.

REST endpoints

The FileManager SysMod exposes a thin wrapper for HTTP/RICREST clients — filelist, fileread, filesection, filehash, filedelete, fileupload, reformatfs. Folder paths use ~ instead of / so they survive URL parsing. See also Built-in REST Endpoints.

Troubleshooting

Symptom	Likely cause	Fix
localFileSystemSetup no file system found at boot	Partition subtype mismatch (spiffs vs 0x83) or partition table not flashed	Match FS_TYPE to the fs partition subtype, then idf.py erase-flash flash
LittleFS image creation failed: too large	FSImage/ exceeds the fs partition size	Grow fs in partitions.csv or trim assets
getContents … too big to read	File larger than the in-memory threshold	Use fileOpen/fileRead or FileSystemChunker instead
First boot wipes the FS image	Mount failed and LocalFsFormatIfCorrupt reformatted	Verify partition subtype; flash the FS image with idf.py flash (not just app-flash)
unknownfs sd from filelist	SD not enabled or pins wrong	Enable RAFT_ENABLE_SD, set SDEnabled and SPI pins, check card
fileupload succeeds but file missing	Cache returned stale listing	Either set CacheFileSysInfo: 0 while debugging, or re-fetch — uploads call markFileCacheDirty automatically
loop sysMod … SLOW … (slow file system write) during uploads or logging	A LittleFS write erasing a flash block stalls all code running from flash	Run the app from PSRAM (CONFIG_SPIRAM_XIP_FROM_PSRAM), or rely on write pacing; see above

See also

• WebUI Build Pipeline — how Web UI assets are merged with FS_IMAGE_PATH content.
• Partitions and Flash Layout — defining the fs partition.
• FileManager SysMod — config keys and REST surface.
• Built-in REST Endpoints — filelist, fileread, filedelete, fileupload, reformatfs.
• File Download Protocol (OKTO) — file streaming over RICREST, both directions.
• REST API Endpoints — deferred responses — replying once work on the worker is done.
• flash-write-blocking-plan.md — measurements behind the file system worker.
• DataLogger — autonomous CSV/JSONL logging onto the same FS.