Skip to content

api: Add a /diag endpoint reporting why the device last restarted - #112

Merged
ccrisan merged 1 commit into
mainfrom
diag-endpoint
Oct 17, 2026
Merged

ccrisan merged 1 commit into
mainfrom
diag-endpoint

Conversation

@ccrisan

@ccrisan ccrisan commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator

Makes the reboot cause observable in a normal release build, without a debug image.

Why

An unexpected reset currently leaves no trace anyone can read in production:

  • system_check_reboot_loop() decodes rst_info->reason at every boot and writes it only to DEBUG_SYSTEM(...), which compiles to nothing in a release build.
  • The allocators call rtc_reset() before system_restart(). That clears FULL_BOOT_TEST_ADDR, so the next boot is treated as a full boot and rtc_get_value() returns 0 for everything — even the unexpected-reset counter is lost.

The endpoint

GET /diag          Authorization: Bearer <admin>

{
    "last_reset": {"reason": "out-of-memory", "alloc_size": 1536},
    "free_mem": 17584,
    "boot_count": 42
}

reason is one of unknown, power, external, software, exception, hardware-watchdog, software-watchdog, deep-sleep-wake, out-of-memory. alloc_size is the size of the allocation that failed, and 0 for every other reason.

free_mem is in bytes — unlike the device's mem_usage, which is a percentage of MAX_AVAILABLE_RAM and hides the real figure. boot_count counts boots since the last full boot, so a burst of resets is visible without history.

This sits on its own endpoint rather than among the device attributes, since it describes the firmware's own health and has nothing to do with the qToggle API.

Two details worth knowing

  • An allocation failure restarts through system_restart(), which the SDK reports as a plain software reset or, depending on where it was called from, as an exception. Both were observed on the bench, so the RTC record is the only thing that identifies it. The magic word is mixed with the size so that both slots have to agree, and the record is cleared on every boot so it always describes exactly one reset.
  • system_record_alloc_failure() allocates nothing. It writes straight to RTC memory through new raw accessors that bypass rtc_get_value()'s full-boot gate — which matters because it runs when the heap is already exhausted, and because rtc_reset() clears that gate on the way out.

Tests

test/test-cases/02-api/18-diag/ — response shape, the auth gate (401/403), and a real POST /reset that waits for the device to drop off the AP and come back, then asserts the reason is exactly software with alloc_size 0. That exercises the whole chain end to end on hardware.

All three pass on the bench, along with the rest of the API suite. DEBUG=false (which adds -Werror) and DEBUG=true DEBUG_FLAGS="system rtc api" both build clean.

Note

This endpoint is how the reboot in #111 was classified. Worth having on field devices regardless of that fix, so the next unexplained reset reports itself.

An unexpected reset currently leaves no trace a release build can report.
system_check_reboot_loop() decodes rst_info->reason at every boot and
writes it only to DEBUG_SYSTEM(), which compiles to nothing unless the
firmware is built with debugging. Worse, the allocators call rtc_reset()
before system_restart(), which clears FULL_BOOT_TEST_ADDR and makes
rtc_get_value() return 0 for everything on the next boot, so even the
unexpected-reset counter is lost.

GET /diag (admin) reports:

    {
        "last_reset": {"reason": "out-of-memory", "alloc_size": 1536},
        "free_mem": 17584,
        "boot_count": 42
    }

reason is one of unknown, power, external, software, exception,
hardware-watchdog, software-watchdog, deep-sleep-wake or out-of-memory.
alloc_size holds the size of the allocation that failed and is 0 for every
other reason. free_mem is in bytes, unlike the device's mem_usage, which is
a percentage of MAX_AVAILABLE_RAM and hides the real figure.

This lives on its own endpoint rather than among the device attributes,
since it describes the firmware's own health and has nothing to do with the
qToggle API.

Two details worth knowing:

  * An allocation failure restarts through system_restart(), which the SDK
    reports as a plain software reset or, depending on where it was called
    from, as an exception. Both were observed on the bench, so the RTC
    record is the only thing that identifies it. The magic is mixed with
    the size so both slots have to agree, and the record is cleared on
    every boot so it always describes exactly one reset.

  * system_record_alloc_failure() allocates nothing. It writes straight to
    RTC memory through new raw accessors that bypass rtc_get_value()'s
    full-boot gate, which matters because it runs when the heap is already
    exhausted and because rtc_reset() clears that gate on the way out.

Tests under test/test-cases/02-api/18-diag/ cover the response shape, the
auth gate, and a real POST /reset followed by asserting the reason is
exactly "software" with alloc_size 0.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It modifies the reliability-critical allocator-reset path and hardware-specific RTC memory, which can only be built and validated on real ESP8266 hardware and not in this environment.

No open issues 🎉

What changed in this PR

Adds a new admin-only GET /diag endpoint that reports why the device last restarted (including an out-of-memory cause that the SDK otherwise masks), the free heap in bytes, and the boot count. To make out-of-memory restarts observable in release builds, the allocator failure path now records the failed allocation size to RTC memory via new raw accessors that bypass the full-boot gate, and system_check_reboot_loop() decodes that record into a machine-readable reason string instead of only emitting a debug log.

Changes:

  • New GET /diag API endpoint (api_get_diag) exposing last_reset (reason, alloc_size), free_mem, and boot_count, gated to admin access.
  • Allocator (my_malloc/my_realloc/my_zalloc) records the failed size to RTC before restarting; system_check_reboot_loop() detects it via a magic-XOR-size record and maps all SDK reset reasons to stable strings.
  • New rtc_get_raw_value/rtc_set_raw_value accessors that skip the full-boot gate, plus three blackbox test cases covering response shape, the auth gate, and reason after a real reset.
File Description
src/​api.c Adds /diag dispatch and the api_get_diag handler building the diagnostics response.
src/​api.h Declares api_get_diag.
src/​espgoodies/​system.c Maps reset reasons to strings, decodes the RTC alloc-failure record, and adds system_record_alloc_failure/getters.
src/​espgoodies/​system.h Declares the new system diagnostics functions with doc comments.
src/​espgoodies/​utils.c Records the failed allocation size before resetting in all three allocator wrappers.
src/​espgoodies/​rtc.c Implements raw (un-gated) RTC read/write accessors.
src/​espgoodies/​rtc.h Declares the raw RTC accessors.
test/​test-cases/​02-api/​18-diag/​01-get-diag.json Validates the full /diag response schema.
test/​test-cases/​02-api/​18-diag/​02-diag-after-reset.json Asserts reason software/alloc_size 0 after a real POST /reset.
test/​test-cases/​02-api/​18-diag/​03-diag-security.json Verifies the 401/403 auth gate.

Review effort: Balanced

@ccrisan
ccrisan merged commit f944166 into main Oct 17, 2026
4 checks passed
@ccrisan
ccrisan deleted the diag-endpoint branch October 17, 2026 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants