FSoE master: operate a safe drive from the API and the Console - #8
Draft
jbck wants to merge 13 commits into
Draft
Conversation
Motion Master can open, hold and diagnose a Safety-over-EtherCAT connection.
libs/etg/fsoe_master.{h,cc} is the ETG.5100 ch. 8.4 master state machine, over
fsoe_frame.{h,cc} for the Safety PDU layout and fsoe_crc.{h,cc} for the Annex A
hash. No new dependency.
This is a protocol master, not a safety master, and the header says so. A
certified master needs certified hardware and an assessed stack; this one runs
in an ordinary process. What makes it useful anyway is that the slave is the one
that stays safe: it authenticates every frame and drops its outputs when the
frames stop, whatever the master does. So a tool can release STO, read safe
position, velocity and torque, and explain a connection fault, without
weakening the safety function. None of that was reachable before without
borrowing a safety PLC.
It lives in etg because it is the same kind of thing as the ESI parser: a
specification implemented as a pure transform. No socket, no thread, no clock,
no allocation after construction. One call per bus cycle takes the octets that
arrived and returns the octets to send, so the transport stays the caller's
business and the state machine is testable without hardware.
Verification is in three layers. Unit tests pin the CRC against known answers,
the frame layout including every illegal length, and the state machine against
a slave double that authenticates every frame. Because a peer written by the
same author repeats any misreading of the standard, the master was then linked
directly against the drive firmware's own FSoE slave and run through a full
connection. fsoe_master_interop_test.cc replays eight cycles of that recorded
exchange octet for octet, with no firmware present. There is no conformance
test for the master role in the lab, which is why that trace carries the weight.
Rationale, and what is deliberately not here: NEXTGEN.md, Session 2026-08-22.
The harness moved next to the slave it links against, so that deleting or moving that library takes its tooling with it.
The FSoE master state machine now reaches the wire. A caller opens a connection to a drive over HTTP, releases Safe Torque Off, and reads the safe position, velocity and torque the drive publishes — each with the drive's own validity flag. The Console has a device Safety page that does all of it. It is still a protocol master, not a safety master, and every surface says so: the header, the swagger text, GET /api/fsoe's own answer, and a banner on the page. What makes it useful is that the drive stays safe by itself — it authenticates every frame and drops its outputs when the frames stop. Two things decided the shape. The frame goes out through parameter cells. exchangeProcessData zeroes the output image and recomposes it from each object's cell every cycle, so a raw write there is erased before it is sent. Every octet of the master frame is a mapped object, so the connection writes the frame it built back through those cells and the composer puts it on the wire. No new RT mechanism. The frame comes back raw, out of the image the cycle just captured. It has to: a 32-bit safe value spans two CRC sections, and the ESI maps its high half as an alignment gap with no object behind it. There is nothing to read it from — and reading a "safe position" object would hand out a number that no CRC, sequence number or watchdog had vouched for. Binding belongs to the control plane. Opening a connection reads the drive's PDO mapping over SDO, derives both SafeData lengths from that mapping rather than from configuration, checks every field, and allocates; the cycle allocates nothing. A connection whose process image was re-mapped reports bound: false and stops driving rather than write a Safety PDU into whatever now occupies those octets. The integration test runs a whole drive on a fake bus whose mapping is the shipping ESI entry for entry, alignment gaps included, against the ETG.5100 slave double. It also pins a requirement the first run taught us: a transport must not present the same frame twice. The master already ignores a repeat, but a slave handed one twice answers twice and the second answer breaks the CRC chain — the drive's own glue does that check, and so does the double. 995 tests pass. The endpoints were exercised against the running server. No hardware has driven this yet; the bench axis is next. Rationale: NEXTGEN.md, Session 2026-08-22.
The Safety page showed what the safety wire carries and nothing about why. The wire has room for validity and nothing else - three spare bits against twenty-nine causes - so the reasons, the channel disagreement and the filter lengths live on the object dictionary, and this is the view of them. The centrepiece is the headroom bar. Choosing a cross-check tolerance has been guesswork; this shows the band the tolerance defines, the span the channels have actually covered since the connection came up, and where they are right now. A span that nearly fills the band means the axis is about to nuisance-trip; a sliver means the tolerance is far looser than the machine needs. "From measured" turns that reading into the tolerance, doubled for margin, so the number comes from the machine rather than from a guess. Reasons are named rather than numbered, and the three that are not faults are styled as information: a single-encoder axis reports "no verification channel" for the whole of its life and it is not broken. Every cause seen since the connection came up appears as a chip beside the live one, because a discrepancy that trips for three milliseconds is invisible to a one-second poll. Requested and in-force filter lengths are shown side by side, with the difference called out, so a window the averaging ring could not honour reads as truncated instead of as ignored. The panel sits outside the connection block on purpose. These objects read over SDO, and the moment they are most wanted is when the connection will NOT open - a wrong safety address, a tolerance nobody set - which is exactly when there is no connection to hang them off. It falls back from readParameter to an SDO upload for the same reason: readParameter answers from the enumerated object dictionary, which is empty right after a firmware update.
…an run The drive firmware now declares Safe Stop 1, and ETG.6100 activation is inverted: a master that leaves bit 1 at 0 is REQUESTING SS1 on every cycle. With the standard's all-zero default parameters that is an immediate STO, so without this change the axis would simply sit in the safe state and look broken. SdpControl grows ss1Requested, defaulting to requested for the same reason stoRequested does. The consequence is stated on the field, because it is the trap: a caller that clears only stoRequested still requests SS1, and permitting motion means clearing both. 0x01 no longer means "torque permitted" - 0x03 does. /api/fsoe/:pos/sto now read-modify-writes the staged controlword instead of writing the whole octet from SdpControl's defaults. That was harmless while STO was the only function; with a second activation bit it would have silently re-requested SS1 every time someone released STO. FsoeConnectionState gains safetyControl() so an endpoint owning one bit can leave the others alone. /api/fsoe/:pos/ss1 is new. Its documentation says what the standard requires and what that means for a caller: releasing the request mid-stop does NOT abort the stop, because ch. 8.2.1.1 requires the function be finalized once activated. The endpoint can start a stop; it cannot cancel one. 996 tests pass. Two expectations moved from 0x01 to 0x03, which is the change itself rather than a workaround for it.
Configuration and troubleshooting for SS1 (ETG.6100.2 ch. 8.2), beside the safe
sensor panel and reading over plain SDO for the same reason: a parameter set is
usually being chosen when there is no safety connection to hang a panel off.
The five parameters are editable with the meaning of each on hover, and the
header says which mode is in force - "Time monitoring" versus "Time +
deceleration" - derived from whether a_SS1 is set, because that is the only thing
that selects it.
The half worth having is the right-hand column, which answers a question the
objects do not: what will a stop actually DO. It puts the deceleration in force
(0x2606:05) next to the present speed, works out how long that ramp needs to
reach the standstill window, and compares it against t_SS1. Two configurations
that look fine and are not get called out:
- a deceleration of 0, where an SS1 stop will not slow the axis at all and
will coast to the deadline before losing torque;
- a t_SS1 shorter than the ramp plus the dwell needs, where the standstill
window can never be reached in time, so every stop reports the Table 13
error while nothing is broken. That one is easy to ship and impossible to
diagnose from the safety statusword, which reads identically either way.
It also offers the specification's own sizing floor for a_SS1,
|n_max|/(t_SS1 - t_D_SS1), as a clickable value - a floor, labelled as one,
because the comparison carries no tolerance band.
The deceleration-limit bar greys itself out when the limit is not armed rather
than drawing a limit nobody is enforcing, which would invite exactly the wrong
conclusion.
Deliberately NO button to command SS1. Starting a stop is a safety master's job,
and ch. 8.2.1.1 makes it irreversible once begun - releasing the request does not
abort it. A one-click "stop the axis" in a diagnostics page would be a decision
nobody could take back.
A stop completes in a couple of hundred milliseconds and the finalizing STO is
visible for a SINGLE cycle, so this cannot be polled: measured SDO polls landed
~150 ms apart, and the only way I got a visible ramp on the bench was to slow the
deceleration to 300 rpm/s on purpose. So the capture lives on the FSoE cycle
thread, in FsoeConnection::step, where both directions of one cycle are in hand
and the input has already been authenticated by the master - CRC, sequence
number, watchdog. Recording anywhere else would either miss a direction or store
octets nobody vouched for.
The existing process_data_ring was the obvious candidate and is the wrong tool:
it has no trigger, its dump span is the whole ring, and the safe velocity is not
object-decodable from it at all because the interleaved FSoE CRCs split SafeData
and the ESI maps the second half as an alignment gap. It stays what it is.
Ss1Recorder (libs/node/ss1_trace.{h,cc}) is a separate, pure unit taking explicit
per-cycle fields rather than FsoeConnectionState - the connection must include it
to hold one, so depending on its state type would be circular, and a recorder
that takes only what it uses can be tested by feeding it synthetic cycles with no
connection, no device manager and no bus. Two slots and a generation counter
rather than a lock: the writer fills the slot the published generation does not
name, so a reader copying the published slot cannot be overwritten under it.
Four things that would each have been a silent wrong answer:
The edge detector is seeded TRUE. SafeOutputs start all-zero and ETG.6100
activation is inverted, so an unconfigured frame decodes as SS1 REQUESTED -
seeded false, every connection would record a phantom stop on its first cycle.
The trigger is also gated on Data/ProcessData, so a request arriving while the
drive already holds fail-safe data is not recorded as a stop.
The anchor is latched from the cycle AFTER the trigger. The controlword built
during cycle N is not on the wire until N+1, so the input arriving with the
trigger is the drive's answer to the previous frame and cannot reflect the
request. Anchoring on it shifts the whole limit line. Two tests pin this, and it
was a real bug on the first run - beginCapture ran inside the trigger cycle's own
observe() and latched immediately.
Losing the frame mid-stop ends the trace as Unbound rather than just ceasing to
produce samples, which would read back as an axis that held its speed.
A retrigger publishes the superseded run rather than discarding it - it is
evidence - and still gets a correct pre-roll, because the pre-roll ring is
written on every cycle whether capturing or not.
The endpoint returns only what was recorded; the SS1 parameters are read by the
console over SDO instead, so a plot poll costs no bus traffic and cannot stall on
an SDO timeout. GET /api/fsoe/{pos}/ss1-trace, positional rows with columns given
once - a trace is up to 4096 samples and naming fields per row would triple the
bytes for nothing.
MonitoringChart gains annotations and per-series dashes. Annotations are DATA,
not a plugin, because the rebuild is keyed on prop content and a plugin is a
function: handing one in would either rebuild the canvas every render - which the
component's own comment records as collapsing it to zero height and yanking the
page scroll - or never pick up a change.
PUT /api/fsoe/{pos}/ss1 was entirely absent from swagger.yml and the generated
client; it shipped undocumented in the previous commit. Both paths are now
specified and the client regenerated.
The plot's limit line is ported from sdp_ss1_update rather than approximated: a
staircase on the SAFETY cycle (0x2605:04, 1 ms on SIM but 250 us on AM2612 - the
bus period is not it), with the floor applied to the cumulative product, which is
what reproduces the firmware's carried remainder.
What it is FOR: margin. The objects can say a stop passed or tripped; they cannot
say by how much, because 0x2606:04 is a per-cycle value reading 0 outside an
armed mode-B cycle and the armed window is often under 100 ms, so an SDO poll
essentially never lands inside it. The panel headlines limit headroom, measured
stopping distance from the safe position channel, and reaction dead time - the
last two being exactly what a guard-distance calculation needs and neither
derivable from any parameter.
17 recorder tests, 1013 MM6 tests, console builds.
The recorded trace looked like a staircase because it WAS one: it stored a sample on every bus cycle, but FSoE is a ping-pong and each direction costs a cycle, so a new SafeInputs frame lands roughly every third one. Measured on the bench, the safe velocity changed every 12750 us while the trace sampled every 4001 - each value stored 3.2 times over. That was not just wasteful. measuredCyclePeriodUs reported the BUS period rather than the exchange period, so anything derived from it scaled wrong, and the plot implied a resolution the transport does not have. Cycles without a fresh frame now only accumulate time; the pending interval is added to the next recorded sample, so no elapsed time is lost and the reported period is the exchange period. An unbound cycle still records, because the end of a trace has to be truthful even when nothing arrived. The console panel now prefers the firmware's own published anchor (0x2606:06/:07) over the one the recorder inferred. They normally agree, and when they do not the firmware's is the one the limit was actually enforced against - drawing from a different anchor is indistinguishable from the monitor being broken, which is a mistake worth not making twice. The stop-trace panel also moves out of the connection block and under the SS1 parameters it is a picture of, rather than sitting above the safe sensor area where it made no sense. 18 recorder tests, 1014 MM6 tests, console builds.
Triggering SS1 with the axis already still produces a legitimate capture with nothing in it: there is no deceleration, every sample sits inside the standstill window, and the plot is measurement noise around zero. One such trigger was enough to overwrite the trace somebody was reading - and verifying that SS1 fires at all is the obvious first thing to try during commissioning, usually from standstill, so this was not a corner case. A completed capture whose anchor is below one rpm is now counted instead of published, and the count and its timestamp are reported. The event is not dropped silently: "I triggered SS1 and the plot did not change" deserves a better answer than nothing, and the panel shows a "+N from standstill" chip. The exception is the very first capture. With nothing published yet, something honest beats nothing at all, and the panel says what it is looking at and how to get a better one. A fixed one-rpm floor rather than the configured n_Zero_SS1, because the recorder runs on the cycle thread and has no access to the object dictionary. One rpm is far below any plausible standstill window and far above measurement noise, so the classification does not depend on tuning. Two of the retrigger tests needed their second run given a moving anchor - they jumped straight from the trigger to a stopped axis, so the new rule correctly declined to publish them. That is the rule working, not a regression. 20 recorder tests, 1016 MM6 tests, console builds, spec and client regenerated.
Driving a slave into BOOT reprograms its mailbox sync managers to the BOOT geometry, where the bootloader speaks FoE rather than CoE. transitionToState deliberately does not hold controlPlaneMutex_ across the whole transition, so a concurrent SDO transaction interleaves with that reprogramming - and the first SDO issued once the slave has actually reached BOOT takes the process down. Reproduced on an AM2612 axis, four times, always with the same last log line: an SDOread immediately after "reached state 0x03". The contrast pins it: BOOT with no SDO traffic survives a single SDO while resting in BOOT survives (clean abort) six non-BOOT transitions under continuous SDO survives BOOT under continuous SDO polling dies, 4/4 bootMailboxSlaves_ is the exact predicate: it is maintained under the same lock and set at the instant the sync managers are reprogrammed, not when the slave finishes reaching BOOT - so it covers the window from the first dangerous moment. Refusing there costs a caller an error string instead of the daemon. The console side is what made this reachable: both safety panels poll their parameters every second and fall back to a raw SDO upload when the object dictionary cannot answer - which is precisely the case during a firmware update. They now poll only while the device is in PRE-OP, SAFE-OP or OP.
…n the sections Two problems, one page. The controls could not express the state a bench needs. `Release STO` set bit 0 and nothing else, so from an all-zero controlword the axis stayed held: bit 1 was still clear, which is a request for SS1. There was no way to run an axis up and then stop it with SS1 - the one sequence this page exists to allow. Each stop function now has its own row and its own pair of buttons, and a footer names whichever of the four conditions for torque is not met rather than leaving a disabled button to be interpreted. Error acknowledge is there too, as bit 7, pulsed through the raw SafeOutputs endpoint because the drive acts on the rising edge. The page had also grown three different section treatments at once - a bordered section with a display-font heading, a bordered div with an uppercase span, and an unbordered block - and the connection's data command, reset and close sat inside the Safe Torque Off block, where they read as things that act on STO. They act on the link that carries it. One `Section` container now holds every panel, the lifecycle controls moved to the connection they belong to, and four group rules - link, control, measurements, functions - give a long page a shape somebody can navigate. The page itself drops from 493 lines to 144, the removed bulk becoming the panels it now composes. Verified against the C021 axis: with the controlword at 0x02 - SS1 released while STO is still requested, a state the old UI could not represent - the gating is Release STO enabled, Apply STO disabled, Request SS1 enabled, Release request disabled, and the footer narrowed to "STO is requested".
…afety page The pre-roll bug is the substantive one. beginCapture spaced the pre-roll by an average of the dt values collected DURING a capture - and reset that average at the end of every capture, so on the first stop there was nothing in it and it fell back to a nominal 1 ms. The samples were real; they were laid down at a twelfth of their true pitch, which reads as a burst of impossibly fine-grained history that abruptly coarsens the moment the stop begins. The interval is now measured on every cycle, bound or not, so it is already known when the first capture needs it. Measured on the C021 axis: pre-roll pitch 11954 us against a reported exchange period of 11953 us, where it used to be 1000 us. The pre-roll also gets a time budget rather than being drawn to the ring's full depth. At one exchange every ~12 ms, 256 cycles is over three seconds of history, which would leave a one-second stop squeezed into a quarter of the x axis. The ring stays large for a fast bus; 750 ms is what gets drawn. The rest is the page: - Both stop functions now present the same two controls in the same order - release left, activate right - in fixed columns, so the position of a button means the same thing on every row. The gating is symmetric too, on one rule: releasing needs a live process-data link, activating never does, because refusing to make an axis safer on account of the transport is the wrong way round. - The connection section says what ProcessData and FailSafeData actually are, which was a fair question to have to ask of two unexplained buttons. - Safe process values and Safe sensor were one subject in two sections: the tolerances in the second exist solely to judge the values in the first. Now one "Safe measurements" section, still two bodies, because the values need a connection and the configuration does not. - Chart legends: the swatch is a dash carrying the series' own dash pattern rather than a filled square, and the shaded regions - which uPlot cannot list, being annotations rather than series - get their own legend with filled squares. Shape says what kind of thing it is; colour says which one.
"Safe velocity not believable" was an interpretation. The region is exactly the cycles where the drive published its safe velocity with the validity bit clear, so it now says that: "velocity validity flag false". Naming the flag raises the obvious next question, and the answer already existed - just in another section. The Safety PDU carries the validity bit and NOT its cause (ETG.6100.2 ch. 5.4 has a bit, not a reason); the cause lives in the non-safe diagnosis objects, which is where ETG.6100 puts it. So a trace containing such a span now reports how long the flag was false, the velocity channel's reason now (0x2602:07) and every reason latched since the connection came up (0x2602:08) - stated as connection-scoped rather than stop-scoped, because the recorder physically cannot attribute a cause per cycle. The SafeSensorInvalidReason vocabulary moves to safeSensorReasons.tsx: two panels now decode the same enum from the same objects for the same fact from opposite directions - one reporting a channel's health, the other explaining a span on a plot. Also gates the stop-trace panel's SDO poll on an active mailbox. The other two safety panels got that in cc1aec3 and this one was missed, which left it able to poll an SDO into a BOOT transition and take the daemon down with it. Verified on the C021 axis that 0x2602:07/:08/:09 read (0, 1048617, 1) and that the mask decodes to the chips the safe-sensor panel already shows: sensor fault, position discrepancy, verification channel down. The amber region itself is NOT exercised on hardware - this axis publishes a valid velocity throughout a stop, and forcing otherwise means inducing a cross-check failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an FSoE master to Motion Master, the API to drive it, and a Console page to operate a safe drive from a bench.
Important
This is a protocol master, not a safety master. It implements ETG.5100; it does not implement the integrity of the machine running it, and it cannot — that needs certified hardware. What makes it useful anyway is that the drive stays safe on its own: it authenticates every frame and drops its outputs to the safe state when the frames stop, whatever this master does. It is for commissioning, diagnosis, and moving a safe axis on a bench.
What is here
libs/etg— the FSoE master: session/connection/parameter/data handshake, CRC and sequence validation, watchdog, fault reporting.libs/node—FsoeConnection, stepped once per bus cycle from the game loop. Connections are appended and never removed, so the cycle thread walks a fixed pointer array with no lock.apps/motion_master—/api/fsoeand friends: open/close/reset, data command, STO, SS1, raw SafeOutputs, and the SS1 stop trace.web/apps/console— a Safety page: connection, safety control, safe measurements, Safe Stop 1 parameters, and a plot of the last stop.Safe Stop 1
The controlword's SS1 bit is driven independently of STO. That matters more than it sounds: both bits are inverted on the wire, so an all-zero controlword requests every stop function at once, and a caller that releases only STO leaves SS1 requested and the axis held. The page gives each function its own pair of controls and names whichever of the four conditions for torque is unmet.
The stop recorder captures each stop on the cycle thread — pre-roll, the stop itself, and a post-roll past torque removal — and the panel plots it against the parameters that judged it: the
a_SS1deceleration limit, thet_SS1deadline, then_Zero_SS1standstill window and thet_D_SS1monitoring delay. It reports the numbers no object exposes: limit headroom, reaction dead time, stopping distance.One fix worth reading on its own
cc1aec3refuses CoE to a slave holding BOOT mailbox sync managers.transitionToStatedeliberately does not holdcontrolPlaneMutex_across a transition, so an SDO issued while entering BOOT interleaves with the mailbox reprogramming and kills the daemon. Reproduced on hardware; the contrast pins it:Any panel that polls parameters is enough to trigger it, which is why the safety panels now gate their polling on an active mailbox.
Validation
Exercised end to end on an NG-Platform C021 (AM2612) axis: handshake to
Data, safe process values, STO in both directions, and Safe Stop 1 stopping a spinning axis — 29.8 rpm anchor, 19.9 rpm/s measured deceleration against a 15 rpm/s limit, standstill at 1.500 s, torque off at 1.596 s against a 3 s deadline. Trace resolution matches the FSoE exchange period (11.95 ms), not the bus cycle.Draft, because