Repository navigation
Release 5.3.9a2 - #402
Closed
github-actions[bot] wants to merge 420 commits into
Closed
Release 5.3.9a2#402github-actions[bot] wants to merge 420 commits into
github-actions[bot] wants to merge 420 commits into
Conversation
Every satellite floods on its own schedule, and answering a flood costs one send per connected peer, so unthrottled the node's work per round grows with the square of the mesh size — the point where a large site stops completing rounds at all. ping_flood_interval (default 30s) caps the mesh-wide fan-out. A peer that pings inside the window is still answered, directly and immediately, so its own map is never wrong; only the mesh-wide fan-out waits. Both flood caches are now sized from the client count instead of a fixed 1000. Every connected peer can start a flood of its own, so a fixed cap thrashes exactly on the meshes that can least afford the extra floods, and each eviction of a live flood_id buys a duplicate flood. The throttle runs BEFORE the _seen_flood_ids claim, and that order is load-bearing. The claim means "this node owes the master an answer for this flood, and the other half must not send a second one". Claiming it and then returning from the throttle would leave the answer owed by nobody: the throttled half never sends it, and the slave half stays suppressed because the claim is shared (bind_upstream). The master would silently stop seeing this node — no error, no dropped message, just a node that falls out of the hive map. tests/test_ping_flood_throttle_ upstream.py pins that ordering; moving the claim back above the throttle fails it and nothing else. _last_ping_flood and ping_flood_interval are declared as dataclass fields with plain defaults, as #219 asked whoever landed this to do, so fixtures built with object.__new__ do not raise AttributeError on the ping path. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* perf: cut per-peer cost in message fan-out Three measured defects in HiveMindClientConnection.send() and its six fan-out call sites: 1. Per-peer LOG.debug() calls used ovos_utils.log.LOG, whose debug() classmethod runs inspect.stack() before checking the log level, so a SUPPRESSED debug call cost 3+ orders of magnitude more than it should. Measured ~570ms of ~816ms in a 1000-peer fan-out. Switched the hot-path calls in send() to a module-level stdlib logger (level check first, ~0.2us when suppressed) with lazy %s formatting so the strings aren't built when suppressed either. 2. None of the six fan-out loops guarded the per-peer send(). One peer raising (e.g. a binarize=True client whose metadata exceeds the WIRE-1 binary cap, or a peer in a broken crypto state) escaped the loop and silenced delivery to every peer ordered after it. Each loop now wraps its per-peer send in try/except and logs+continues instead of aborting the whole broadcast. 3. serialize() was called once per peer, but the plaintext does not depend on the peer (only encryption does, and that stays per-peer). Measured 9.8-41us per call depending on route depth, ~10-40ms of pure waste per 1000-peer fan-out. send() now takes an optional pre-serialized plaintext, computed once per fan-out and passed to every peer. Added a regression test that a raising peer no longer blocks delivery to the rest of the mesh, and a test that serialize() runs once per fan-out rather than once per peer. * fix: avoid autospec+wraps mock gotcha in fanout serialize-once test patch.object(HiveMessage, "serialize", wraps=..., autospec=True) returns a MagicMock instead of forwarding the real str return value on Python 3.10 (fixed upstream in 3.11), making the spied call look like it produced a non-str plaintext. HiveMessage.serialize() itself is version-independent (always json.dumps -> str). Replace the spy with a plain forwarding wrapper that counts calls without relying on autospec's signature-checking wrapper.
* feat: OVOS transformer pipelines on the text/bus path hivemind-core now runs OVOS transformer chains for text clients: - utterance + metadata transformers rewrite recognizer_loop:utterance messages after policy admission and before injection into the agent bus (handle_inject_agent_msg) - utterance transformers also gate the QUERY/CASCADE local-answer path, and dialog transformers rewrite each answer chunk before it is sent back to clients - an OVOS-TRANSFORM §8.1 cancellation terminates the lifecycle: the message never reaches the agent and the client receives the §8.2 terminal events (ovos.utterance.cancelled -> ovos.utterance.handled) Loading is config-gated and opt-in via new utterance_transformers / metadata_transformers / dialog_transformers blocks in server.json; empty chains are no-ops. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: transformer pipelines on the text/bus path Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * build: ovos-plugin-manager is a runtime dependency protocol.py imports the transformer runner services unconditionally, so the package must be a core dependency, not only part of the integration extra. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: use the cancel plugin's entry-point name in config examples Transformer config keys are entry-point names; the cancel plugin registers as ovos-utterance-cancel-plugin, not its package name. Verified against a live hub: the package-name key silently loads nothing. * test: give the bypass-built provenance node empty transformer services _answer_query_locally now runs the utterance and dialog pipelines. __post_init__ builds them for a real node, but this fixture uses object.__new__, so the fields kept their None default and the query path raised AttributeError on .plugins. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The type is removed from the protocol. hivemind-core never had a dispatch branch for it, so this is a documentation-only change. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
There is no listener component: there is only hivemind-core. The name was shorthand for hivemind-core with the audio binary protocol plugin enabled. Use hivemind-core, or name the plugin, where the prose called a listener a thing you run. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix: keep the node up when the agent backend or a listener fails An unreachable agent backend used to abort startup with a traceback before any listener was bound: on a slow board the OVOS messagebus is simply a few seconds behind, and the whole hive went offline over it. Retry building the agent protocol instead, the way the upstream link already degrades. Listener threads were fire-and-forget. A protocol whose run() raised (port in use, unreadable certificate) died in silence while the service kept reporting itself ready with no transport at all. Watch the threads: one dead transport is survivable, all of them dead is an error. * fix: mark the node ready before the transports can fail set_ready and set_error both assign ProcessStatus.state, so the last writer wins. A transport that fails the instant it starts - port already bound, a certificate that will not read - reached set_error while the main thread was still on its way to set_ready, and the node settled on READY with nothing listening: exactly the state this code exists to prevent. Set ready first, so any later failure is the state that sticks. The test now asserts the final state and the order of the transitions, not merely that set_error was called - it passed either way before.
… purge on disconnect (#211) - lock the check-then-act on _pending_cascades: _route_query_response runs on both the tornado thread and the upstream slave thread, and the racy version could construct two collectors for one query_id, silently discarding whichever lost - CascadeCollector.add_response used the same racy get-then-set shape for _by_responder; switch to setdefault so two threads can never split one responder's chunks across two entries - drop a client's TOFU pubkey pin on disconnect once its api key no longer exists in the db (revoked/replaced), never on a bare disconnect that still leaves the key live for reconnects - purge a disconnected originator's pending cascade collectors immediately instead of waiting for cap-churn eviction: once the originator is gone, _route_query_response's own originator_peer-in-clients guard means no future message can ever touch them again
…ts (#219) Fixture code across ~10 test files builds the protocol with object.__new__(HiveMindListenerProtocol), skipping __init__/__post_init__, then hand-sets a handful of attributes. Every attribute added to __post_init__ since has broken those fixtures with AttributeError, four times in one day (last_seen_update_interval, _identity_rsa_key, ping_flood_interval/_last_ping_flood, _seen_flood_ids). last_seen_update_interval and _identity_rsa_key become real dataclass fields with plain (non-factory) defaults, which dataclasses expose as class attributes -- a bypass-built instance now sees them for free. __post_init__ still overwrites last_seen_update_interval from config unconditionally, so operator config keeps winning. _seen_flood_ids can't use a plain class default since FloodIdCache is mutable and must not be shared between nodes. It's backed by a private _flood_id_cache field (default None, safe to share since it's immutable) and exposed as a property that lazily creates one FloodIdCache per instance on first access, with a setter so fixtures and bind_upstream can still assign a specific instance directly. Adds tests/test_bypassed_construction_defaults.py as the regression guard: it fails the moment someone adds a new __post_init__ attribute without a default, pinpointing it instead of breaking unrelated tests. ping_flood_interval/_last_ping_flood don't exist on this branch yet -- they're presumably introduced by #208/#216. Whoever lands those needs the same treatment for those two attributes.
WIRE-1 §4.3 gives INTERCOM no 5-bit message-type code, so it must travel as a text frame even when the connection negotiated binary framing. The server binarized whatever it sent as soon as binarize was set, and get_bitstring raises for a type with no code, so a server could not send an INTERCOM to a binarize client at all. Apply the gate the client has always had (BINARY_ENCODABLE_TYPES).
directory: "/requirements" points at a directory this repo does not have, so Dependabot has errored on every weekly run. It has never opened a single PR here; Renovate (renovate.json) does the dependency updates and does open them. The config was pure red noise, and it masked real dev-branch failures. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
) `handle_inject_agent_msg` wrote `destination` on the way IN: `"audio"` for a speak, `"skills"` otherwise, and it kept whatever a client had sent when the key was present. `_admit_for_query` wrote `"skills"` the same way. Both writes date from 5facb2c, 2024-04. The project's rule: the node owns both keys and a peer controls neither. `source` is stamped inbound with the peer id the node mints; `destination` is read-only to the node and becomes that same peer on the way out, which is what the node routes on. By construction a peer cannot address anything: it sends a request and receives the answer, and it does not know another peer exists. BRIDGE-1 §3.1 makes the inbound `source` the server's to stamp and §3.2 makes the outbound `destination` the server's to route on. No clause authorizes the server to write `destination` inbound with a service label, and none authorizes a peer to write it at all. The node now stamps a label of its own: the entry-point name the agent plugin was loaded under, read from `plugin_id` (hivemind-plugin-manager), and the generic `hivemind` when no name was recorded. The read is `getattr` with a fallback and a `str()` coercion, so this needs no floor bump and a plugin_id that is not a string cannot reach the context. The fallback is deliberately not the node id, which is the node's public key: this field returns to every peer as the `source` of its replies, so stamping it there would publish the key to every peer and cut against the topology hiding BRIDGE-1 §3.3 allows. The write sits beside the existing `peer`/`source` re-stamp, AFTER the transformer chain on the injection path and AFTER the policy chain on both. A transformer replaces `message.context` wholesale and POLICY-1 lets a policy plugin mutate the message, so a value written before either is not the value the agent bus sees. Written earlier, a metadata transformer can put a peer id back in the key; measured by two lanes independently. `tests/test_inject_speak_destination.py` is deleted. It came from #357 and its three cells pin the removed behaviour, one of them asserting that a CLIENT's `destination` is KEPT. #357's OVOS-MSG-1 §3.3 single-string requirement is carried forward in a cell of its own. Same venv, both arms: dev 628 passed 3 skipped, head 638 passed 3 skipped. Node ids 630 and 640, 13 added and 3 removed, the 3 named. Five mutation controls: dev's behaviour restored reds 8; the injection placement moved back before the chain reds exactly 1; the stamp emitting a list reds 8; the QUERY stamp moved back above the policy chain reds exactly 1; the coercion removed reds exactly 1. Co-authored-by: JarbasAi <jarbasai@mailfence.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
run 36486243619
…ork_protocol (#386) A server.json that existed but had no network_protocol key skipped the first-run transport filter. The key was backfilled from the unfiltered defaults, which name hivemind-http-plugin. When that plugin was not installed, listen logged an ERROR saying the transport "is enabled in server.json", which was false. get_server_config now writes the installed transports into a file whose network_protocol is missing, null or not a block, once, as a first run does. A non-block value also logs a WARNING. An explicit operator block is unchanged. Tests: 4 new in tests/test_missing_network_transport.py (3 fail on the unfixed code, 1 guards the explicit block). The null-fallback test in tests/test_malformed_hello_and_config.py now declares both transports installed. Unit suite: 581 passed, 1 skipped. e2e: 61 passed, 2 skipped. Co-authored-by: JarbasAi <jarbasai@mailfence.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
run 39485672996
The current voice satellite sends ovos.utterance.handle (OVOS-PIPELINE-1 section 9.1). The hub special-cased only the legacy recognizer_loop:utterance name, so a spec-topic utterance skipped the utterance and metadata transformer chains (and with them any cancellation policy) and was refused by QUERY/CASCADE extraction. The add-client hint told the admin to allow only the legacy name, so a satellite configured from it was denied acl_disallowed_type on every spoken utterance. UTTERANCE_TOPICS now names both topics and is used by the inject path, the QUERY/CASCADE path and the transformer helper. The message keeps the topic it arrived with when it reaches the agent bus. The add-client hint grants both names in one allow-msg call, and the docs name both. The allowed-types gate is unchanged: it admits the exact topic a client sends, per HIVEMIND-POLICY-1 section 4. Fail-before: tests/test_utterance_topics.py is 4 failed / 4 passed on dev (every ovos.utterance.handle case and the hint) and 8 passed after. Co-authored-by: JarbasAi <jarbasai@mailfence.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
run 39507889210
handle_query_message and handle_cascade_message iterated the agent's answer_query generator on the IOLoop thread. The OVOS agent plugin's generator blocks until the skill is done, up to its 10 s idle bound when a skill waits on speech (speak_dialog(wait=True), get_response). Every other client's frames and the query's own chunks waited for that. Both handlers now hand the answer to a bounded pool of daemon worker threads (8 workers, 64 requests running or queued; a request over the limit counts as declined). shutdown() drops queued answers and refuses new ones, which count as declined, so an answer stream that never ends cannot hold the process open. The worker sends each chunk as it is produced, in order, followed by hive.query.complete. For a QUERY the escalation (upstream relay or the no_answer error) is a continuation the worker runs when the backend declines. A CASCADE fans out to the other peers without waiting for the local answer. An error in a worker closes the connection with 1011, as an error in the handler did. Unit tests that stub the backend run the pool inline through a conftest fixture. The relay-chain e2e test waits for the backend's decline now that the sender no longer blocks on it. Live run (real hub, ovos-core, agent plugin, hello-world skill): a second client's BUS utterance during a QUERY answered in 0.047 s instead of 8.37 s, and the QUERY chunk arrived after 0.1 s instead of 10 s. tests/test_query_off_loop.py: 4 of 6 fail on dev, 6 of 6 pass. Co-authored-by: JarbasAi <jarbasai@mailfence.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
run 39508003007
…chain denied (#390) handle_query_message and handle_cascade_message ran the admission chain inside the answering worker and read a False return as "declined". A denial and a decline looked the same, so a denied QUERY was escalated to the upstream and a denied CASCADE was fanned out to peers and the upstream. The client received hive.policy.denied and then the answer of a node that never applied its restrictions. can_escalate defaults to true, so any authenticated client of a node with an upstream could do this. The utterance is now admitted on the loop, before the worker pool, in _admit_query_request, for both utterance topics. A denial drops the request: it is not answered, escalated, forwarded or fanned out (HIVEMIND-MSG-1 5.1), and the client has received the hive.policy.denied of HIVEMIND-POLICY-1 6 and no other answer. Admitting ahead of the pool also closes a bypass in which a full pool counted a request as declined and escalated it unreviewed. _answer_query_locally takes the admitted utterance. A request that holds no utterance is only forwarded, and a decline, where the agent has no answer, escalates as before. Fail-before: the e2e tests test_denied_query_is_not_answered_upstream and test_denied_cascade_runs_on_no_peer, run for each utterance topic, fail at dev (the upstream agent received the denied utterance), as do the two unit tests in test_query_off_loop.py. The relay e2e fixtures granted their relay no utterance permission upstream and passed only because the denied request escalated and came back as a no_answer error; they grant it and assert the answer text. Co-authored-by: JarbasAi <jarbasai@mailfence.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
run 39513902363
) The test looked for the skill in the entry-point group ovos.plugin.skill. The hello-world skill registers under opm.skill, so the check never matched and the test was always skipped. The lookup now uses opm.skill. The test passes against the real OVOS stack, and it now asserts the spoken text: it must be one of the lines in the skill's own how_are_you.dialog, not only that some answer arrived. Before: 1 skipped (needs ovos-skill-hello-world). After: 1 passed. Co-authored-by: JarbasAi <jarbasai@mailfence.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
) A QUERY or CASCADE waiting for a worker blocked on the pool's semaphore. After shutdown() the pool is closed, but when the slot holders never finish (a stuck answer stream) the waiters stayed blocked for good. Each shutdown and reuse cycle with stuck streams leaked up to 56 daemon threads. shutdown() now releases one extra slot. The first waiter wakes, sees the closed flag, drops its job and releases on exit, which wakes the next one. The wait keeps the semaphore's arrival order, so queued queries still start first come, first served. Fail-before: the shutdown test starts 10 queries on a 2-worker pool with stuck streams, calls shutdown(), and counts live pool threads. On dev 10 threads stay alive; with the fix 2 remain (the holders). The arrival order test fails on a polling wait and passes here. Co-authored-by: JarbasAi <jarbasai@mailfence.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
run 39583524181
…#394) A request that arrived without metadata.query_id was given an id by the first hub, but the envelope forwarded upstream was rebuilt from the original metadata. The upstream hub minted a different id, and the relay dropped its answer as unbacked (no request seen for that id). The first hub now writes the minted query_id and originator_peer into the metadata it forwards (HIVEMIND-MSG-1 section 5.2). An empty value counts as absent. Fail-before: the new two-hub tests fail on dev. The upstream binds an id different from the relay's, and a QUERY without query_id gets no answer. They pass with the fix. Co-authored-by: JarbasAi <jarbasai@mailfence.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
run 39583546089
run 39583569290
…rame (#398) The relay-chain tests read the satellite's answer from the return value of Recorder.wait_for, which is the newest matching record. When the stream finishes before the woken waiter reads the recorder, that record is the hive.query.complete frame, and the test fails with KeyError: 'utterance'. The tests now wait until a received frame carries an utterance and assert the spoken text, so the read no longer depends on frame timing. The async test asserts its text too, where it checked only that a frame existed. Under a 4-process CPU load, 48 runs of the two relay tests passed with 0 failures. On dev, 1 failure in 24 runs under the same load. Co-authored-by: JarbasAi <jarbasai@mailfence.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
run 39586382425
Contributor
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Human review requested! This supersedes any earlier open release proposal for this repository.