Repository navigation
feat(audio): remember speakers from diarization - #12414
Merged
Merged
Conversation
Add the versioned profile schema for explicit speaker enrollment. Validate compatibility against separately supplied loaded-encoder metadata. Reject unusable speakers, invalid vectors, and inconsistent clean spans. This slice does not change HTTP routes, backend integration, or the UI. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Export opt-in speaker profiles and trusted encoder metadata. Replay registrations by ID so duplicate display names keep independent vectors. Use one profile-capable diarization for slots, names, and clean spans. Assign timestamped ASR words to those slots without a second diarization. Preserve legacy opt-out and no-ASR behavior, and propagate failures. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Gate profile exports with voice-recognition permission and validate registration against metadata from the loaded encoder. Preserve audio enrollment and independent registrations with duplicate display names. Exclude diarization and registration exchanges before API trace capture so persisted traces cannot retain profile vectors or JSON audio. Defer candidate dimensions to trusted loaded metadata. Sort candidates by registration ID so incompatible profiles cannot suppress legacy voices through registry iteration order. Keep portable identity checks closed when trusted metadata is unavailable. Test persisted traces, explicit slot zero, and selection through offline and live transport. Document privacy and the ephemeral registry lifecycle. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a Studio page for diarization and opt-in speaker profiles. Preview clean intervals from the original recording before explicit registration. Join profiles by raw speaker labels, preserve duplicate names, and relabel turns only after a successful save. Discard stale results when the model or recording changes. Share registration metadata with voice management without storing vectors or recordings from this flow. Document permissions and the global, ephemeral registry. Cover enrollment, permissions, previews, and asynchronous races with mocked Playwright tests. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Replace the stale enrollment limitation with the current HTTP workflow. Distinguish native transport from explicit registration and link its docs. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Use the merged commit from mudler/parakeet.cpp#80. Its tree matches the previously accepted native pin. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Connect the existing gallery modes to the speaker enrollment workflow. Show installation, private profile export, explicit raw-slot registration, and later recognition without another export. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Put the diarization walkthrough on the LocalAI website in the feature PR. Cover the three gallery modes, explicit enrollment, and privacy limits. Link setup instructions and keep availability conditional on feature support. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Explain what users can do with recordings before the setup steps. Replace the technical walkthrough with a short Studio guide and link readers to the existing reference for model names and developer use. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Present speaker recognition through everyday uses and a short UI flow. Keep technical reference details in the existing documentation. Assisted-by: OpenAI:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mudler
force-pushed
the
feat/diarization-profiles
branch
from
October 2, 2026 00:22
2d333cc to
9a3aa17
Compare
Avoid copying protobuf message state when extending backend status, check the multipart reader close result, and document the focused testing.T lint exemptions. Assisted-by: nib:gpt-5.6-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Remember speakers from arbitrary multi-speaker recordings without preparing
separate clean enrollment clips. Diarization exports one profile per usable
speaker, and users preview clean speech before explicitly naming and saving it.
Follow-up to #12382 and mudler/parakeet.cpp#79; no issue-closing claim.
Dependency: mudler/parakeet.cpp#80 is merged.
The backend pins the merged native commit
bee7c14dfcc23613df58176c59a40459e7b47095.include_speaker_profilesto diarization and transport theversioned profiles through the parakeet backend. Normal responses stay
unchanged. Unsupported profile export fails explicitly rather than
silently falling back.
/v1/voice/registerendpoint for explicit profileenrollment while preserving audio enrollment. Validate profiles against
identity and dimension supplied by the trusted loaded encoder, not the
caller. Exact GGUF bytes determine encoder identity.
assign timestamped ASR words without a second diarization. Associate
profiles by raw speaker slot, including zero and sparse slots, never by
display order or name.
“Name and remember” action. Relabel only after successful enrollment,
reject stale results after recording/model changes, and keep independent
registration IDs even when display names match.
existing diarization and model-access checks. Exclude diarization and
registration exchanges before API trace body capture, including the
diarization alias and JSON audio. This flow does not persist vectors or
recordings in browser storage.
Notes for Reviewers
Verification
The recorded focused Go suites pass. With protobuf tooling installed, run
from the repository root:
make protogen-go GOMAXPROCS=2 go test -count=1 -p 2 \ ./pkg/grpc/proto \ ./core/schema \ ./core/services/voicerecognition \ ./core/backend \ ./backend/go/parakeet-cpp \ ./core/http/endpoints/localai \ ./core/http/auth \ ./core/http/middlewareCoverage includes profile validation, trusted encoder compatibility,
permissions, raw-slot mapping, duplicate-name registrations, and trace
privacy. These tests use synthetic data or mocked inference.
The UI production build passes under Node 22. The focused Playwright
suite passes 10 tests, with mocked API responses, not real inference.
To reproduce with Chromium installed, use two terminals:
Swagger JSON, YAML, and
docs.gopaths/definitions were checked for semanticconsistency. Generator execution was blocked by a missing
go.sumdependency; this is not a successful regeneration claim.
Limitations and compatibility
per-user durable storage. Registrations disappear on restart and are not
synchronized across independent frontends.
or consent. Enrollment remains explicit; exporting never registers anyone.
visual inspection remain unverified. No full
make buildor full testsuite success is claimed.
skips. See the dependency PR for its CMake/CTest reproduction commands.
the entire UI passes lint. The production build retains chunk-size warnings.
requires the pinned native symbols and a configured speaker encoder.
Signed commits
The commits are not cryptographically signed; the checkbox remains unchecked.