diff --git a/docs/diarization.md b/docs/diarization.md index dec4fd6..674398d 100644 --- a/docs/diarization.md +++ b/docs/diarization.md @@ -243,10 +243,12 @@ protect a memory-mapped model from later writes either. An application must gate profile export and enrollment with its recognition permissions. Only after explicit user confirmation should it validate version, unavailable status, finite/nonzero vector, trusted identity and dimension, then -call the existing raw-vector registration path. The native registry validates -vectors/dimensions but does **not** store model identity: callers must enforce -identity and avoid mixing same-dimension encoders. Profiles are not signed and -are not proof of identity. Treat exported voice vectors as sensitive data. +call the registration path. Pass the trusted encoder family and identity with +the vector (`parakeet_capi_speaker_registry_add_embedding_fp`, see "Encoder +fingerprint" below), so the registry records which encoder made it and every +named call checks it. The plain `add_embedding` still works and records +nothing. Profiles are not signed and are not proof of identity. Treat exported +voice vectors as sensitive data. The native registry is **name-keyed and aggregating**: `parakeet_capi_speaker_registry_add_embedding` calls `SpeakerRegistry::enroll`, @@ -263,3 +265,106 @@ application. This is a downstream application responsibility, not functionality implemented by this backend's profile export. Relabel only after registration succeeds. Profile export adds no persistence or automatic enrollment; it does not change an application's global, in-memory registry lifecycle. + +### Encoder fingerprint + +Equal embedding sizes do not mean the same embedding space: ECAPA and CAM++ +both give 192 values, and a registry enrolled with one names the wrong people +when the other is used. A registry therefore records which encoder made its +voices, and the encoder in use is checked against it before any name is +assigned. Two strings make the fingerprint: + +- **Family**: `voicedetect:::`, + read from the encoder GGUF metadata, for example + `voicedetect:ecapa_tdnn:speechbrain/spkrec-ecapa-voxceleb:192`. It names the + embedding space. `parakeet_capi_speaker_encoder_family` returns it. +- **Weights**: `sha256:<64 hex>` of the exact bytes of the encoder GGUF file, + the same string as `parakeet_capi_speaker_identity`. A GGUF has no recorded + source hash, so the file bytes are the definition. Another quantization of the + same encoder has the same family and another weights hash. (A `voice` + component of a bundle reports `sha256:` plus the `source_sha256` of its header, + the hash of the single-model GGUF it came from, so the two agree when the + component is an unchanged copy of that file.) + +What the check does, the same everywhere (`parakeet-cli scene`, +`parakeet_capi_speaker_identify_pcm_json`, `parakeet_capi_diarize_named_pcm_json`, +`parakeet_capi_diarize_profiles_pcm_json`, +`parakeet_capi_transcribe_and_diarize_named_json`, +`parakeet_capi_scene_stream_begin_speaker`), with the same message text: + +| Registry vs encoder | Result | +|---|---| +| Other embedding size | Error: `registry holds N-value embeddings, this model produces M`. | +| Other family | Error naming both families. No name is assigned. | +| Same family, other weights | Warning only: logged to stderr, kept in `parakeet_capi_speaker_last_warning`. Names are assigned. | +| No fingerprint (a version 1 file, or embeddings added without one) | Accepted with a warning: the encoder is unverified. | +| No fingerprint and strict mode (`parakeet_capi_speaker_registry_set_strict`, `parakeet-cli scene --strict-registry`) | Error. | +| Empty registry | Nothing to check. | + +An error is reported like any other failure of that call: NULL (or nonzero) +with the message on the speaker context, and a non-zero exit in the CLI. + +Enrolment records the fingerprint of the encoder that computed the voice: +`parakeet-cli enroll`, `parakeet_capi_speaker_enroll` and +`SpeakerIdentifier` enrolment do it themselves. An empty registry takes the +fingerprint of its first voice. Enrolling with another family is refused. A +registry that has voices but no fingerprint refuses a fingerprinted voice, and a +fingerprinted registry refuses a voice with none: it is never stamped +silently, because nothing can verify what made the old voices. A caller that +builds registries from stored embeddings passes the family and identity it +stored with them to `parakeet_capi_speaker_registry_add_embedding_fp`. + +To stamp a registry that has no fingerprint, say which encoder made it: + +``` +parakeet-cli registry reg.bin # show the file +parakeet-cli registry reg.bin --restamp --encoder speaker.gguf # stamp a version 1 file +``` + +`registry` prints the format version, the embedding size, the family, the +weights hash and the speaker names. `--restamp` only works on a registry with no +fingerprint, checks the embedding size against the encoder, and trusts you for +the rest: it cannot tell ECAPA from CAM++ when both give 192 values. + +#### Registry file format + +Little-endian. `PKSR` magic, then: + +``` +version 1 (no fingerprint; also what a registry without one is saved as) + offset size field + 0 4 "PKSR" + 4 4 u32 version = 1 + 8 4 i32 dim + 12 4 u32 n, the number of speakers + 16 ... n speaker records + +version 2 (with a fingerprint) + 0 4 "PKSR" + 4 4 u32 version = 2 + 8 4 i32 dim + 12 4 u32 n + 16 4 u32 family length (at most 4096) + 20 ... family bytes (UTF-8, not terminated) + ... 4 u32 weights length (at most 4096) + ... ... weights bytes + ... ... n speaker records + +speaker record (both versions) + 4 u32 name length (1 to 4096) + ... name bytes + 4 i32 count of enrolled clips (at least 1) + 4*dim f32 sum of the L2-normalized embeddings +``` + +A version 2 file has at least one non-empty fingerprint string. Readers refuse +an unknown version, a truncated file, trailing bytes and implausible lengths. A +reader from before this change only knows version 1, so it refuses a version 2 +file with "unsupported version" instead of reading it wrong. A version 1 file +loads unchanged. + +The C-API additions are `parakeet_capi_speaker_registry_add_embedding_fp`, +`parakeet_capi_speaker_registry_encoder_family`, `..._encoder_weights`, +`parakeet_capi_speaker_registry_set_strict`, `parakeet_capi_speaker_encoder_family` +and `parakeet_capi_speaker_last_warning`. They are additive; the ABI version +stays 10 and no signature changed. diff --git a/docs/speaker.md b/docs/speaker.md index b821509..f3c009a 100644 --- a/docs/speaker.md +++ b/docs/speaker.md @@ -59,9 +59,14 @@ well. Sizes are for the f32 files. ERes2Net has not been run through any of the tests here. A registry belongs to the encoder that made it. The embedding sizes differ, and -even two encoders with the same size do not share a space, so enroll again if -you switch models. `scene` checks the size and stops if it does not match; it -cannot tell two encoders of the same size apart. +even two encoders with the same size do not share a space (ECAPA and CAM++ both +give 192 values), so enroll again if you switch models. The registry records +the encoder's family and weights hash, and `scene` and the C-API check them +before they assign a name: another family stops with an error that names both, +another quantization of the same family only warns, and a registry from before +this check (no fingerprint) warns, or stops with `--strict-registry`. See +"Encoder fingerprint" in [diarization.md](diarization.md) for the rules and the +file format. `parakeet-cli registry ` shows what a registry records. The speaker-model weights have their own licences (WeSpeaker, 3D-Speaker and SpeechBrain each publish theirs). voice-detect.cpp's own licence does @@ -104,7 +109,7 @@ the margin. Names are compared exactly, so near-duplicate names (`Ada` and ``` parakeet-cli scene --model --diar \ --speakers --registry [--speaker-threshold F] \ - --input + [--strict-registry] --input ``` `--speakers` needs `--diar` and `--registry`. Real output on the same fixture @@ -179,7 +184,10 @@ speaker GGUF loads through `parakeet_capi_load` into a context of kind parakeet_capi_speaker_dim # embedding size, -1 if not a speaker ctx parakeet_capi_speaker_registry_new / _free / _size / _last_error parakeet_capi_speaker_enroll # embed PCM and add it under a name -parakeet_capi_speaker_registry_save / _load # binary file +parakeet_capi_speaker_registry_save / _load # binary file (version 2 holds the encoder fingerprint) +parakeet_capi_speaker_registry_add_embedding_fp # add_embedding plus the encoder family and weights +parakeet_capi_speaker_registry_encoder_family / _encoder_weights / _set_strict +parakeet_capi_speaker_encoder_family / _last_warning parakeet_capi_speaker_identify_pcm_json # {"name":"alice","score":0.71} parakeet_capi_scene_stream_begin_speaker # scene stream with a speaker ctx + registry parakeet_capi_transcribe_and_diarize_named_json # offline speaker-attributed ASR with names diff --git a/examples/cli/main.cpp b/examples/cli/main.cpp index 95ef737..7ebc497 100644 --- a/examples/cli/main.cpp +++ b/examples/cli/main.cpp @@ -1732,7 +1732,9 @@ static std::string registry_read_error(const std::string& path, int e) { static const char* kEnrollUsage = "usage: parakeet-cli enroll --model [--component NAME] --name " - "--input [--input ...] --registry \n"; + "--input [--input ...] --registry \n" + " The registry records the encoder (family and weights hash). An existing\n" + " registry with speakers and no fingerprint is refused: see `parakeet-cli registry`.\n"; // parakeet-cli enroll --model --name --input [--input ...] // --registry @@ -1781,6 +1783,7 @@ static int cmd_enroll(int argc, char** argv) { } } int clips = 0; + bool warned = false; for (const std::string& in : inputs) { pk::Audio audio; if (!load_audio_arg_16k_mono(in, audio)) { @@ -1794,8 +1797,14 @@ static int cmd_enroll(int argc, char** argv) { enc->last_error().c_str()); return 1; } - try { reg.enroll(name, emb); } - catch (const std::exception& e) { + try { + // Records the encoder that made the voice (family and weights hash). + const pk::FingerprintVerdict v = reg.enroll(name, emb, enc->fingerprint()); + if (v.is_warning() && !warned) { + std::fprintf(stderr, "parakeet-cli enroll: warning: %s\n", v.message.c_str()); + warned = true; + } + } catch (const std::exception& e) { std::fprintf(stderr, "parakeet-cli enroll: %s\n", e.what()); return 1; } @@ -1813,15 +1822,90 @@ static int cmd_enroll(int argc, char** argv) { return 0; } +static const char* kRegistryUsage = + "usage: parakeet-cli registry [--restamp --encoder ]\n" + " Prints the speaker registry's format version, embedding size, encoder\n" + " fingerprint and speakers. With --restamp it writes the fingerprint of\n" + " the given encoder into a registry that has none (a version 1 file).\n" + " Only restamp with the encoder that made the voices: nothing can verify it.\n"; + +// parakeet-cli registry [--restamp --encoder ] +static int cmd_registry(int argc, char** argv) { + std::string path, encoder; + bool restamp = false; + for (int i = 0; i < argc; ++i) { + if (std::strcmp(argv[i], "--restamp") == 0) restamp = true; + else if (std::strcmp(argv[i], "--encoder") == 0 && i + 1 < argc) encoder = argv[++i]; + else if (argv[i][0] != '-' && path.empty()) path = argv[i]; + else { std::fprintf(stderr, "%s", kRegistryUsage); return 2; } + } + if (path.empty() || (restamp && encoder.empty()) || (!restamp && !encoder.empty())) { + std::fprintf(stderr, "%s", kRegistryUsage); + return 2; + } + std::string blob; + const int rerr = read_file_bytes(path, blob); + if (rerr != 0) { + std::fprintf(stderr, "parakeet-cli registry: %s\n", registry_read_error(path, rerr).c_str()); + return 1; + } + pk::SpeakerRegistry reg; + try { reg = pk::SpeakerRegistry::deserialize(blob); } + catch (const std::exception& e) { + std::fprintf(stderr, "parakeet-cli registry: %s is not a speaker registry: %s\n", path.c_str(), e.what()); + return 1; + } + if (restamp) { + if (!reg.fingerprint().empty()) { + std::fprintf(stderr, "parakeet-cli registry: %s already has a fingerprint (%s); " + "enroll again to change the encoder\n", path.c_str(), reg.fingerprint().family.c_str()); + return 1; + } + if (!pk::SpeakerEncoder::available()) { + std::fprintf(stderr, "parakeet-cli: built without speaker identification (PARAKEET_WITH_VOICEDETECT=OFF)\n"); + return 2; + } + auto enc = pk::SpeakerEncoder::load(encoder); + if (!enc) { + std::fprintf(stderr, "parakeet-cli registry: failed to load speaker model %s\n", encoder.c_str()); + return 1; + } + if (reg.dim() != 0 && reg.dim() != enc->dim()) { + std::fprintf(stderr, "parakeet-cli registry: registry holds %d-value embeddings, %s produces %d; " + "this is not the encoder that made them\n", reg.dim(), encoder.c_str(), enc->dim()); + return 1; + } + reg.set_fingerprint(enc->fingerprint()); + std::string werr; + if (!pk::write_file_atomic(path, reg.serialize(), &werr)) { + std::fprintf(stderr, "parakeet-cli registry: %s\n", werr.c_str()); + return 1; + } + std::printf("restamped %s with the fingerprint of %s\n", path.c_str(), encoder.c_str()); + } + uint32_t ver = 0; + std::memcpy(&ver, blob.data() + 4, 4); + if (restamp) ver = 2; + const pk::EncoderFingerprint& fp = reg.fingerprint(); + std::printf("format version: %u\n", ver); + std::printf("embedding size: %d\n", reg.dim()); + std::printf("encoder family: %s\n", fp.family.empty() ? "(none)" : fp.family.c_str()); + std::printf("encoder weights: %s\n", fp.weights.empty() ? "(none)" : fp.weights.c_str()); + std::printf("speakers: %zu\n", reg.size()); + for (const std::string& n : reg.names()) std::printf(" %s\n", n.c_str()); + return 0; +} + static const char* kSceneUsage = "usage: parakeet-cli scene [--model ] [--diar ] " "[--sound ] [--speakers --registry " - "[--speaker-threshold F]] --input " + "[--speaker-threshold F] [--strict-registry]] --input " "[--latency model|low|very_low|ultra_low] [--chunk-ms N] " "[--show-speech] [--json]\n" " each model may be a bundle GGUF (the only component of the right kind is used); name another with\n" " --asr-component, --diar-component, --sound-component or --speakers-component\n" - " --speaker-threshold: default 0.5; ECAPA needs about 0.7, see docs/speaker.md\n"; + " --speaker-threshold: default 0.5; ECAPA needs about 0.7, see docs/speaker.md\n" + " --strict-registry: refuse a registry with no encoder fingerprint\n"; // parakeet-cli scene [--model ] [--diar ] [--sound ] // [--speakers --registry [--speaker-threshold F]] @@ -1836,6 +1920,7 @@ static const char* kSceneUsage = static int cmd_scene(int argc, char** argv) { std::string model, diar, sound, input, latency_str; std::string speakers, registry_path; + bool strict_registry = false; std::string asr_comp_arg, diar_comp_arg, sound_comp_arg, speakers_comp_arg; bool have_threshold = false; float speaker_threshold = 0.0f; @@ -1861,6 +1946,8 @@ static int cmd_scene(int argc, char** argv) { speakers = argv[++i]; } else if (std::strcmp(argv[i], "--registry") == 0 && i + 1 < argc) { registry_path = argv[++i]; + } else if (std::strcmp(argv[i], "--strict-registry") == 0) { + strict_registry = true; } else if (std::strcmp(argv[i], "--speaker-threshold") == 0 && i + 1 < argc) { char* end = nullptr; const char* txt = argv[++i]; @@ -1921,8 +2008,8 @@ static int cmd_scene(int argc, char** argv) { std::fprintf(stderr, "parakeet-cli: built without sound tagging (PARAKEET_WITH_CED=OFF)\n"); return 2; } - if (speakers.empty() && (!registry_path.empty() || have_threshold)) { - std::fprintf(stderr, "parakeet-cli scene: --registry and --speaker-threshold need --speakers\n"); + if (speakers.empty() && (!registry_path.empty() || have_threshold || strict_registry)) { + std::fprintf(stderr, "parakeet-cli scene: --registry, --speaker-threshold and --strict-registry need --speakers\n"); return 2; } pk::SpeakerIdOpts speaker_opts; @@ -2001,13 +2088,16 @@ static int cmd_scene(int argc, char** argv) { registry_path.c_str(), e.what()); return 1; } - if (registry.dim() != speaker_enc->dim()) { - std::fprintf(stderr, - "parakeet-cli scene: registry %s holds %d-dim voices but %s makes %d-dim embeddings " - "(enroll again with this model)\n", - registry_path.c_str(), registry.dim(), speakers.c_str(), speaker_enc->dim()); + // Same check and same text as the C-API, before any name is assigned. + const pk::FingerprintVerdict v = pk::check_registry_for_encoder( + registry, speaker_enc->dim(), speaker_enc->fingerprint(), strict_registry); + if (v.is_error()) { + std::fprintf(stderr, "parakeet-cli scene: %s (registry %s, speaker model %s)\n", + v.message.c_str(), registry_path.c_str(), speakers.c_str()); return 1; } + if (v.is_warning()) + std::fprintf(stderr, "parakeet-cli scene: warning: %s\n", v.message.c_str()); } pk::Audio audio; @@ -2256,6 +2346,8 @@ int main(int argc, char** argv) { return run_and_shutdown(cmd_bench, argc - 2, argv + 2); if (argc >= 2 && std::strcmp(argv[1], "enroll") == 0) return run_and_shutdown(cmd_enroll, argc - 2, argv + 2); + if (argc >= 2 && std::strcmp(argv[1], "registry") == 0) + return run_and_shutdown(cmd_registry, argc - 2, argv + 2); if (argc >= 2 && std::strcmp(argv[1], "scene") == 0) return run_and_shutdown(cmd_scene, argc - 2, argv + 2); if (argc >= 2 && std::strcmp(argv[1], "vad") == 0) @@ -2286,12 +2378,13 @@ int main(int argc, char** argv) { "[--batch-sizes 1,4,8,16] [--threads N] [--reps R] [--json ]\n" " parakeet-cli scene [--model ] [--diar ] " "[--sound ] [--speakers --registry " - "[--speaker-threshold F]] --input " + "[--speaker-threshold F] [--strict-registry]] --input " "[--latency model|low|very_low|ultra_low] [--chunk-ms N] " "[--show-speech] [--json]\n" " --speaker-threshold: default 0.5; ECAPA needs about 0.7, see docs/speaker.md\n" " each model may be a bundle (--asr-component, --diar-component, --sound-component, --speakers-component)\n" " parakeet-cli enroll --model [--component NAME] --name " - "--input [--input ...] --registry \n"); + "--input [--input ...] --registry \n" + " parakeet-cli registry [--restamp --encoder ]\n"); return 2; } diff --git a/include/parakeet_capi.h b/include/parakeet_capi.h index 53b9c81..4640853 100644 --- a/include/parakeet_capi.h +++ b/include/parakeet_capi.h @@ -67,6 +67,9 @@ typedef struct parakeet_ctx parakeet_ctx; // and diarize-only naming (parakeet_capi_diarize_named_pcm_json), for // callers that keep speaker embeddings themselves. Additive: no // existing signature changed. +// Speaker registry encoder fingerprint (parakeet_capi_speaker_registry_add_embedding_fp, +// _encoder_family/_encoder_weights/_set_strict, parakeet_capi_speaker_encoder_family, +// parakeet_capi_speaker_last_warning) is additive and keeps ABI v10. // Standalone VAD (parakeet_capi_vad_*, parakeet_capi_vad_stream_*, // parakeet_capi_transcribe_path_json_vad*) and Silero VAD contexts are // additive and keep ABI v10: a caller that needs them checks for the symbols @@ -824,6 +827,50 @@ int parakeet_capi_speaker_enroll(parakeet_speaker_registry* reg, parakeet_ctx* s int parakeet_capi_speaker_registry_add_embedding(parakeet_speaker_registry* reg, const char* name, const float* embedding, int dim); +// --- Encoder fingerprint (additive; ABI stays 10) --------------------------- +// A registry remembers which speaker encoder made its embeddings, because equal +// embedding sizes do not mean the same embedding space. Two strings: +// family: "voicedetect:::" from the encoder GGUF +// metadata. A different family is an error. +// weights: "sha256:<64 hex>" of the encoder GGUF file bytes (the string +// parakeet_capi_speaker_identity returns). A different weights hash +// of the same family (another quantization) is only a warning. +// Every named entry point (diarize_named_pcm_json, diarize_profiles_pcm_json, +// transcribe_and_diarize_named_json, scene_stream_begin_speaker, +// speaker_identify_pcm_json, speaker_enroll) checks the registry against the +// speaker ctx before it assigns any name; see docs/diarization.md. An error is +// reported like a size mismatch (NULL or nonzero, message on the speaker ctx). +// A warning is logged to stderr and kept in parakeet_capi_speaker_last_warning. + +// Same as _add_embedding, and records which encoder made the embedding. Pass the +// values of the model that computed it: parakeet_capi_speaker_encoder_family and +// parakeet_capi_speaker_identity of its ctx, or the strings you stored with the +// embedding. NULL or "" for both means no fingerprint (exactly _add_embedding). +// An empty registry takes the fingerprint. Refused (nonzero, message on the +// registry): another family than the registry's; a fingerprint for a registry +// that already holds unfingerprinted speakers (no silent stamping); no +// fingerprint for a fingerprinted registry. Another weights hash of the same +// family is accepted and the registry keeps its first hash. +int parakeet_capi_speaker_registry_add_embedding_fp(parakeet_speaker_registry* reg, const char* name, + const float* embedding, int dim, + const char* encoder_family, + const char* encoder_weights); +// The registry's fingerprint, "" when it has none. Borrowed until the registry +// changes or is freed. NULL registry gives "". +const char* parakeet_capi_speaker_registry_encoder_family(const parakeet_speaker_registry* reg); +const char* parakeet_capi_speaker_registry_encoder_weights(const parakeet_speaker_registry* reg); +// With strict != 0, the named entry points refuse a registry that has speakers +// and no fingerprint. Not saved to the file; default 0 (such a registry is +// accepted with a warning). Does nothing on NULL. +void parakeet_capi_speaker_registry_set_strict(parakeet_speaker_registry* reg, int strict); +// Family of a speaker ctx's encoder, "voicedetect:::"; NULL +// for NULL or a context that is not a speaker model. Borrowed until free. +const char* parakeet_capi_speaker_encoder_family(const parakeet_ctx* speaker); +// Warning from the latest check or enrollment on this speaker ctx (for example +// "same family, other weights"), "" if none. Cleared by the next check. Borrowed. +// NULL ctx gives "". +const char* parakeet_capi_speaker_last_warning(const parakeet_ctx* speaker); + // Binary file. 0 on success; nonzero on error (message on the registry). int parakeet_capi_speaker_registry_save(const parakeet_speaker_registry* reg, const char* path); // NULL when the file is missing or is not a valid registry. Free with _free. diff --git a/src/parakeet_capi.cpp b/src/parakeet_capi.cpp index 9d8a93a..b3d5e5c 100644 --- a/src/parakeet_capi.cpp +++ b/src/parakeet_capi.cpp @@ -108,6 +108,7 @@ struct parakeet_ctx { std::unique_ptr speaker; std::unique_ptr silero; // Silero VAD model (architecture "silero_vad") std::string speaker_identity; + std::string speaker_warning; // latest registry/encoder check warning ErrorSlot last_error; }; @@ -115,6 +116,7 @@ struct parakeet_ctx { struct parakeet_speaker_registry { pk::SpeakerRegistry reg; std::string last_error; + bool strict = false; // refuse a registry with speakers and no fingerprint }; // The opaque streaming session: a pk::StreamingSession over the ctx's model plus @@ -295,15 +297,8 @@ parakeet_ctx* load_bundle_component(const char* path, const std::string& name) { // Profiles are tied to the exact model. For a component the identity is the sha256 // of the single-model file it was built from, recorded in the bundle header, so a // voice enrolled with the standalone file matches the same model inside a bundle. - const std::string& h = c->source_sha256; - bool hex = h.size() == 64; - for (char ch : h) hex = hex && ((ch >= '0' && ch <= '9') || (ch >= 'a' && ch <= 'f')); - if (!hex) { - g_load_error = "voice component \"" + name + "\" has no valid source_sha256 in the bundle header, " - "which is its speaker model identity"; - return nullptr; - } - ctx->speaker_identity = "sha256:" + h; + // The encoder carries the same value in its fingerprint, which the registry checks. + ctx->speaker_identity = ctx->speaker->fingerprint().weights; } else if (c->kind == pk::kBundleKindVad && pk::bundle_vad_is_slice(path, name)) { // A VAD-only slice: same context as a standalone slice file. ctx->vad_model = pk::Model::load_vad_only(path, name); @@ -378,11 +373,11 @@ parakeet_ctx* load_plain(const char* gguf_path) { // A voice-detect GGUF (architecture "voicedetect") is a speaker encoder. if (pk::gguf_is_voicedetect(gguf_path)) { - try { - ctx->speaker = pk::load_speaker_with_identity(gguf_path, ctx->speaker_identity, - [&] { return pk::SpeakerEncoder::load(gguf_path); }); - } catch (...) { delete ctx; return nullptr; } - if (ctx->speaker) return ctx; + ctx->speaker = pk::SpeakerEncoder::load(gguf_path); // hashes the file on both sides of the load + if (ctx->speaker) { + ctx->speaker_identity = ctx->speaker->fingerprint().weights; + return ctx; + } delete ctx; return nullptr; } @@ -1487,6 +1482,22 @@ bool require_speaker(parakeet_ctx* ctx) { return true; } +// Runs where an encoder meets a registry, before any name is assigned: embedding +// size, then encoder family (error), weights and missing fingerprint (warning, +// or an error in strict mode). Sets speaker->last_error on an error, and +// speaker->speaker_warning plus a log line on a warning. +bool check_registry(parakeet_ctx* speaker, const parakeet_speaker_registry* reg) { + speaker->speaker_warning.clear(); + const pk::FingerprintVerdict v = pk::check_registry_for_encoder( + reg->reg, speaker->speaker->dim(), speaker->speaker->fingerprint(), reg->strict); + if (v.is_error()) { speaker->last_error = v.message; return false; } + if (v.is_warning()) { + speaker->speaker_warning = v.message; + PK_LOG("warning: %s", v.message.c_str()); + } + return true; +} + std::string diar_result_json_string(const pk::DiarizationResult& r) { std::string json = "{\"speakers\":"; pk::append_json_int(json, r.n_speakers); @@ -2133,13 +2144,7 @@ extern "C" parakeet_scene_stream* parakeet_capi_scene_stream_begin_speaker( so.max_voice_sec = scene_float_field(o, offsetof(parakeet_scene_opts, speaker_max_voice_sec), so.max_voice_sec); const std::string err = pk::validate_speaker_opts(so); if (!err.empty()) { speaker->last_error = "invalid speaker options: " + err; return nullptr; } - const int rd = registry->reg.dim(); - if (rd != 0 && rd != speaker->speaker->dim()) { - speaker->last_error = "registry holds " + std::to_string(rd) + - "-value embeddings, this model produces " + - std::to_string(speaker->speaker->dim()); - return nullptr; - } + if (!check_registry(speaker, registry)) return nullptr; } try { pk::SceneParts p; @@ -2276,7 +2281,12 @@ extern "C" int parakeet_capi_speaker_enroll(parakeet_speaker_registry* reg, para if (!name || !*name) { speaker->last_error = "speaker name is empty"; return 1; } std::vector emb; if (!speaker_embed_pcm(speaker, pcm, n, sample_rate, emb)) return 1; - reg->reg.enroll(name, emb); + speaker->speaker_warning.clear(); + const pk::FingerprintVerdict v = reg->reg.enroll(name, emb, speaker->speaker->fingerprint()); + if (v.is_warning()) { + speaker->speaker_warning = v.message; + PK_LOG("warning: %s", v.message.c_str()); + } speaker->last_error.clear(); return 0; } catch (const std::exception& e) { @@ -2304,6 +2314,48 @@ extern "C" int parakeet_capi_speaker_registry_add_embedding(parakeet_speaker_reg return 1; } +extern "C" int parakeet_capi_speaker_registry_add_embedding_fp(parakeet_speaker_registry* reg, const char* name, + const float* embedding, int dim, + const char* encoder_family, + const char* encoder_weights) { + if (!reg) return 1; + try { + if (!name || !*name) { reg->last_error = "speaker name is empty"; return 1; } + if (!embedding || dim <= 0) { reg->last_error = "no embedding"; return 1; } + pk::EncoderFingerprint fp{encoder_family ? encoder_family : "", encoder_weights ? encoder_weights : ""}; + const pk::FingerprintVerdict v = + reg->reg.enroll(name, std::vector(embedding, embedding + dim), fp); + if (v.is_warning()) PK_LOG("warning: %s", v.message.c_str()); + reg->last_error.clear(); + return 0; + } catch (const std::exception& e) { + reg->last_error = e.what(); + } catch (...) { + reg->last_error = "unknown error"; + } + return 1; +} + +extern "C" const char* parakeet_capi_speaker_registry_encoder_family(const parakeet_speaker_registry* reg) { + return reg ? reg->reg.fingerprint().family.c_str() : ""; +} + +extern "C" const char* parakeet_capi_speaker_registry_encoder_weights(const parakeet_speaker_registry* reg) { + return reg ? reg->reg.fingerprint().weights.c_str() : ""; +} + +extern "C" void parakeet_capi_speaker_registry_set_strict(parakeet_speaker_registry* reg, int strict) { + if (reg) reg->strict = strict != 0; +} + +extern "C" const char* parakeet_capi_speaker_encoder_family(const parakeet_ctx* speaker) { + return speaker && speaker->speaker ? speaker->speaker->fingerprint().family.c_str() : nullptr; +} + +extern "C" const char* parakeet_capi_speaker_last_warning(const parakeet_ctx* speaker) { + return speaker ? speaker->speaker_warning.c_str() : ""; +} + extern "C" int parakeet_capi_speaker_registry_save(const parakeet_speaker_registry* reg, const char* path) { if (!reg) return 1; auto* mreg = const_cast(reg); // only last_error is written @@ -2358,6 +2410,7 @@ extern "C" char* parakeet_capi_speaker_identify_pcm_json(parakeet_speaker_regist try { if (!require_speaker(speaker)) return nullptr; if (!reg) { speaker->last_error = "registry is NULL"; return nullptr; } + if (!check_registry(speaker, reg)) return nullptr; std::vector emb; if (!speaker_embed_pcm(speaker, pcm, n, sample_rate, emb)) return nullptr; const pk::SpeakerIdOpts d; @@ -2390,13 +2443,7 @@ extern "C" char* parakeet_capi_transcribe_and_diarize_named_json( int n_speakers = 0; if (!run_sas(asr_ctx, diar_ctx, samples, n_samples, sample_rate, words, n_speakers, &segs, &pcm16k)) return nullptr; - const int rd = registry->reg.dim(); - if (rd != 0 && rd != speaker->speaker->dim()) { - speaker->last_error = "registry holds " + std::to_string(rd) + - "-value embeddings, this model produces " + - std::to_string(speaker->speaker->dim()); - return nullptr; - } + if (!check_registry(speaker, registry)) return nullptr; std::map names; try { names = pk::identify_offline(pcm16k, segs, speaker->speaker->embedder(), registry->reg, @@ -2461,13 +2508,7 @@ static char* diarize_named_pcm_json(bool export_profiles, parakeet_ctx* diar_ctx diar_ctx->last_error = "invalid sample rate"; return nullptr; } - const int rd = registry->reg.dim(); - if (rd != 0 && rd != speaker->speaker->dim()) { - speaker->last_error = "registry holds " + std::to_string(rd) + - "-value embeddings, this model produces " + - std::to_string(speaker->speaker->dim()); - return nullptr; - } + if (!check_registry(speaker, registry)) return nullptr; pk::SpeakerIdOpts o; if (accept_threshold != 0.0f) o.accept_threshold = accept_threshold; if (margin != 0.0f) o.margin = margin; diff --git a/src/speaker_encoder.cpp b/src/speaker_encoder.cpp index cd74f47..a7b0564 100644 --- a/src/speaker_encoder.cpp +++ b/src/speaker_encoder.cpp @@ -1,8 +1,12 @@ #include "speaker_encoder.hpp" +#include "bundle.hpp" #include "bundle_extract.hpp" +#include "speaker_model_identity.hpp" #include "gguf.h" +#include + #ifdef PARAKEET_WITH_VOICEDETECT #include "voicedetect_capi.h" #endif @@ -20,23 +24,85 @@ bool gguf_is_voicedetect(const std::string& path) { return vd; } +std::string speaker_encoder_family(const std::string& path, int dim_fallback) { + gguf_init_params p{/*no_alloc=*/true, /*ctx=*/nullptr}; + gguf_context* g = gguf_init_from_file(path.c_str(), p); + if (!g) return ""; + auto str = [&](const char* key) { + const int64_t id = gguf_find_key(g, key); + return (id >= 0 && gguf_get_kv_type(g, id) == GGUF_TYPE_STRING) + ? std::string(gguf_get_val_str(g, id)) : std::string(); + }; + std::string out; + if (str("general.architecture") == "voicedetect") { + long long dim = dim_fallback; + const int64_t id = gguf_find_key(g, "voicedetect.embedding_dim"); + if (id >= 0) { + const gguf_type t = gguf_get_kv_type(g, id); + if (t == GGUF_TYPE_UINT32) dim = gguf_get_val_u32(g, id); + else if (t == GGUF_TYPE_INT32) dim = gguf_get_val_i32(g, id); + } + out = "voicedetect:" + str("voicedetect.arch") + ":" + str("general.name") + ":" + + (dim > 0 ? std::to_string(dim) : std::string()); + } + gguf_free(g); + return out; +} + #ifdef PARAKEET_WITH_VOICEDETECT bool SpeakerEncoder::available() { return true; } +// The weights identity of a standalone file is the sha256 of its bytes, taken on both +// sides of the load: a file replaced while loading is refused. Model files must stay +// immutable in service. The identity of a bundle component is the sha256 of the +// single-model file it was built from, recorded in the bundle header, so an encoder +// enrolled from the standalone file matches the same encoder inside a bundle. std::unique_ptr SpeakerEncoder::load(const std::string& path, const std::string& component, std::string* err) { - voicedetect_ctx* c = nullptr; - if (component.empty()) { - c = voicedetect_capi_load(path.c_str()); - } else { - std::string e; - std::unique_ptr cf = ComponentFile::create(path, component, &e); - if (!cf) { if (err) *err = e; return nullptr; } - c = voicedetect_capi_load(cf->path().c_str()); // the file is removed when cf goes out of scope + std::string weights; + std::unique_ptr cf; + std::string load_path = path; + try { + if (component.empty()) { + weights = speaker_model_identity(path); + } else { + BundleInfo info; + std::string e; + if (!read_bundle_info(path, info, &e)) { if (err) *err = e; return nullptr; } + const BundleComponent* c = info.find(component); + if (!c) { if (err) *err = "the bundle has no component \"" + component + "\""; return nullptr; } + const std::string& h = c->source_sha256; + bool hex = h.size() == 64; + for (char ch : h) hex = hex && ((ch >= '0' && ch <= '9') || (ch >= 'a' && ch <= 'f')); + if (!hex) { + if (err) *err = "voice component \"" + component + "\" has no valid source_sha256 in the bundle " + "header, which is its speaker model identity"; + return nullptr; + } + weights = "sha256:" + h; + cf = ComponentFile::create(path, component, &e); + if (!cf) { if (err) *err = e; return nullptr; } + load_path = cf->path(); // the file is removed when cf goes out of scope + } + std::unique_ptr enc = load_unchecked(load_path, weights, err); + if (enc && component.empty() && speaker_model_identity(path) != weights) { + if (err) *err = "the speaker model changed during load"; + return nullptr; + } + return enc; + } catch (...) { + if (err && err->empty()) *err = "cannot read the speaker model " + (component.empty() ? path : component); + return nullptr; } +} + +std::unique_ptr SpeakerEncoder::load_unchecked(const std::string& path, + const std::string& weights, + std::string* err) { + voicedetect_ctx* c = voicedetect_capi_load(path.c_str()); if (!c) { - if (err && err->empty()) *err = "cannot load the speaker model " + (component.empty() ? path : component); + if (err && err->empty()) *err = "cannot load the speaker model"; return nullptr; } const int dim = voicedetect_capi_embedding_dim(c); @@ -48,6 +114,8 @@ std::unique_ptr SpeakerEncoder::load(const std::string& path, co std::unique_ptr e(new SpeakerEncoder()); e->ctx_ = c; e->dim_ = dim; + e->fp_.family = speaker_encoder_family(path, dim); + e->fp_.weights = weights; return e; } diff --git a/src/speaker_encoder.hpp b/src/speaker_encoder.hpp index b5e2b75..3a9b657 100644 --- a/src/speaker_encoder.hpp +++ b/src/speaker_encoder.hpp @@ -29,6 +29,11 @@ class SpeakerEncoder { SpeakerEncoder& operator=(const SpeakerEncoder&) = delete; int dim() const { return dim_; } + // Which encoder this is: family from the GGUF metadata, weights as the + // sha256 of the GGUF file bytes, or, for a bundle component, the sha256 of the + // single-model file recorded in the bundle header (source_sha256). See + // speaker_encoder_family. + const EncoderFingerprint& fingerprint() const { return fp_; } // L2-normalized embedding of 16 kHz mono PCM. False on failure (see last_error). bool embed(const float* pcm, int n, std::vector& emb); // A SpeakerEmbed bound to this encoder; valid while the encoder lives. @@ -37,11 +42,21 @@ class SpeakerEncoder { private: SpeakerEncoder() = default; + static std::unique_ptr load_unchecked(const std::string& path, + const std::string& weights, std::string* err); void* ctx_ = nullptr; // voicedetect_ctx* int dim_ = 0; + EncoderFingerprint fp_; std::string last_error_; }; +// Encoder family id of a voice-detect GGUF, read from its header only: +// "voicedetect:::" +// A missing key leaves its field empty ("voicedetect::name:256"). "" when the file +// is not a readable voice-detect GGUF. `dim_fallback` is used when the GGUF has no +// embedding_dim key (0 leaves the field empty). +std::string speaker_encoder_family(const std::string& gguf_path, int dim_fallback = 0); + // True when the GGUF's general.architecture is "voicedetect". Reads only the header. bool gguf_is_voicedetect(const std::string& gguf_path); diff --git a/src/speaker_registry.cpp b/src/speaker_registry.cpp index ccc1e3d..94ec2d9 100644 --- a/src/speaker_registry.cpp +++ b/src/speaker_registry.cpp @@ -10,7 +10,9 @@ namespace pk { namespace { constexpr char kMagic[4] = {'P', 'K', 'S', 'R'}; -constexpr uint32_t kVersion = 1; +constexpr uint32_t kVersion = 1; // no fingerprint +constexpr uint32_t kVersion2 = 2; // with a fingerprint +constexpr uint32_t kMaxFpLen = 4096; constexpr uint32_t kMaxSpeakers = 1u << 20; constexpr uint32_t kMaxNameLen = 4096; constexpr int kMaxDim = 1 << 16; @@ -41,8 +43,52 @@ struct Reader { } }; +std::string shown(const std::string& s) { return s.empty() ? "unknown" : s; } + } // namespace +FingerprintVerdict check_fingerprint(const EncoderFingerprint& reg, const EncoderFingerprint& enc, + bool strict) { + FingerprintVerdict v; + if (enc.family.empty()) return v; + if (reg.empty()) { + v.status = strict ? FingerprintStatus::Required : FingerprintStatus::Unfingerprinted; + v.message = strict + ? "speaker registry has no encoder fingerprint and strict mode is on; enroll again " + "with this encoder (" + enc.family + ") or re-stamp the registry" + : "speaker registry has no encoder fingerprint; its encoder is unverified, so names " + "may come from the wrong embedding space (current encoder: " + enc.family + ")"; + return v; + } + if (!reg.family.empty() && reg.family != enc.family) { + v.status = FingerprintStatus::FamilyDiffer; + v.message = "speaker registry was made with encoder family " + shown(reg.family) + + ", but the encoder in use is " + enc.family + + "; the embeddings are not comparable, enroll again with this encoder"; + return v; + } + if (!reg.weights.empty() && !enc.weights.empty() && reg.weights != enc.weights) { + v.status = FingerprintStatus::WeightsDiffer; + v.message = "speaker registry was made with other weights of encoder family " + enc.family + + " (" + reg.weights + ", now " + enc.weights + + "); a different quantization is expected to work, scores may shift slightly"; + } + return v; +} + +FingerprintVerdict check_registry_for_encoder(const SpeakerRegistry& reg, int enc_dim, + const EncoderFingerprint& enc, bool strict) { + if (reg.dim() != 0 && reg.dim() != enc_dim) { + FingerprintVerdict v; + v.status = FingerprintStatus::DimDiffer; + v.message = "registry holds " + std::to_string(reg.dim()) + + "-value embeddings, this model produces " + std::to_string(enc_dim); + return v; + } + if (reg.size() == 0 && reg.fingerprint().empty()) return {}; // nothing enrolled, nothing to mix up + return check_fingerprint(reg.fingerprint(), enc, strict); +} + std::vector SpeakerRegistry::names() const { std::vector out; out.reserve(entries_.size()); @@ -51,6 +97,14 @@ std::vector SpeakerRegistry::names() const { } void SpeakerRegistry::enroll(const std::string& name, const std::vector& emb) { + if (!fp_.empty()) + throw std::invalid_argument( + "speaker registry is fingerprinted (" + shown(fp_.family) + + "); the embedding came with no encoder fingerprint, so it is not added"); + enroll_raw(name, emb); +} + +void SpeakerRegistry::enroll_raw(const std::string& name, const std::vector& emb) { if (name.empty()) throw std::invalid_argument("speaker name is empty"); if (emb.empty()) throw std::invalid_argument("speaker embedding is empty"); if (dim_ != 0 && (int)emb.size() != dim_) @@ -68,6 +122,26 @@ void SpeakerRegistry::enroll(const std::string& name, const std::vector& entries_.push_back({name, n, 1}); } +FingerprintVerdict SpeakerRegistry::enroll(const std::string& name, const std::vector& emb, + const EncoderFingerprint& fp) { + FingerprintVerdict v; + if (fp.empty()) { + enroll(name, emb); + return v; + } + if (!fp_.empty()) { + v = check_fingerprint(fp_, fp, false); + if (v.is_error()) throw std::invalid_argument(v.message); + } else if (!entries_.empty()) { + throw std::invalid_argument( + "speaker registry has speakers but no encoder fingerprint; they are not stamped " + "silently. Re-stamp the registry with the encoder that made them, or enroll into a new registry"); + } + enroll_raw(name, emb); // throws before anything changes + if (fp_.empty()) fp_ = fp; + return v; +} + bool SpeakerRegistry::remove(const std::string& name) { for (size_t i = 0; i < entries_.size(); ++i) { if (entries_[i].name == name) { @@ -107,12 +181,19 @@ SpeakerMatch SpeakerRegistry::identify(const std::vector& emb, float acce std::string SpeakerRegistry::serialize() const { std::string s; put(s, kMagic, 4); - const uint32_t ver = kVersion; + const uint32_t ver = fp_.empty() ? kVersion : kVersion2; put(s, &ver, 4); const int32_t dim = dim_; put(s, &dim, 4); const uint32_t n = (uint32_t)entries_.size(); put(s, &n, 4); + if (ver == kVersion2) { + for (const std::string* f : {&fp_.family, &fp_.weights}) { + const uint32_t len = (uint32_t)f->size(); + put(s, &len, 4); + put(s, f->data(), f->size()); + } + } for (const Entry& e : entries_) { const uint32_t len = (uint32_t)e.name.size(); put(s, &len, 4); @@ -132,7 +213,7 @@ SpeakerRegistry SpeakerRegistry::deserialize(const std::string& blob) { throw std::runtime_error("speaker registry: bad magic"); uint32_t ver = 0; r.get(&ver, 4); - if (ver != kVersion) throw std::runtime_error("speaker registry: unsupported version"); + if (ver != kVersion && ver != kVersion2) throw std::runtime_error("speaker registry: unsupported version"); int32_t dim = 0; r.get(&dim, 4); uint32_t n = 0; @@ -140,6 +221,16 @@ SpeakerRegistry SpeakerRegistry::deserialize(const std::string& blob) { if (dim < 0 || dim > kMaxDim || n > kMaxSpeakers || (dim < 1 && n > 0)) throw std::runtime_error("speaker registry: implausible header"); SpeakerRegistry out(dim); + if (ver == kVersion2) { + for (std::string* f : {&out.fp_.family, &out.fp_.weights}) { + uint32_t len = 0; + r.get(&len, 4); + if (len > kMaxFpLen) throw std::runtime_error("speaker registry: bad fingerprint"); + f->resize(len); + if (len) r.get(&(*f)[0], len); + } + if (out.fp_.empty()) throw std::runtime_error("speaker registry: bad fingerprint"); + } for (uint32_t i = 0; i < n; ++i) { uint32_t len = 0; r.get(&len, 4); diff --git a/src/speaker_registry.hpp b/src/speaker_registry.hpp index d96d576..e4452cc 100644 --- a/src/speaker_registry.hpp +++ b/src/speaker_registry.hpp @@ -11,6 +11,56 @@ struct SpeakerMatch { float score = 0.0f; }; +// Which speaker encoder made a set of embeddings. Equal embedding sizes do not +// mean the same embedding space, so a registry remembers its encoder. +// family: names the embedding space. Built from the encoder GGUF metadata: +// "voicedetect:::". +// A mismatch is a hard error. +// weights: names the exact weights: "sha256:<64 hex>" of the encoder GGUF +// file bytes. Another quantization of the same family differs here. +// A mismatch is a warning. +// Either field may be empty (unknown). Both empty means "no fingerprint". +struct EncoderFingerprint { + std::string family; + std::string weights; + bool empty() const { return family.empty() && weights.empty(); } +}; + +enum class FingerprintStatus { + Match, // same family and same (or unrecorded) weights + Unfingerprinted, // the registry has no fingerprint: accepted with a warning + WeightsDiffer, // same family, other weights: accepted with a warning + FamilyDiffer, // other family: refused + Required, // no fingerprint and strict mode: refused + DimDiffer, // other embedding size: refused (checked before the fingerprint) +}; + +struct FingerprintVerdict { + FingerprintStatus status = FingerprintStatus::Match; + std::string message; // "" for Match; the one text used by every entry point + bool is_error() const { + return status == FingerprintStatus::FamilyDiffer || status == FingerprintStatus::Required || + status == FingerprintStatus::DimDiffer; + } + bool is_warning() const { + return status == FingerprintStatus::Unfingerprinted || status == FingerprintStatus::WeightsDiffer; + } +}; + +// Compares the fingerprint stored in a registry with the encoder in use. +// `strict` turns "registry has no fingerprint" into an error. An encoder with +// no family is not checked. +FingerprintVerdict check_fingerprint(const EncoderFingerprint& registry, + const EncoderFingerprint& encoder, bool strict); + +class SpeakerRegistry; + +// The check every entry point runs where an encoder meets a registry, before +// any name is assigned: embedding size first, then check_fingerprint. One text +// for the CLI and the C-API. `enc_dim` is the loaded encoder's embedding size. +FingerprintVerdict check_registry_for_encoder(const SpeakerRegistry& reg, int enc_dim, + const EncoderFingerprint& enc, bool strict); + // Enrolled speakers. Each speaker is the L2-normalized mean of the L2-normalized // embeddings enrolled under its name (a centroid), so enrolling more clips // tightens the match. Model-independent: it never sees audio. Not thread-safe. @@ -24,8 +74,25 @@ class SpeakerRegistry { std::vector names() const; // enrollment order // Throws std::invalid_argument: empty name, empty or all-zero embedding, - // or a size different from dim() when dim() != 0. + // a size different from dim() when dim() != 0, or a fingerprinted registry + // (use the overload with a fingerprint). void enroll(const std::string& name, const std::vector& emb); + // Same, and records which encoder made `emb`. With a non-empty `fp`: + // - an empty registry takes `fp` as its fingerprint; + // - a registry with another family throws std::invalid_argument (same text + // as check_fingerprint); + // - a registry with the same family and other weights keeps its own + // weights hash and the returned status is WeightsDiffer; + // - a non-empty registry with no fingerprint throws: it is never stamped + // silently (use set_fingerprint, for example `parakeet-cli registry + // --restamp`). + // With an empty `fp` and a fingerprinted registry it throws too, because the + // new vector would be unverifiable. Nothing changes when it throws. + FingerprintVerdict enroll(const std::string& name, const std::vector& emb, + const EncoderFingerprint& fp); + const EncoderFingerprint& fingerprint() const { return fp_; } + // Replaces the fingerprint with no checks. For an explicit re-stamp only. + void set_fingerprint(const EncoderFingerprint& fp) { fp_ = fp; } bool remove(const std::string& name); // Best speaker by cosine. Known only if the best score >= accept and, when @@ -34,20 +101,29 @@ class SpeakerRegistry { // is set). An all-zero embedding is unknown, not an error. SpeakerMatch identify(const std::vector& emb, float accept, float margin) const; - // Binary blob: "PKSR", u32 version 1, i32 dim, u32 n, then per speaker - // u32 name length, name bytes, i32 count, dim x f32 sum. Little-endian. + // Binary blob, little-endian. Version 1 (no fingerprint, written when the + // registry has none, so old readers keep working): + // "PKSR", u32 1, i32 dim, u32 n, then per speaker + // u32 name length, name bytes, i32 count, dim x f32 sum. + // Version 2 (written when there is a fingerprint) puts the fingerprint + // after the header and before the speakers: + // "PKSR", u32 2, i32 dim, u32 n, + // u32 family length, family bytes, u32 weights length, weights bytes, + // then the same per-speaker records. std::string serialize() const; // Throws std::runtime_error on bad magic or version, truncation, an absurd // count, or trailing bytes. static SpeakerRegistry deserialize(const std::string& blob); private: + void enroll_raw(const std::string& name, const std::vector& emb); struct Entry { std::string name; std::vector sum; // sum of L2-normalized enrollments int count = 0; }; int dim_; + EncoderFingerprint fp_; std::vector entries_; }; diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index d015ced..7a77577 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -108,6 +108,14 @@ set_tests_properties(test_speaker_identify PROPERTIES LABELS "model") pk_add_test(test_capi_speaker) target_compile_definitions(test_capi_speaker PRIVATE PK_SOURCE_DIR="${CMAKE_SOURCE_DIR}") set_tests_properties(test_capi_speaker PROPERTIES LABELS "model") +pk_add_test(test_speaker_fingerprint) +target_compile_definitions(test_speaker_fingerprint PRIVATE PK_SOURCE_DIR="${CMAKE_SOURCE_DIR}") +set_tests_properties(test_speaker_fingerprint PROPERTIES LABELS "model") +if(TARGET parakeet-cli AND PARAKEET_WITH_VOICEDETECT) + add_test(NAME test_cli_registry COMMAND sh ${CMAKE_CURRENT_SOURCE_DIR}/test_cli_registry.sh + $ ${CMAKE_SOURCE_DIR}/tests/fixtures/two_speakers.wav) + set_tests_properties(test_cli_registry PROPERTIES LABELS "model" SKIP_RETURN_CODE 77) +endif() pk_add_test(test_capi_diarize_named) target_compile_definitions(test_capi_diarize_named PRIVATE PK_SOURCE_DIR="${CMAKE_SOURCE_DIR}") set_tests_properties(test_capi_diarize_named PROPERTIES LABELS "model") diff --git a/tests/test_capi_speaker.cpp b/tests/test_capi_speaker.cpp index fb09566..64a1171 100644 --- a/tests/test_capi_speaker.cpp +++ b/tests/test_capi_speaker.cpp @@ -192,7 +192,7 @@ int main() { CHECK(parakeet_capi_speaker_enroll(wrong, alt, "x", a0.data(), (int)a0.size(), 16000) == 0); // identify with the main model against the alt-sized registry CHECK(parakeet_capi_speaker_identify_pcm_json(wrong, spk, a0.data(), (int)a0.size(), 16000) == nullptr); - CHECK(std::strstr(parakeet_capi_last_error(spk), "expects") != nullptr); + CHECK(std::strstr(parakeet_capi_last_error(spk), "-value embeddings") != nullptr); // and a scene stream refuses it up front parakeet_scene_opts o; parakeet_capi_scene_opts_default(&o); diff --git a/tests/test_capi_speaker_registry.cpp b/tests/test_capi_speaker_registry.cpp index 033becb..4bc6214 100644 --- a/tests/test_capi_speaker_registry.cpp +++ b/tests/test_capi_speaker_registry.cpp @@ -66,6 +66,52 @@ int main() { std::filesystem::remove(p); parakeet_capi_speaker_registry_free(reg); + + // Encoder fingerprint (additive, ABI stays 10). + { + const char* fam = "voicedetect:ecapa_tdnn:ecapa:4"; + const char* w1 = "sha256:1111"; + parakeet_speaker_registry* f = parakeet_capi_speaker_registry_new(); + CHECK(std::strlen(parakeet_capi_speaker_registry_encoder_family(f)) == 0); + CHECK(std::strlen(parakeet_capi_speaker_registry_encoder_weights(f)) == 0); + CHECK(parakeet_capi_speaker_registry_add_embedding_fp(f, "ada", a, 4, fam, w1) == 0); + CHECK(std::strcmp(parakeet_capi_speaker_registry_encoder_family(f), fam) == 0); + CHECK(std::strcmp(parakeet_capi_speaker_registry_encoder_weights(f), w1) == 0); + // Same family again, other weights: accepted. + CHECK(parakeet_capi_speaker_registry_add_embedding_fp(f, "ben", b, 4, fam, "sha256:2222") == 0); + CHECK(std::strcmp(parakeet_capi_speaker_registry_encoder_weights(f), w1) == 0); + // Other family, and no fingerprint at all, are refused with a message. + CHECK(parakeet_capi_speaker_registry_add_embedding_fp(f, "cy", a, 4, "voicedetect:campplus:c:4", w1) != 0); + CHECK(std::strstr(parakeet_capi_speaker_registry_last_error(f), "campplus") != nullptr); + CHECK(parakeet_capi_speaker_registry_add_embedding(f, "cy", a, 4) != 0); + CHECK(parakeet_capi_speaker_registry_size(f) == 2); + // NULL fingerprint arguments on the _fp call mean "none". + parakeet_speaker_registry* n = parakeet_capi_speaker_registry_new(); + CHECK(parakeet_capi_speaker_registry_add_embedding_fp(n, "ada", a, 4, nullptr, nullptr) == 0); + CHECK(std::strlen(parakeet_capi_speaker_registry_encoder_family(n)) == 0); + // A fingerprint into a non-empty unfingerprinted registry is refused, not stamped. + CHECK(parakeet_capi_speaker_registry_add_embedding_fp(n, "ben", b, 4, fam, w1) != 0); + CHECK(parakeet_capi_speaker_registry_encoder_family(nullptr) != nullptr); + // The fingerprint survives save and load. + const std::filesystem::path q = std::filesystem::temp_directory_path() / "pk_test_fp_registry.bin"; + CHECK(parakeet_capi_speaker_registry_save(f, q.string().c_str()) == 0); + parakeet_speaker_registry* back2 = parakeet_capi_speaker_registry_load(q.string().c_str()); + CHECK(back2 && std::strcmp(parakeet_capi_speaker_registry_encoder_family(back2), fam) == 0); + CHECK(back2 && std::strcmp(parakeet_capi_speaker_registry_encoder_weights(back2), w1) == 0); + // An unfingerprinted registry is still a v1 file. + CHECK(parakeet_capi_speaker_registry_save(n, q.string().c_str()) == 0); + { std::FILE* fh = std::fopen(q.string().c_str(), "rb"); unsigned char h[8] = {0}; + CHECK(fh && std::fread(h, 1, 8, fh) == 8 && h[4] == 1); + if (fh) std::fclose(fh); } + std::filesystem::remove(q); + parakeet_capi_speaker_registry_set_strict(n, 1); // no crash on NULL either + parakeet_capi_speaker_registry_set_strict(nullptr, 1); + CHECK(parakeet_capi_speaker_encoder_family(nullptr) == nullptr); + CHECK(parakeet_capi_speaker_last_warning(nullptr) != nullptr); + parakeet_capi_speaker_registry_free(back2); + parakeet_capi_speaker_registry_free(n); + parakeet_capi_speaker_registry_free(f); + } if (failures) { std::fprintf(stderr, "%d failure(s)\n", failures); return 1; } std::printf("test_capi_speaker_registry: PASS\n"); return 0; diff --git a/tests/test_cli_registry.sh b/tests/test_cli_registry.sh new file mode 100755 index 0000000..78029c1 --- /dev/null +++ b/tests/test_cli_registry.sh @@ -0,0 +1,36 @@ +#!/bin/sh +# parakeet-cli enroll / registry / scene against real voice-detect GGUFs. +# usage: test_cli_registry.sh +# PARAKEET_TEST_VD_GGUF speaker encoder (required, else skip 77) +# PARAKEET_TEST_VD_ALT_GGUF optional: other family, same embedding size +set -u +CLI=$1; WAV=$2 +[ -n "${PARAKEET_TEST_VD_GGUF:-}" ] || exit 77 +T=$(mktemp -d) || exit 1 +trap 'rm -rf "$T"' EXIT +fail() { echo "FAIL: $*" >&2; exit 1; } + +"$CLI" enroll --model "$PARAKEET_TEST_VD_GGUF" --name ada --input "$WAV" --registry "$T/r.bin" >/dev/null || fail enroll +"$CLI" registry "$T/r.bin" | grep -q '^format version: 2$' || fail "enroll did not write v2" +"$CLI" registry "$T/r.bin" | grep -q '^encoder family: voicedetect:' || fail "no family" +"$CLI" registry "$T/r.bin" | grep -q '^encoder weights: sha256:' || fail "no weights" +if [ -n "${PARAKEET_TEST_VD_ALT_GGUF:-}" ]; then + "$CLI" enroll --model "$PARAKEET_TEST_VD_ALT_GGUF" --name bob --input "$WAV" --registry "$T/r.bin" 2>"$T/err" \ + && fail "enroll with another family was accepted" + grep -q 'encoder family' "$T/err" || fail "no family message" + # a v1 registry (no fingerprint) is refused for enrolment and is never stamped silently + python3 - "$T/v1.bin" <<'PY' +import struct, sys +d = 192 +open(sys.argv[1], "wb").write(b"PKSR" + struct.pack("/dev/null \ + && fail "enroll into an unfingerprinted registry was accepted" + "$CLI" registry "$T/v1.bin" | grep -q '^format version: 1$' || fail "v1 changed by a refused enroll" + "$CLI" registry "$T/v1.bin" --restamp --encoder "$PARAKEET_TEST_VD_GGUF" >/dev/null || fail restamp + "$CLI" registry "$T/v1.bin" | grep -q '^format version: 2$' || fail "restamp did not write v2" + "$CLI" registry "$T/v1.bin" --restamp --encoder "$PARAKEET_TEST_VD_GGUF" >/dev/null 2>&1 && fail "second restamp accepted" +fi +echo "test_cli_registry: PASS" diff --git a/tests/test_speaker_fingerprint.cpp b/tests/test_speaker_fingerprint.cpp new file mode 100644 index 0000000..6c816ca --- /dev/null +++ b/tests/test_speaker_fingerprint.cpp @@ -0,0 +1,271 @@ +// Encoder fingerprint against real voice-detect GGUFs, through the C-API. +// +// PARAKEET_TEST_VD_GGUF speaker encoder GGUF (required, else skip 77) +// PARAKEET_TEST_VD_ALT_GGUF optional: another speaker encoder with the SAME embedding +// size but a different family (for example ECAPA and CAM++, +// both 192). Adds the real model-swap case. +// PARAKEET_TEST_DIAR_GGUF optional diarization GGUF; adds the diarize_named and +// scene paths +// PARAKEET_TEST_VD_BUNDLE optional: a bundle (docs/bundle.md) whose voice component was +// built from PARAKEET_TEST_VD_GGUF, unchanged. Checks that the +// same encoder gives the same family and weights identity from +// the standalone file and from the bundle component (name in +// PARAKEET_TEST_VD_BUNDLE_COMPONENT, default "voice"). +// +// Without a second encoder the swap is simulated: a copy of the first GGUF with a +// changed general.name has the same size and the same weights but another family. +// A copy with one added metadata key has the same family and other file bytes: the +// "other weights" case. +#include "parakeet_capi.h" + +#include "audio_io.hpp" +#include "gguf.h" + +#include +#include +#include +#include +#include +#include + +static int failures = 0; +#define CHECK(cond) \ + do { \ + if (!(cond)) { \ + std::fprintf(stderr, "FAIL: %s (line %d)\n", #cond, __LINE__); \ + ++failures; \ + } \ + } while (0) + +static std::vector slice(const std::vector& x, double a, double b) { + return std::vector(x.begin() + (long)(a * 16000), x.begin() + (long)(b * 16000)); +} + +// Copies `src` to `dst` through the GGUF API with one change. +static bool rewrite(const std::string& src, const std::string& dst, const char* key, const char* val) { + ggml_context* data = nullptr; + gguf_init_params p{/*no_alloc=*/false, &data}; + gguf_context* g = gguf_init_from_file(src.c_str(), p); + if (!g) return false; + gguf_context* out = gguf_init_empty(); + gguf_set_kv(out, g); + gguf_set_val_str(out, key, val); + for (ggml_tensor* t = ggml_get_first_tensor(data); t; t = ggml_get_next_tensor(data, t)) + gguf_add_tensor(out, t); + const bool ok = gguf_write_to_file(out, dst.c_str(), false); + gguf_free(out); + gguf_free(g); + ggml_free(data); + return ok; +} + +static bool contains(const char* s, const std::string& sub) { return s && std::strstr(s, sub.c_str()); } + +int main() { + const char* vd = std::getenv("PARAKEET_TEST_VD_GGUF"); + if (!vd) return 77; + const char* alt = std::getenv("PARAKEET_TEST_VD_ALT_GGUF"); + const char* diar_path = std::getenv("PARAKEET_TEST_DIAR_GGUF"); + + CHECK(parakeet_capi_abi_version() == 10); + parakeet_ctx* a = parakeet_capi_load(vd); + if (!a) { std::fprintf(stderr, "FAIL: load speaker model\n"); return 77; } + const std::string fam_a = parakeet_capi_speaker_encoder_family(a); + std::printf("encoder A: %s %s dim %d\n", fam_a.c_str(), parakeet_capi_speaker_identity(a), + parakeet_capi_speaker_dim(a)); + CHECK(fam_a.rfind("voicedetect:", 0) == 0); + CHECK(fam_a.size() > 12 && fam_a.compare(fam_a.size() - std::to_string(parakeet_capi_speaker_dim(a)).size(), + std::string::npos, std::to_string(parakeet_capi_speaker_dim(a))) == 0); + CHECK(std::strncmp(parakeet_capi_speaker_identity(a), "sha256:", 7) == 0); + + pk::Audio wav; + if (!pk::load_audio_16k_mono(std::string(PK_SOURCE_DIR) + "/tests/fixtures/two_speakers.wav", wav)) return 1; + const auto a0 = slice(wav.samples, 0.6, 4.6), b0 = slice(wav.samples, 6.9, 10.9); + const auto probe = slice(wav.samples, 14.9, 18.7); + + // Enrolment stamps the registry with the encoder that made the voices. + parakeet_speaker_registry* reg = parakeet_capi_speaker_registry_new(); + CHECK(parakeet_capi_speaker_enroll(reg, a, "speaker_a", a0.data(), (int)a0.size(), 16000) == 0); + CHECK(parakeet_capi_speaker_enroll(reg, a, "speaker_b", b0.data(), (int)b0.size(), 16000) == 0); + CHECK(fam_a == parakeet_capi_speaker_registry_encoder_family(reg)); + CHECK(std::strcmp(parakeet_capi_speaker_identity(a), parakeet_capi_speaker_registry_encoder_weights(reg)) == 0); + CHECK(std::strlen(parakeet_capi_speaker_last_warning(a)) == 0); + + const std::string path = (std::filesystem::temp_directory_path() / "pk_test_fp_registry_model.bin").string(); + CHECK(parakeet_capi_speaker_registry_save(reg, path.c_str()) == 0); + parakeet_speaker_registry* loaded = parakeet_capi_speaker_registry_load(path.c_str()); + CHECK(loaded && fam_a == parakeet_capi_speaker_registry_encoder_family(loaded)); + + // Same encoder: names come out, no warning. + char* j = parakeet_capi_speaker_identify_pcm_json(loaded, a, probe.data(), (int)probe.size(), 16000); + CHECK(j && contains(j, "\"name\":\"speaker_a\"")); + parakeet_capi_free_string(j); + CHECK(std::strlen(parakeet_capi_speaker_last_warning(a)) == 0); + + // Swap, simulated: same weights, same size, other family. Refused before any name. + const std::filesystem::path tmp = std::filesystem::temp_directory_path(); + const std::string renamed = (tmp / "pk_test_fp_renamed.gguf").string(); + const std::string extra = (tmp / "pk_test_fp_extra_key.gguf").string(); + CHECK(rewrite(vd, renamed, "general.name", "some-other-encoder")); + CHECK(rewrite(vd, extra, "general.comment", "requantized elsewhere")); + { + parakeet_ctx* b = parakeet_capi_load(renamed.c_str()); + CHECK(b && parakeet_capi_speaker_dim(b) == parakeet_capi_speaker_dim(a)); + CHECK(b && fam_a != parakeet_capi_speaker_encoder_family(b)); + if (b) { + j = parakeet_capi_speaker_identify_pcm_json(loaded, b, probe.data(), (int)probe.size(), 16000); + CHECK(j == nullptr); + CHECK(contains(parakeet_capi_last_error(b), fam_a)); + CHECK(contains(parakeet_capi_last_error(b), "some-other-encoder")); + // Enrolling with it into the stamped registry is refused as well. + CHECK(parakeet_capi_speaker_enroll(loaded, b, "speaker_c", b0.data(), (int)b0.size(), 16000) != 0); + CHECK(parakeet_capi_speaker_registry_size(loaded) == 2); + parakeet_capi_free(b); + } + } + // Swap, real: another encoder with the same embedding size. + if (alt) { + parakeet_ctx* b = parakeet_capi_load(alt); + CHECK(b != nullptr); + if (b) { + std::printf("encoder B: %s dim %d\n", parakeet_capi_speaker_encoder_family(b), parakeet_capi_speaker_dim(b)); + CHECK(parakeet_capi_speaker_dim(b) == parakeet_capi_speaker_dim(a)); + CHECK(fam_a != parakeet_capi_speaker_encoder_family(b)); + j = parakeet_capi_speaker_identify_pcm_json(loaded, b, probe.data(), (int)probe.size(), 16000); + CHECK(j == nullptr); + CHECK(contains(parakeet_capi_last_error(b), fam_a)); + CHECK(contains(parakeet_capi_last_error(b), parakeet_capi_speaker_encoder_family(b))); + parakeet_capi_free(b); + } else { + std::printf("note: PARAKEET_TEST_VD_ALT_GGUF did not load\n"); + } + } else { + std::printf("note: no PARAKEET_TEST_VD_ALT_GGUF, real swap case skipped\n"); + } + // Same family, other weights: a warning and a result. + { + parakeet_ctx* w = parakeet_capi_load(extra.c_str()); + CHECK(w && fam_a == parakeet_capi_speaker_encoder_family(w)); + CHECK(w && std::strcmp(parakeet_capi_speaker_identity(w), parakeet_capi_speaker_identity(a)) != 0); + if (w) { + j = parakeet_capi_speaker_identify_pcm_json(loaded, w, probe.data(), (int)probe.size(), 16000); + CHECK(j && contains(j, "\"name\":\"speaker_a\"")); + parakeet_capi_free_string(j); + CHECK(contains(parakeet_capi_speaker_last_warning(w), "other weights")); + // Strict mode does not turn a weights difference into an error. + parakeet_capi_speaker_registry_set_strict(loaded, 1); + j = parakeet_capi_speaker_identify_pcm_json(loaded, w, probe.data(), (int)probe.size(), 16000); + CHECK(j != nullptr); + parakeet_capi_free_string(j); + parakeet_capi_speaker_registry_set_strict(loaded, 0); + parakeet_capi_free(w); + } + } + + // A registry with no fingerprint (v1 file, or add_embedding without one): accepted + // with a soft warning; a strict registry is refused. + { + parakeet_speaker_registry* plain = parakeet_capi_speaker_registry_new(); + const int dim = parakeet_capi_speaker_dim(a); + std::vector e1((size_t)dim, 0.0f), e2((size_t)dim, 0.0f); + e1[0] = 1.0f; + e2[1] = 1.0f; + CHECK(parakeet_capi_speaker_registry_add_embedding(plain, "x", e1.data(), dim) == 0); + CHECK(parakeet_capi_speaker_registry_add_embedding(plain, "y", e2.data(), dim) == 0); + const std::string v1 = (tmp / "pk_test_fp_v1.bin").string(); + CHECK(parakeet_capi_speaker_registry_save(plain, v1.c_str()) == 0); + { std::FILE* f = std::fopen(v1.c_str(), "rb"); unsigned char h[8] = {0}; + CHECK(f && std::fread(h, 1, 8, f) == 8 && h[4] == 1); + if (f) std::fclose(f); } + parakeet_speaker_registry* old = parakeet_capi_speaker_registry_load(v1.c_str()); + CHECK(old && std::strlen(parakeet_capi_speaker_registry_encoder_family(old)) == 0); + j = parakeet_capi_speaker_identify_pcm_json(old, a, probe.data(), (int)probe.size(), 16000); + CHECK(j != nullptr); + parakeet_capi_free_string(j); + CHECK(contains(parakeet_capi_speaker_last_warning(a), "no encoder fingerprint")); + parakeet_capi_speaker_registry_set_strict(old, 1); + j = parakeet_capi_speaker_identify_pcm_json(old, a, probe.data(), (int)probe.size(), 16000); + CHECK(j == nullptr && contains(parakeet_capi_last_error(a), "strict")); + // Calling enroll with the encoder on a registry that already has unstamped speakers is refused. + CHECK(parakeet_capi_speaker_enroll(old, a, "z", a0.data(), (int)a0.size(), 16000) != 0); + // add_embedding_fp with the right fingerprint into an empty registry then works end to end. + parakeet_speaker_registry* fresh = parakeet_capi_speaker_registry_new(); + CHECK(parakeet_capi_speaker_registry_add_embedding_fp(fresh, "x", e1.data(), dim, fam_a.c_str(), + parakeet_capi_speaker_identity(a)) == 0); + CHECK(parakeet_capi_speaker_registry_add_embedding_fp(fresh, "y", e2.data(), dim, fam_a.c_str(), + parakeet_capi_speaker_identity(a)) == 0); + parakeet_capi_speaker_registry_set_strict(fresh, 1); + j = parakeet_capi_speaker_identify_pcm_json(fresh, a, probe.data(), (int)probe.size(), 16000); + CHECK(j != nullptr); + parakeet_capi_free_string(j); + parakeet_capi_speaker_registry_free(fresh); + parakeet_capi_speaker_registry_free(old); + parakeet_capi_speaker_registry_free(plain); + std::filesystem::remove(v1); + } + + // The same encoder inside a bundle: same family, same weights identity (the bundle's + // recorded source hash), so a registry enrolled with the standalone file is used with + // the bundle component with no warning, and the other way round. + if (const char* bundle = std::getenv("PARAKEET_TEST_VD_BUNDLE")) { + const char* comp = std::getenv("PARAKEET_TEST_VD_BUNDLE_COMPONENT"); + parakeet_ctx* c = parakeet_capi_load_component(bundle, comp ? comp : "voice"); + CHECK(c != nullptr); + if (c) { + CHECK(fam_a == parakeet_capi_speaker_encoder_family(c)); + CHECK(std::strcmp(parakeet_capi_speaker_identity(a), parakeet_capi_speaker_identity(c)) == 0); + char* jb = parakeet_capi_speaker_identify_pcm_json(loaded, c, probe.data(), (int)probe.size(), 16000); + CHECK(jb && contains(jb, "\"name\":\"speaker_a\"")); + parakeet_capi_free_string(jb); + CHECK(std::strlen(parakeet_capi_speaker_last_warning(c)) == 0); + // Enrolled with the bundle component, used with the standalone file. + parakeet_speaker_registry* rb = parakeet_capi_speaker_registry_new(); + CHECK(parakeet_capi_speaker_enroll(rb, c, "speaker_a", a0.data(), (int)a0.size(), 16000) == 0); + CHECK(std::strcmp(parakeet_capi_speaker_registry_encoder_weights(rb), + parakeet_capi_speaker_registry_encoder_weights(loaded)) == 0); + jb = parakeet_capi_speaker_identify_pcm_json(rb, a, probe.data(), (int)probe.size(), 16000); + CHECK(jb != nullptr); + parakeet_capi_free_string(jb); + CHECK(std::strlen(parakeet_capi_speaker_last_warning(a)) == 0); + parakeet_capi_speaker_registry_free(rb); + parakeet_capi_free(c); + } + } else { + std::printf("note: no PARAKEET_TEST_VD_BUNDLE, bundle identity case skipped\n"); + } + + // The same check guards the diarization and scene entry points. + if (diar_path) { + parakeet_ctx* diar = parakeet_capi_load(diar_path); + parakeet_ctx* b = parakeet_capi_load(renamed.c_str()); + CHECK(diar && b); + if (diar && b) { + char* d = parakeet_capi_diarize_named_pcm_json(diar, b, loaded, wav.samples.data(), + (int)wav.samples.size(), 16000, 0, 0); + CHECK(d == nullptr && contains(parakeet_capi_last_error(b), fam_a)); + parakeet_capi_free_string(d); + parakeet_scene_opts o; + parakeet_capi_scene_opts_default(&o); + parakeet_scene_stream* s = parakeet_capi_scene_stream_begin_speaker(nullptr, diar, nullptr, b, loaded, &o); + CHECK(s == nullptr && contains(parakeet_capi_last_error(b), fam_a)); + parakeet_capi_scene_stream_free(s); + // And the right encoder still names the speakers. + d = parakeet_capi_diarize_named_pcm_json(diar, a, loaded, wav.samples.data(), + (int)wav.samples.size(), 16000, 0, 0); + CHECK(d && contains(d, "\"name\":\"speaker_a\"")); + parakeet_capi_free_string(d); + } + parakeet_capi_free(b); + parakeet_capi_free(diar); + } + + parakeet_capi_speaker_registry_free(loaded); + parakeet_capi_speaker_registry_free(reg); + parakeet_capi_free(a); + std::filesystem::remove(path); + std::filesystem::remove(renamed); + std::filesystem::remove(extra); + if (failures) { std::fprintf(stderr, "%d failure(s)\n", failures); return 1; } + std::printf("test_speaker_fingerprint: PASS\n"); + return 0; +} diff --git a/tests/test_speaker_registry.cpp b/tests/test_speaker_registry.cpp index 7919a0b..bbf8d4b 100644 --- a/tests/test_speaker_registry.cpp +++ b/tests/test_speaker_registry.cpp @@ -2,6 +2,8 @@ #include "speaker_registry.hpp" #include +#include +#include #include #include #include @@ -206,6 +208,140 @@ static void test_non_finite() { CHECK(empty.size() == 0 && empty.dim() == 0); } + +static bool throws_deser(const std::string& blob) { + try { SpeakerRegistry::deserialize(blob); } catch (const std::runtime_error&) { return true; } + return false; +} + +// ---- encoder fingerprint --------------------------------------------------- + +static const EncoderFingerprint kFpA{"voicedetect:wespeaker_resnet34:wespeaker:256", "sha256:aaaa"}; +static const EncoderFingerprint kFpA16{"voicedetect:wespeaker_resnet34:wespeaker:256", "sha256:bbbb"}; +static const EncoderFingerprint kFpB{"voicedetect:ecapa_tdnn:ecapa:192", "sha256:cccc"}; +static const EncoderFingerprint kFpC{"voicedetect:campplus:campplus:192", "sha256:dddd"}; + +static void put_u32(std::string& s, uint32_t v) { s.append(reinterpret_cast(&v), 4); } + +static void test_fp_enroll_stamps_and_roundtrips() { + SpeakerRegistry r; + CHECK(r.fingerprint().empty()); + CHECK(r.enroll("alice", unit(4, 0), kFpA).status == FingerprintStatus::Match); + CHECK(r.fingerprint().family == kFpA.family && r.fingerprint().weights == kFpA.weights); + const SpeakerRegistry back = SpeakerRegistry::deserialize(r.serialize()); + CHECK(back.fingerprint().family == kFpA.family && back.fingerprint().weights == kFpA.weights); + CHECK(back.size() == 1 && back.dim() == 4); + CHECK(back.identify(unit(4, 0), 0.5f, 0.05f).name == "alice"); + // A fingerprint with an empty weights hash is kept as given. + SpeakerRegistry fam; + fam.enroll("a", unit(4, 0), EncoderFingerprint{"fam", ""}); + CHECK(SpeakerRegistry::deserialize(fam.serialize()).fingerprint().family == "fam"); +} + +static void test_fp_enroll_rules() { + SpeakerRegistry r; + r.enroll("alice", unit(4, 0), kFpA); + // Same family, other weights: allowed, reported, registry keeps its own hash. + FingerprintVerdict v = r.enroll("bob", unit(4, 1), kFpA16); + CHECK(v.status == FingerprintStatus::WeightsDiffer && !v.message.empty()); + CHECK(r.fingerprint().weights == kFpA.weights && r.size() == 2); + // Other family: refused, nothing added. + std::string msg; + try { r.enroll("cy", unit(4, 2), kFpC); } catch (const std::invalid_argument& e) { msg = e.what(); } + CHECK(msg.find(kFpA.family) != std::string::npos && msg.find(kFpC.family) != std::string::npos); + CHECK(r.size() == 2); + // No fingerprint into a fingerprinted registry: refused. + msg.clear(); + try { r.enroll("dee", unit(4, 3)); } catch (const std::invalid_argument& e) { msg = e.what(); } + CHECK(!msg.empty() && r.size() == 2); + // A fingerprint into a non-empty unfingerprinted registry: refused, never stamped silently. + SpeakerRegistry old; + old.enroll("alice", unit(4, 0)); + msg.clear(); + try { old.enroll("bob", unit(4, 1), kFpA); } catch (const std::invalid_argument& e) { msg = e.what(); } + CHECK(!msg.empty() && old.size() == 1 && old.fingerprint().empty()); + // An empty fingerprint behaves as before. + SpeakerRegistry plain; + plain.enroll("alice", unit(4, 0), EncoderFingerprint{}); + CHECK(plain.fingerprint().empty()); +} + +static void test_fp_check() { + // Match. + CHECK(check_fingerprint(kFpA, kFpA, false).status == FingerprintStatus::Match); + CHECK(check_fingerprint(kFpA, kFpA, true).status == FingerprintStatus::Match); + // Same family, other weights: a warning, even in strict mode. + FingerprintVerdict w = check_fingerprint(kFpA, kFpA16, true); + CHECK(w.status == FingerprintStatus::WeightsDiffer && !w.is_error() && !w.message.empty()); + // Same dimension, other family: a hard error that names both families. + FingerprintVerdict f = check_fingerprint(kFpB, kFpC, false); + CHECK(f.status == FingerprintStatus::FamilyDiffer && f.is_error()); + CHECK(f.message.find(kFpB.family) != std::string::npos); + CHECK(f.message.find(kFpC.family) != std::string::npos); + // No fingerprint: soft warning, or an error in strict mode. + FingerprintVerdict u = check_fingerprint(EncoderFingerprint{}, kFpA, false); + CHECK(u.status == FingerprintStatus::Unfingerprinted && !u.is_error() && !u.message.empty()); + FingerprintVerdict s = check_fingerprint(EncoderFingerprint{}, kFpA, true); + CHECK(s.status == FingerprintStatus::Required && s.is_error() && !s.message.empty()); + // Family only in the registry: weights are not compared. + CHECK(check_fingerprint(EncoderFingerprint{kFpA.family, ""}, kFpA16, true).status == FingerprintStatus::Match); + // An encoder that cannot name its family is not checked. + CHECK(check_fingerprint(kFpA, EncoderFingerprint{}, true).status == FingerprintStatus::Match); +} + +static void test_fp_registry_check() { + SpeakerRegistry r; + r.enroll("a", unit(192, 0), kFpB); + FingerprintVerdict v = check_registry_for_encoder(r, 256, kFpA, false); + CHECK(v.status == FingerprintStatus::DimDiffer && v.is_error()); + CHECK(v.message.find("192") != std::string::npos && v.message.find("256") != std::string::npos); + CHECK(check_registry_for_encoder(r, 192, kFpC, false).status == FingerprintStatus::FamilyDiffer); + CHECK(check_registry_for_encoder(r, 192, kFpB, true).status == FingerprintStatus::Match); + SpeakerRegistry empty; // nothing enrolled: nothing to mix up, even in strict mode + CHECK(check_registry_for_encoder(empty, 192, kFpB, true).status == FingerprintStatus::Match); +} + +static void test_fp_v1_compat() { + // v1 bytes exactly as the previous release wrote them. + std::string v1("PKSR", 4); + put_u32(v1, 1); + put_u32(v1, 2); // dim + put_u32(v1, 1); // n + put_u32(v1, 5); + v1 += "alice"; + put_u32(v1, 1); // count + const float sum[2] = {1.0f, 0.0f}; + v1.append(reinterpret_cast(sum), sizeof(sum)); + SpeakerRegistry r = SpeakerRegistry::deserialize(v1); + CHECK(r.size() == 1 && r.dim() == 2 && r.fingerprint().empty()); + CHECK(r.identify(unit(2, 0), 0.5f, 0.05f).name == "alice"); + // An unfingerprinted registry is still written as v1, byte for byte. + CHECK(r.serialize() == v1); + // A fingerprinted one is v2, and a reader that only knows v1 refuses it. + r.set_fingerprint(kFpA); + const std::string v2 = r.serialize(); + CHECK(v2.size() > v1.size()); + uint32_t ver = 0; + std::memcpy(&ver, v2.data() + 4, 4); + CHECK(ver == 2); + std::string patched = v2; + const uint32_t three = 3; + std::memcpy(&patched[4], &three, 4); + CHECK(throws_deser(patched)); // unknown version + std::string as_v1 = v2; + const uint32_t one = 1; + std::memcpy(&as_v1[4], &one, 4); + CHECK(throws_deser(as_v1)); // a v1 parser meets the extra bytes: refused, no crash + // Truncation inside the fingerprint block. + CHECK(throws_deser(v2.substr(0, 4 + 4 + 4 + 4 + 2))); + CHECK(throws_deser(v2.substr(0, 4 + 4 + 4 + 4 + 4 + 3))); + // An absurd fingerprint length is refused. + std::string big = v2; + const uint32_t huge = 0x7fffffff; + std::memcpy(&big[16], &huge, 4); + CHECK(throws_deser(big)); +} + int main() { test_enroll_and_identify(); test_centroid_averages_enrollments(); @@ -217,6 +353,11 @@ int main() { test_serialize_roundtrip(); test_deserialize_corrupt(); test_non_finite(); + test_fp_enroll_stamps_and_roundtrips(); + test_fp_enroll_rules(); + test_fp_check(); + test_fp_registry_check(); + test_fp_v1_compat(); if (failures) { std::fprintf(stderr, "%d failure(s)\n", failures); return 1; } std::printf("test_speaker_registry: PASS\n"); return 0;