Apply shared term normalization to search terms and alias index (#346)

[Bernhard Merkle (bmerkle)]

Part of #346 (items 1 and 2). Item 3, the different term sets the two fuzzy-index update paths collect, will be a separate PR.

#325 made the memory and SQLite term and property indexes share normalize_term (NFC, lowercase, trim, collapse whitespace). Search terms and query matching still only lowercased, and the related-terms alias index stored and matched keys verbatim, so they could disagree with the index keys.

Search side

• QueryCompiler.validate_and_prepare_term and searchlib._parse_search_term normalize term text with normalize_term.
• query.get_matching_term_for_text (used by match_entity_name_or_type → boost_entities) compares normalized text. For entity "Space Needle" and search term "space needle", the entity now gets its boost.

Alias index (memory and SQLite)

• TermToRelatedTermsMap and SqliteRelatedTermsAliases normalize keys and aliases on add_related_term, lookup_term, remove_term, deserialize (and SQLite set_related_terms). An alias added for "Space Needle" is now found for " Space\tNeedle"; before, any alias key with uppercase letters was never found, since search terms were lowercased before the lookup.
• Aliases are now stored normalized, so "ML" comes back as "ml" (one expectation in tests/test_sqlite_indexes.py updated). Search re-normalizes related terms after resolving them anyway, so results don't change.
• Memory aliases keep their weight; lookup_term still returns copies (fix: stop queries from mutating the in-memory alias index #349).

Compatibility: SQLite aliases stored by earlier versions keep their old keys and may no longer match; the CHANGELOG entry (under 0.6.0 breaking changes, next to the #322 entries) says to re-ingest such databases. Memory/JSON data is re-normalized on load.

Tests: 13 new tests in tests/test_term_normalization.py for search-term parsing, the query matchers and the alias index, run against both backends. 12 of them fail without the fix.

Testing: make locally: format, ruff and pyright (3.12 and 3.15) clean, build OK; 920 passed. 7 tests fail locally on my machine because of a broken jiter native module (and an MCP sampling test that depends on it); they fail identically on main, so CI is the reference for those.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Failed normalization during memory alias deserialization can erase existing aliases.

1 open issue

• Preserve aliases when deserialize encounters malformed term text · New

What changed in this PR

Extends shared term normalization to search and alias indexes, aligning matching behavior across memory and SQLite.

Changes:

• Normalize search terms, query comparisons, and alias keys and values.
• Add regression coverage across both storage backends.
• Document compatibility implications for existing SQLite aliases.

File	Description
tests/​test_term_normalization.py	Adds search and alias normalization tests.
tests/​test_sqlite_indexes.py	Expects normalized alias text.
src/​typeagent/​storage/​sqlite/​related_terms_index.py	Normalizes alias reads, writes, and deserialization.
src/​typeagent/​storage/​memory/​related_terms_index.py	Normalizes alias keys and values while preserving copies.
src/​typeagent/​knowpro/​searchlib.py	Normalizes parsed search terms.
src/​typeagent/​knowpro/​search.py	Normalizes terms during query preparation.
src/​typeagent/​knowpro/​query.py	Uses normalized text for matching.
CHANGELOG.md	Documents normalization and SQLite compatibility changes.

_{Review effort: Balanced}

[Bernhard Merkle (bmerkle)]

Kevin Turcios (@KRRT7) could you take a look? This is the #346 follow-up for search terms and the alias index; it builds on your #349 (alias map copies) and #350 (idempotent normalize_term).