You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Search words: keep combining marks, leave fact dates out, and treat long words alike on both providers #2597
Part of #2430 (F-SRC). Found by the review of #2566. Three ways the words search indexes differ from what a person searches for.
Combining marks split words.SearchTerms matches [\p{L}\p{N}]+, so a mark (\p{M}) is a separator: "हिन्दी" becomes "ह", "न" and "द". A Hindi query then turns into three one-letter prefix terms that match almost any Hindi entry. Bengali, Tamil and Arabic with vowel marks behave the same, and Latin typed in decomposed form loses its accent.
Fact dates are indexed. A fact's validFrom, validUntil and lastVerified are strings in the fields, so they become words such as "2026 10 19t05". A short number such as "10" or "2026" then matches every entity with a dated fact. The search issue asks for a fact's value and source only.
Long words differ between providers. PostgreSQL leaves a word over 2,047 bytes out of a tsvector, while SQLite FTS5 keeps it. This only matters for long runs without separators, such as 700 or more Chinese characters with no punctuation.
Scope
Words keep their combining marks, and text is normalised to NFC before it is split, for both the indexed text and the query.
A fact's value and source are indexed, and its dates and owner are not.
A word longer than both providers index is cut, or left out, the same way on both.
Activity