Skip to content

Search words: keep combining marks, leave fact dates out, and treat long words alike on both providers #2597

Open

Description

@pournasserian

Part of #2430 (F-SRC). Found by the review of #2566. Three ways the words search indexes differ from what a person searches for.

  • Combining marks split words. SearchTerms matches [\p{L}\p{N}]+, so a mark (\p{M}) is a separator: "हिन्दी" becomes "ह", "न" and "द". A Hindi query then turns into three one-letter prefix terms that match almost any Hindi entry. Bengali, Tamil and Arabic with vowel marks behave the same, and Latin typed in decomposed form loses its accent.
  • Fact dates are indexed. A fact's validFrom, validUntil and lastVerified are strings in the fields, so they become words such as "2026 10 19t05". A short number such as "10" or "2026" then matches every entity with a dated fact. The search issue asks for a fact's value and source only.
  • Long words differ between providers. PostgreSQL leaves a word over 2,047 bytes out of a tsvector, while SQLite FTS5 keeps it. This only matters for long runs without separators, such as 700 or more Chinese characters with no punctuation.

Scope

  • Words keep their combining marks, and text is normalised to NFC before it is split, for both the indexed text and the query.
  • A fact's value and source are indexed, and its dates and owner are not.
  • A word longer than both providers index is cut, or left out, the same way on both.
  • The indexing of existing entries (Index existing entries for search without depending on the database locale #2589) uses the same rules.

Out of scope

  • Stemming, synonyms and language-specific tokenizers.

Acceptance

  • "हिन्दी" is one word in the indexed text and in a query, and finds the entry that contains it on SQLite and PostgreSQL.
  • An accent typed in decomposed form finds the entry written with the precomposed one.
  • An entity with a dated fact is not found by "2026" unless its value or source contains it.
  • An entry with a 3,000-character word is found the same way on both providers.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:infraPersistence, host, CI and buildtype:bugDefectv2FluentCMS v2 (v2 branch)

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions