OPENNLP-1887: Lexical expansion over the knowledge base seam: synonyms, hypernyms, optional hyponyms#1167
Draft
krickert wants to merge 10 commits into
Draft
OPENNLP-1887: Lexical expansion over the knowledge base seam: synonyms, hypernyms, optional hyponyms#1167krickert wants to merge 10 commits into
krickert wants to merge 10 commits into
Conversation
krickert
changed the title
krickert
changed the title
krickert
force-pushed
the
wordnet-expansion
branch
6 times, most recently
from
July 17, 2026 20:45
5db6a8a to
917e4be
Compare
krickert
force-pushed
the
wordnet-expansion
branch
from
July 18, 2026 08:00
917e4be to
1b90059
Compare
…s and a Morphy lemmatizer Adds the LexicalKnowledgeBase contract in opennlp.tools.wordnet and the opennlp-wordnet module implementing it twice: WnLmfReader for WN-LMF XML and WndbReader for the legacy WNDB database files, with reader-equivalence coverage over miniature fixtures of both formats. The WN-LMF reader skips DOCTYPE declarations unresolved with DTD support off, so Open English WordNet releases parse unmodified while entity expansion stays closed; the WNDB fixtures are pinned to LF so their embedded byte offsets survive Windows checkout. The Morphy lemmatizer resolves inflected forms through suffix rules and the format's exception lists. Null arguments fail loudly with IllegalArgumentException, malformed data raises the checked InvalidFormatException, and the public seam carries no brand name: WordNet stays in the names of the classes that actually read WordNet formats.
…nsion over the lexical knowledge base Expands a term into the synonyms sharing its synsets, the lemmas of its hypernym ancestors up to a configured depth (following both the direct and the instance relation), and optionally its direct hyponyms. Each expansion carries a deterministic heuristic weight: sense rank and every relation step multiply configurable decays, so consumers can discount looser expansions instead of treating them as the original term. Results exclude the input, deduplicate case-insensitively keeping the highest weight, and order stably by weight, kind, and term. Inflected input works through an optional Lemmatizer invoked with the WordNetPos name as the tag, which the Morphy lemmatizer understands: dogs expands through dog, mice through the exception list to mouse. Hypernym walks are visited-checked so malformed cyclic data terminates. Tests cover the graph behavior on a hand-built lexicon (sense ranking, decay arithmetic, dedup, cycle termination, validation) and run the whole stack over the miniature WN-LMF and WNDB fixtures, asserting both readers produce identical expansions.
… definition A blank check under the toolkit's whitespace definition, which unlike String.isBlank covers the no-break spaces, so annotators validating labels and identifiers share one predicate instead of each carrying a private copy. Reads whole code points; tests pin the no-break and figure spaces, the empty string, and a supplementary-plane letter.
…yping SynsetSimilarity scores noun synset pairs with the path, Wu-Palmer, and Leacock-Chodorow measures over the knowledge base seam. HypernymTyper labels a word by walking its senses' hypernym and instance-hypernym chains to the nearest caller-registered anchor concept, so a knowledge base can type names as person, organization, or location without a model. Blank checks follow the toolkit whitespace definition.
krickert
force-pushed
the
wordnet-expansion
branch
from
July 19, 2026 11:41
1b90059 to
f4cc90c
Compare
…examples Add the lexical expansion section to docbkx/wordnet.xml and LexicalExpansionUsageExampleTest asserting the expansion values the chapter prints; carry the WordNet usage example alongside it.
…ryLemmatizer Add PoliMorfDictionaryReader, adapting a surfaceForm/lemma/tag table (the layout PoliMorf publishes under BSD 2-Clause and morfologik text export produces) into the existing DictionaryLemmatizer: re-order columns, lower-case forms to match the case-insensitive lookup, and merge alternative lemmas per form and tag. Pure JVM, no carrot2 dependency; dictionary data stays caller-supplied. Manual section and mirror-tested example.
Read the morfologik CFSA2 automaton format (BSD) with no dependency on that library and enumerate the accepted byte sequences: header, flags and label-table parsing, v-coded targets, next-arc and target-next resolution, and terminating-node stop. Validated against a ground-truth automaton generated by morfologik's own serializer. This is the finite-state layer under bring-your-own morfologik-format dictionaries; decoding sequences into form/lemma/tag entries follows.
…izer Add MorfologikDictionaryReader: enumerate a CFSA2 automaton, split each sequence on the separator into form/encodedBase/tag, and decode the base form for all four morfologik encoders (NONE, SUFFIX, PREFIX, INFIX; control bytes offset by 'A'). Reads metadata from an explicit charset/encoder/separator or from a .info stream. Clean-room, no carrot2 dependency; dictionary data caller-supplied. Decode cases and the end-to-end CFSA2 fixture were generated by morfologik's own encoder and serializer.
Add FSA5Reader (the older fixed-goto morfologik automaton format) and an FsaSequenceReader interface whose read() dispatches on the version byte, so the morfologik dictionary reader now accepts both FSA5 and CFSA2. Validated against an FSA5 ground-truth automaton and an end-to-end FSA5 dictionary from morfologik's own serializer. Manual section documents reading morfologik dictionaries.
Walk the automaton with a single growable byte buffer, pushing each arc label on descent and popping on return, so only accepted sequences allocate. The prior code copied the whole prefix at every arc, internal ones included. Uses plain byte[] and int offsets, no ByteBuffer, keeping the reader allocation-light on large dictionaries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
STACKED on #1155 (wordnet-api): this branch builds on that PR's LexicalKnowledgeBase seam, and since that head lives on a fork it cannot be this PR's base ref. Until #1155 merges, the diff here shows its commits too; only the commits from 1404a50 onward are this PR. One of them (082592d) is a carried copy of the OPENNLP-1888 StringUtil.isBlank utility this code uses; it drops out automatically once that lands. After #1155 lands, the branch rebases onto main and the diff collapses to this change alone.
Adds LexicalExpander to the opennlp-wordnet module: given a term, it produces weighted related terms from a LexicalKnowledgeBase, the synonyms sharing its synsets, the lemmas of hypernym ancestors up to a configured depth (following both the direct and the instance relation, visited-checked so cyclic data terminates), and optionally direct hyponyms. Each expansion carries a deterministic heuristic weight (sense-rank decay times depth decay, both configurable), results are deduplicated case-insensitively keeping the highest weight, the input term is never returned, and ordering is stable. An optional Lemmatizer fallback expands inflected input; the Morphy lemmatizer of this module plugs in directly, so dogs expands via dog and mice via mouse.
Tests cover the behavior over a hand-built graph with a controlled shape (sense ranking, depth decay, dedupe, cycle termination, hyponym opt-in, configuration validation) plus integration over the miniature WN-LMF and WNDB fixtures asserting both readers expand identically.