Extraction: default relation vocabulary and no early stop in relation extraction - #321
Conversation
…ocabulary Entity extraction has always shipped a default type list, so the LLM is told what kinds of things to look for. Relations shipped nothing: with no ontology the prompt said "use a descriptive label in UPPER_SNAKE_CASE", and the model invented a fresh name for almost every edge. Measured on an 11-document corpus: 447 distinct relation labels against 30 in the gold annotation, 68.2% of them used exactly once, and only 17.3% of edges carrying a label the gold data also uses. Two of gold's most common predicates were emitted zero times - `contains` (123 triples) and `authored` (54). Supplying any fixed list doubles exact triple F1 (0.0725 -> 0.1490, +2.06x). A list written without reference to the gold vocabulary scored as well as the gold vocabulary itself (0.1490 vs 0.1456), so the gain comes from being consistent, not from guessing the right words. That is exactly what makes a shipped default worth having. The list is domain-neutral and pairs with DEFAULT_ENTITY_TYPES. Like entity_types it is guidance, not a filter: it steers the prompt, and a relation labelled outside it is still kept. The hard-filtering path remains Ontology.relations, which prunes non-conforming edges in IngestionPipeline._prune. A declared ontology still wins. relation_types=[] stays meaningful and restores the previous open-vocabulary behaviour verbatim, for anyone who wants the old free-for-all.
…hort The relationship step stopped early because it decided it had said enough. Measured (RESULTS.md P5.6): "Extract ALL factual connections" is not enough on its own - the model treats the task as a summary, returns roughly 12 relations per chunk and stops while using 2.5k of a 16k reply budget. Doubling the input text grew the reply by 1.3%, which is the signature of a self-imposed ceiling rather than a token limit. The prompt now states explicitly that this is an exhaustive extraction task and not a summary, that there is no maximum, that a dense paragraph often yields 20 or more relationships, and that a long list is correct rather than a mistake. The entity-removal instruction is also condensed. The specific examples it listed (+=, ->, sh, cd, dt) are now enforced in code by is_valid_entity_name, so spelling them out in the prompt spent tokens re-stating a rule that no longer depends on the model complying. A comment is added above the prompt recording that removing the entity verification job entirely was tried and REVERTED: without it the LLM emitted 813 entities instead of 719 and entity precision fell 0.645 -> 0.551 (RESULTS.md P2.12). That instruction is doing real work and should not be deleted again without re-running the measurement.
…ype prompt instruction
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
🟡 Changes recommended
The updated prompt text can be interpreted as allowing relationship endpoints from the pre-extracted entity list rather than the verified output entities, which can produce dangling relationships that get dropped downstream.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR improves step-2 (LLM verification + relationship extraction) consistency and completeness by introducing a default relation vocabulary (analogous to default entity types) and strengthening the prompt to discourage early termination during relation extraction.
Changes:
- Add
DEFAULT_RELATION_TYPESand thread it throughGraphExtractionvia a newrelation_typesparameter (with[]restoring open-vocabulary behavior). - Update the step-2 prompt to emphasize exhaustive relationship extraction (no “summary-mode” early stop).
- Add tests validating default relation-type behavior and prompt rendering.
File summaries
| File | Description |
|---|---|
| graphrag_sdk/src/graphrag_sdk/ingestion/extraction_strategies/graph_extraction.py | Adds a default relation vocabulary, plumbs it into prompt construction, and updates prompt instructions to reduce early stopping. |
| graphrag_sdk/tests/test_graph_extraction.py | Adds coverage for default relation-type behavior, immutability per instance, and prompt inclusion/open-vocabulary behavior. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| "### Relationships\n" | ||
| "- Extract ALL factual connections stated or implied in the text.\n" | ||
| "- source and target must be entity names from the verified entity list.\n" | ||
| "- source and target must be entity names from the entity list above.\n" |
Extraction — a default relation vocabulary, and no more early stop
Part of FalkorDB/research#88. Stacked on #320. Both changes are to the one step-2 prompt (entities + relations per chunk).
1.
DEFAULT_RELATION_TYPESProblem: entities always had a default ontology; relations had none — the prompt said "invent a descriptive label", so the same relation got a different name in every chunk. 447 distinct relation labels on the benchmark, only 17 % matching the answer key's 30; exact triple F1 0.065.
Fix: a 31-label default vocabulary offered to the LLM, the exact counterpart of
entity_types— guidance, not a filter (labels outside the list are kept;relation_types=[]restores open vocabulary; a declared ontology overrides it). Vocabulary 447 → ~94, exact triple F1 2×; better on 5 of 5 unseen domains.2. The relation extractor no longer stops early
Problem: the model treats the task as a summary — ~12 relations per chunk, then stops, using 2.5k of a 16k reply budget. Giving it twice the text grew the reply 1.3 %.
Fix: three prompt sentences ("exhaustive extraction, not a summary; there is no maximum; never end the list early"). +15 % relations for +4 % ingest time, and the best exact score of every variant tried — a second "what did you miss?" call gave more relations at 4.7× the time and a worse score, and is not in this PR.
Verification
Full suite: 1179 passed, 41 skipped; ruff clean.