parser fixes - #16
Open
kirkhess wants to merge 2 commits into
Open
Conversation
Verified by re-parsing an identical set of stored pages before and after, so
every difference below is the parser's doing rather than a page change.
The last subfield in each cell was dropped. Older field pages list subfields as
bare text separated by <br/>, and parse-subfields walked
br/preceding-sibling::text()[1] -- one entry per <br/>. The last entry in a cell
has no <br/> after it and was never seen, losing two subfields per page, one per
column: 017$i and $8, 222$b and $8, 242$n, 542$k and $8.
Labels wrapped in <span class="changed"> were lost. LC marks anything altered in
the current update that way, to print it red. Reading only the list item's first
text node made such a label invisible, so 540$f came out keyed "f -" with an
empty label. This is the fix that matters most for keeping up to date: whatever
is new in any given update is exactly what carries the marker.
Indicator codes not wrapped in <span> were lost. parse-indicators read $td/span,
but on 046 and 588 LC leaves the first indicator's codes as bare text -- and 588
has a stray unopened </span> besides -- so both fields came out with an empty
code list. 046's first indicator is Type of entity, distinguishing Work,
Expression and Manifestation, so the gap was not cosmetic.
Labels were truncated at the first " - " or " (". Both occur inside real labels:
034$d-$g and $j-$n all collapsed to "Coordinates", "Declination" or "Right
ascension", four subfields at a time sharing one label; 041$r lost
"(non-textual)"; 880's three range subfields came out with empty labels.
All four are handled by two new helpers. ms:split-on-br groups a cell's nodes on
<br/> boundaries and takes each run's string value, which reads bare text,
<span>-wrapped text and a trailing entry with no <br/> identically.
ms:strip-repeat-marker removes a trailing (R)/(NR) by pattern instead of cutting
at a delimiter, matching only R and NR so a label genuinely ending in
parentheses survives, and tolerating LC's missing closing paren in 018$a
"Copyright article-fee code (NR".
LC keeps a retired field's page online indefinitely, complete with its subfield and indicator tables, so nothing in the field's own description says it has been withdrawn and the scrape picked six of them up as though current. The breadcrumb distinguishes them. A current field is filed under its group -- Bibliographic >> 01X-09X >> 046 -- while a retired or United-States-local one is filed under Appendix H, Local Data Elements. ms:is-appendix-field tests for that, which excludes exactly 261, 262, 400, 410, 411 and 440 from the bibliographic format and nothing at all from authority or holdings. Match the appendix rather than the group: 440's breadcrumb also links to Community Information 01X-08X, a different format altogether, so a does-it-link-to-its-own-group test would misfire there. The leader has no breadcrumb and is never excluded. Silently emitting a smaller format would be worse than emitting a wrong one, so ms:excluded-fields reports what was dropped and run-parser.xq writes it to excluded_fields.json alongside the schemas, with the page each field is filed under. Also ignores basex/ and build/, since the README tells you to unzip BaseX into the working directory.
Collaborator
|
How did you test the fix? From my network the download does not work anymore because LoC introduced anti-bot-methods. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
initial commit