Skip to content

A number in accounting parentheses is read as negative, so "(1,234)" is -1234 instead of no value - #1422

Open
rudycelekli wants to merge 1 commit into
deeplethe:devfrom
rudycelekli:extract-accounting-negative
Open

rudycelekli wants to merge 1 commit into
deeplethe:devfrom
rudycelekli:extract-accounting-negative

Conversation

@rudycelekli

Copy link
Copy Markdown
Contributor

Why

Financial statements write a negative amount in parentheses: (1,234), $(1.2) billion, (12.5%). The SEC filings the HTML table handling was measured on (the NVIDIA report cited in html.rs) are full of them. Net losses, cash outflows and declines all look like this.

When extraction copies such a cell as the value of a number attribute, normalize_attr_value cannot read it. The plain parse fails on the parentheses. parse_leading_quantity finds no digit at the start. The value is dropped, so the typed graph keeps every positive line of a table and loses every negative one.

What changes

The lenient quantity scan (scan_quantity with strict = false, used by normalize_attr_value and attribute_value) rewrites an accounting negative before reading it. The value must be a number in parentheses, optionally preceded by a currency, with the rest after the closing parenthesis kept: (1,234) → -1,234, $(1.2) billion → $-1.2 billion, (350)万元 → -350万元.

Anything else in parentheses is left alone, such as (see note 4), or Q3 (12) where the text before the parenthesis is not a currency. The strict scan that decides whether an object name is a value (literal_object) is unchanged, so a named (1984) is still a thing.

How it was checked

A new test, an_accounting_negative_is_read_as_negative:

  • (1,234) is -1234.
  • $(1.2) billion is -1200000000.
  • (12.5%) is -12.5.
  • (350)万元 under 万元 is -350.
  • (see note 4) and Q3 (12) still read as no value.
  • literal_object("(1984)", true) is still false.

Results:

  • cargo test -p utopia-extract --lib: 112 passed.
  • With the fix reverted, the test fails with left: None, right: Some(Number(-1234.0)).
  • cargo fmt --all --check and cargo clippy -p utopia-extract --all-targets -- -D warnings pass.

Before review

  • Every commit is signed off (git commit -s)
  • cargo fmt --all --check, cargo clippy and cargo test pass for the touched crate

…is -1234 instead of no value

Signed-off-by: Rudy Celekli <rudy@gradiahq.com>

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea is right, but the text after ) isn't checked: "(1) 1,200 employees" now reads -1 and "(2024) 500" reads -2024, where dev read no value. Please accept only an empty tail or a unit/magnitude/% after the parenthesis, and add those two cases as negative tests. Also, with #1423, "HK$(300) million" still reads nothing, so one of the two needs to accept a - after the currency code. Please rebase after #1416.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants