Telugu TN: Cardinal Semiotic Class by nharidas · Pull Request #447 · NVIDIA/NeMo-text-processing

nharidas · 2026-07-07T06:49:10Z

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Before your PR is "Ready for review"

Pre checks:

PR Type:

New Feature
Bugfix
Documentation
Test

If you haven't finished some of the above items you can still open "Draft" PR.

github-actions · 2026-07-22T02:17:23Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

folivoramanh

make sure both pytest and sparrowhawk test pass, and fix the problem of blank space

folivoramanh · 2026-08-17T02:12:32Z

        }
      }
    }
+        stage('L0: Create TE TN/ITN Grammars') {


add only stage('L0: Codeswitched HI/EN ITN grammars') {
steps {
sh 'CUDA_VISIBLE_DEVICES="" python nemo_text_processing/inverse_text_normalization/inverse_normalize.py --lang=hi_en --text="एक" --cache_dir ${HI_EN_TN_CACHE}'
}
}

(replace info with your language) - only add stage and steps (look how other language added)

keep only stage('L0: TE TN grammars') {
steps {
sh 'CUDA_VISIBLE_DEVICES="" python nemo_text_processing/text_normalization/normalize.py --lang=te --text="౧" --cache_dir ${TE_TN_CACHE}'
}

nharidas · 2026-08-19T07:49:19Z

Thanks for the review. I’ve addressed the comments on this PR.

Removed Telugu post-processing (no PostProcessingFst for te; tests use post_process=False)
Dropped custom add_weight; competing *1 forms are split instead of using a -0.2 bias
Deleted teens_and_ties_thousand.tsv; combining ఒక is composed from ties
magnitudes.tsv is loaded with load_labels
Digit/zero classes come from digit.tsv / zero.tsv (no hardcoded NEMO_TE_DIGIT)
Word tagger simplified to NEMO_NOT_SPACE
Jenkinsfile: removed the commented TE ITN stage
Tests: assert pred == expected (no .strip())

pytest for Telugu cardinals is passing. Sparrowhawk tests are still pending; I’ll update this PR once they are run.

folivoramanh

add init file in data/ (currently only in data/numbers/)

folivoramanh · 2026-08-22T03:21:38Z

        }
      }
    }
+        stage('L0: Create TE TN/ITN Grammars') {


keep only stage('L0: TE TN grammars') {
steps {
sh 'CUDA_VISIBLE_DEVICES="" python nemo_text_processing/text_normalization/normalize.py --lang=te --text="౧" --cache_dir ${TE_TN_CACHE}'
}

folivoramanh · 2026-08-22T03:27:56Z

check this example and see that if you can build a reusable component for thousand/million/billion/trillion
https://github.com/NVIDIA/NeMo-text-processing/blob/main/nemo_text_processing/text_normalization/es/taggers/cardinal.py

Thanks for the pointer. I reviewed, es/taggers/cardinal.py.

Spanish works with one reusable block because it uses the Western scale: each step adds 6 zeros (thousand → million → billion → trillion) and reuses the same 3-digit pattern with a new magnitude word.

Telugu uses the Indian numbering system, which is structured differently:

1,000 → thousand

1,00,000 → lakh (not “hundred thousand”)

1,00,00,000 → crore

higher tiers (arab, kharab, nil, padma, shankh) with different digit groupings

On top of that, the spoken form is not one fixed word per magnitude. It depends on position in the number and what follows, for example:

singular vs plural / “before” forms (e.g. వేల vs వేలు, కోటి vs కోట్లు)

special “one” form when combining (ఒక in 21, 31, … vs ఒకటి for standalone 1)

prefix variants (e.g. నూట vs వంద for hundreds)

special lakh form (లక్షా)

So we cannot drop in one ES-style thousand/million/billion component, both the scale and the word forms change at each tier.

This current code already reuses shared logic where the grammar allows it:

build_group, create_graph_suffix, create_larger_number_graph

magnitude ladders (thousand → lakh → crore → …)

shared strings in magnitudes.tsv

The longer code reflects real Telugu morphology, not unnecessary duplication. Collapsing it into one Spanish-style block would risk wrong readings (e.g. 21 → ఇరవై ఒక, not ఇరవై ఒకటి).

if it uses the Indian numbering system, are we building the graph in the same way that the Hindi graph does?

@mgrafu Yes, same Indian numbering system and the same overall graph structure as Hindi.
Both use the tiered scale (hundred → thousand → lakh → crore → arab/kharab/nil/…) with the same create_graph_suffix / create_larger_number_graph pattern and the same 2+3 digit grouping.
Telugu extends that base for language-specific morphology: singular/plural/before magnitude forms, special “one” handling (ఒక vs ఒకటి), and hundred prefixes (నూట vs వంద), driven by TSVs and build_group ladders. Hindi uses a simpler one-suffix-per-tier model.

nharidas · 2026-08-22T15:41:15Z

Thanks for the follow-up review.

Changes Made:

Added nemo_text_processing/text_normalization/te/data/init.py
Simplified Jenkinsfile to a single L0: TE TN grammars stage (removed the parallel Create TE TN/ITN Grammars wrapper)
Expanded cardinal tests to cover large magnitudes, including 999999999999999 (15 digits) and up to 19-digit shankh-scale numbers, plus Telugu digits, in-sentence, leading zeros, and negatives
Negative numbers: verbalizer now outputs “మైనస్” for the sign (Telugu localized form, same pattern as Spanish “menos”), instead of English “minus”. Test cases updated accordingly.

Testing:

Cardinal pytest test cases passed
Sparrowhawk “testTNCardinal” passed

Re: Spanish reusable component: replied in the thread with details.

mgrafu · 2026-09-01T21:04:09Z

+
+    def __init__(self, deterministic: bool = True):
+        super().__init__(name="punctuation", kind="classify", deterministic=deterministic)
+        s = "!#%&\'()*+,-./:;<=>?@^_`{|}~\""


does the language need additional punctuation?

Telugu uses standard punctuation, same as Hindi. The tagger already covers all Unicode punctuation plus ASCII symbols, so no additional punctuation is needed.

I don't see the poorna viram in the above list. does that mean Telugu doesn't use it?

@mgrafu Poorna viram (।, U+0964) isn’t in the ASCII string s, but it is included via punct_unicode. Same pattern as Hindi TN. So traditional । / ॥ and Western . common in modern Telugu are both covered.

Rebased onto staging/telugu_tn_v1 with TE-only changes for PR NVIDIA#447. Signed-off-by: Haridas Nishitha <nharidas@nvidia.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Use TSV-based digit definitions in graph_utils, restore load_labels in utils, and update Jenkinsfile with correct cache paths and TE CI stages. Signed-off-by: Haridas Nishitha <nharidas@nvidia.com> Co-authored-by: Cursor <cursoragent@cursor.com>

for more information, see https://pre-commit.ci

tokenize_and_classify passed a punctuation argument that WordFst no longer accepts after the staging rebase. Signed-off-by: Haridas Nishitha <nharidas@nvidia.com> Co-authored-by: Cursor <cursoragent@cursor.com>

nharidas · 2026-09-02T09:33:07Z

Thanks for the review.

Changes Made:

Expanded test_cases_cardinal.txt with internal-zero and separator coverage (Indian and Western grouping)
Added separator handling in cardinal.py (indian_grouping / western_grouping) to strip commas before normalization
Moved hardcoded inputs to TSV files.
Removed unused code in test_sparrowhawk_normalization.sh

Testing:

Cardinal pytest test cases passed
Sparrowhawk testTNCardinal passed

nharidas force-pushed the feat-cardinal-te branch from 4a71ee5 to f918e66 Compare July 7, 2026 07:57

nharidas marked this pull request as ready for review July 7, 2026 10:52

github-actions Bot added Stale and removed Stale labels Jul 22, 2026

nharidas changed the title ~~Added Cardinal class for Telugu TN~~ Telugu TN: Cardinal Semiotic Class Aug 6, 2026

folivoramanh requested changes Aug 17, 2026

View reviewed changes

nharidas force-pushed the feat-cardinal-te branch 2 times, most recently from 9d8b75f to b224888 Compare August 19, 2026 06:46

nharidas requested a review from folivoramanh August 19, 2026 07:51

folivoramanh requested changes Aug 22, 2026

View reviewed changes

nharidas requested a review from folivoramanh August 22, 2026 15:44

folivoramanh requested a review from mgrafu August 28, 2026 03:45

mgrafu reviewed Sep 1, 2026

View reviewed changes

Comment thread tests/nemo_text_processing/te/test_sparrowhawk_normalization.sh Outdated

mgrafu reviewed Sep 1, 2026

View reviewed changes

Comment thread tests/nemo_text_processing/te/data_text_normalization/test_cases_cardinal.txt

mgrafu reviewed Sep 1, 2026

View reviewed changes

Comment thread nemo_text_processing/text_normalization/te/taggers/cardinal.py Outdated

nharidas force-pushed the feat-cardinal-te branch from 3fb9a92 to 415b3cf Compare September 2, 2026 07:29

feat(te): add Telugu cardinal text normalization

1e9a8cb

Rebased onto staging/telugu_tn_v1 with TE-only changes for PR NVIDIA#447. Signed-off-by: Haridas Nishitha <nharidas@nvidia.com> Co-authored-by: Cursor <cursoragent@cursor.com>

nharidas force-pushed the feat-cardinal-te branch from 15c6609 to 1e9a8cb Compare September 2, 2026 07:36

nharidas and others added 3 commits September 2, 2026 13:24

[pre-commit.ci] auto fixes from pre-commit.com hooks

db12528

for more information, see https://pre-commit.ci

fix(te): align WordFst call with current word tagger signature

19f137a

tokenize_and_classify passed a punctuation argument that WordFst no longer accepts after the staging rebase. Signed-off-by: Haridas Nishitha <nharidas@nvidia.com> Co-authored-by: Cursor <cursoragent@cursor.com>

nharidas requested a review from mgrafu September 2, 2026 09:35

Uh oh!

Conversation

nharidas commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Before your PR is "Ready for review"

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

folivoramanh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nharidas commented Aug 19, 2026

Uh oh!

folivoramanh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nharidas Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nharidas commented Aug 22, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nharidas commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nharidas commented Jul 7, 2026 •

edited

Loading

nharidas Sep 2, 2026 •

edited

Loading