Tamil TN Cardinal Semiotic Class by surendran-246 · Pull Request #449 · NVIDIA/NeMo-text-processing

surendran-246 · 2026-07-07T07:22:02Z

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Before your PR is "Ready for review"

Pre checks:

PR Type:

New Feature
Bugfix
Documentation
Test

If you haven't finished some of the above items you can still open "Draft" PR.

Signed-off-by: surendran-246 <surendrans@nvidia.com>

for more information, see https://pre-commit.ci Signed-off-by: surendran-246 <surendrans@nvidia.com>

Signed-off-by: surendran-246 <surendrans@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: surendran-246 <surendrans@nvidia.com>

github-actions · 2026-07-23T02:17:40Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

Signed-off-by: surendran-246 <surendrans@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: surendran-246 <surendrans@nvidia.com>

github-actions · 2026-08-08T02:14:10Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

for more information, see https://pre-commit.ci

Signed-off-by: surendran-246 <surendrans@nvidia.com>

surendran-246 · 2026-08-28T09:30:27Z

make sure both pytest and sparrrowhawk test pass

Completed the pytest and sparrowhawk tests, and both passed successfully.

mgrafu · 2026-09-01T19:51:12Z

@@ -0,0 +1,36 @@
+100	நூறு


looking at one hundred, these could be compressed into a rule. would this also apply for the other hundreds? if so, please refactor

(1 > நூ) followed by (delete 00 or insert ற்) followed by (insert று)

Yes, applies to all hundreds. Replaced hundred_ta.tsv with hundred_stem.tsv (digit → stem only).

Exact = stem + delete "00" + insert "று" (100, 200, … 900)
Combining = stem + insert "ற்" + insert "று" (101–999)
One rule now covers both forms from the same stem table.

mgrafu · 2026-09-01T20:02:23Z

+        teens_and_ties = teens_ties
+
+        # digit_oru
+        one_oru = pynini.cross("1", "ஒரு") | pynini.cross("௧", "ஒரு")


let's have everything be files instead of hardcoding

Fixed and moved to one_oru.tsv file.

mgrafu · 2026-09-01T20:03:12Z

+        thousand_exact = thousand_stem + pynutil.insert("ம்")
+        thousand_prefix = thousand_stem + pynutil.insert("த்து")
+
+        self.digit = digit


these don't seem to be used anywhere?

thousand_exact and thousand_prefix are used (they feed into graph_thousands), so they were kept. self.digit was genuinely unused, so it was removed.

mgrafu · 2026-09-01T20:03:39Z

+
+        single_digit = digit | zero
+        self.single_digits_graph = single_digit + pynini.closure(insert_space + single_digit)
+        zero_del = pynutil.add_weight(pynutil.delete(NEMO_ALL_ZERO), -0.1)


is this weight absolutely necessary?

Not necessary, and it has been removed.

mgrafu · 2026-09-01T20:09:26Z

+        self.graph_thousands = graph_thousands
+        tails = [single_digit, teens_ties, graph_hundreds, graph_thousands]
+
+        # TEN-THOUSANDS (10^4): stem + ஆயிரம்


these should also be in tsv files. the process also repeats for graph_ten_thousands, graph_lakhs and graph_ten_lakhs so it could be compressed

Compressed into a single add_scale() helper that wraps band() + tail-list append, so the same 3-line pattern isn't repeated for each scale level. Related hardcoded words (ஆயிரம்/ஆயிரத்து, லட்சம்/லட்சத்து) moved into scale.tsv.

mgrafu · 2026-09-01T20:11:17Z

+            graph_ten_lakhs,  # ten-lakhs of crores
+        ]
+        with open(get_abs_path("data/numbers/crore.tsv"), encoding="utf-8") as f:
+            crore_exact_raw, crore_prefix_raw = [line.strip() for line in f if line.strip()]


why not string_file?

switched to pynini.string_file.

mgrafu · 2026-09-01T20:13:18Z

+
+    def __init__(self, deterministic: bool = True):
+        super().__init__(name="punctuation", kind="classify", deterministic=deterministic)
+        s = "!#%&\'()*+,-./:;<=>?@^_`{|}~\""


does the language require any additional punctuation?

The tagger already covers all Unicode punctuation categories, and Tamil mostly reuses ASCII punctuation, so no additional punctuation is needed.

mgrafu · 2026-09-01T20:15:31Z

remove this file

mgrafu · 2026-09-01T20:15:39Z

remove this file

mgrafu · 2026-09-01T20:15:44Z

remove this file

mgrafu · 2026-09-01T20:16:14Z

+    escaped_written=$(printf '%s' "$written" | sed 's/\\/\\\\/g')
+    denorm_pred=$(echo "$escaped_written" | normalizer_main --config=sparrowhawk_configuration.ascii_proto 2>&1 | tail -n 1 | sed 's/\xC2\xA0/ /g')
+
+    # trim white space


remove commented out code not needed

mgrafu · 2026-09-01T20:17:11Z

@@ -0,0 +1,53 @@
+4 நான்குகள்~நான்கு நான்குகள்


does this cover all rule categories included? at the very least I see negative numbers missing

Yes, it covers all rules, and I’ve added negative numbers and comma-grouped numerals to the test cases

mgrafu · 2026-09-01T20:19:34Z

+class VerbalizeFinalFst(GraphFst):
+    """
+    Finite state transducer that verbalizes an entire sentence, e.g.
+    tokens { name: "its" } tokens { time { hours: "twelve" minutes: "thirty" } } tokens { name: "now" } tokens { name: "." } -> its twelve thirty now .


all definitions should be in the language and follow the format of a real output

Fixed and updated the docstring example with real Tamil input and output.

mgrafu · 2026-09-01T20:20:26Z

+
+        self.optional_sign = pynini.cross("negative: \"true\"", "minus ")
+        if not deterministic:
+            self.optional_sign |= pynini.cross("negative: \"true\"", "negative ")


why is this in English?

Fixed — replaced the English negative with the Tamil equivalent எதிர்மறை.

mgrafu · 2026-09-01T20:20:40Z

+    def __init__(self, deterministic: bool = True):
+        super().__init__(name="cardinal", kind="verbalize", deterministic=deterministic)
+
+        self.optional_sign = pynini.cross("negative: \"true\"", "minus ")


why is this in English?

Fixed — replaced the English minus with the Tamil equivalent கழித்தல்.

mgrafu · 2026-09-01T20:20:45Z

+        self.optional_sign = pynini.cross("negative: \"true\"", "minus ")
+        if not deterministic:
+            self.optional_sign |= pynini.cross("negative: \"true\"", "negative ")
+            self.optional_sign |= pynini.cross("negative: \"true\"", "dash ")


why is this in English?

Fixed — replaced the English dash with the Tamil equivalent கோடுகுறி.

mgrafu · 2026-09-01T20:35:23Z

please make sure that all new directories have init files

Signed-off-by: surendran-246 <surendrans@nvidia.com>

for more information, see https://pre-commit.ci

surendran-246 · 2026-09-03T06:07:44Z

Thanks for the review, @mgrafu. Addressed all the feedback comments.

Testing: Cardinal pytest passed; Sparrowhawk TNCardinal passed.

Signed-off-by: surendran-246 <surendrans@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: surendran-246 <surendrans@nvidia.com>

for more information, see https://pre-commit.ci

surendran-246 force-pushed the feat-ta-cardinal branch from b06d8e8 to 6b431c2 Compare July 7, 2026 07:43

surendran-246 and others added 10 commits July 7, 2026 13:51

feat(ta): add cardinal TN tagger, verbalizer and test cases

856a301

Signed-off-by: surendran-246 <surendrans@nvidia.com>

feat(ta): address review comments

7ebabdf

Signed-off-by: surendran-246 <surendrans@nvidia.com>

message

d68701b

Signed-off-by: surendran-246 <surendrans@nvidia.com>

Updated Ta Cardinal

c2fc100

Signed-off-by: surendran-246 <surendrans@nvidia.com>

cardinal implementation

f9d575d

Signed-off-by: surendran-246 <surendrans@nvidia.com>

cardinal file

fcfcb3e

Signed-off-by: surendran-246 <surendrans@nvidia.com>

Updated cardinal file

423f239

Signed-off-by: surendran-246 <surendrans@nvidia.com>

Updated cardinal

5a39ceb

Signed-off-by: surendran-246 <surendrans@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

a0815e9

for more information, see https://pre-commit.ci Signed-off-by: surendran-246 <surendrans@nvidia.com>

Cardinal updated file

eaf2223

Signed-off-by: surendran-246 <surendrans@nvidia.com>

surendran-246 force-pushed the feat-ta-cardinal branch from 231dd35 to eaf2223 Compare July 7, 2026 08:21

Updated cardinal

c0e51f1

Signed-off-by: surendran-246 <surendrans@nvidia.com>

surendran-246 changed the title ~~Tamil language cardinal semiotic class~~ Tamil TN Cardinal Semiotic Class Jul 7, 2026

surendran-246 marked this pull request as ready for review July 7, 2026 10:50

surendran-246 and others added 6 commits July 8, 2026 11:04

Updated Cardinal

a0cd909

Signed-off-by: surendran-246 <surendrans@nvidia.com>

Added Cardinal

4761e67

Signed-off-by: surendran-246 <surendrans@nvidia.com>

Ta cardinal

301db6f

Signed-off-by: surendran-246 <surendrans@nvidia.com>

Added Cardinal file

60b9d58

Signed-off-by: surendran-246 <surendrans@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

fe805eb

for more information, see https://pre-commit.ci

Updated Cardinal file

e2e7996

Signed-off-by: surendran-246 <surendrans@nvidia.com>

github-actions Bot added the Stale label Jul 23, 2026

Updated TA Cardinal

2907744

Signed-off-by: surendran-246 <surendrans@nvidia.com>

surendran-246 force-pushed the feat-ta-cardinal branch from 0e62120 to 2907744 Compare July 23, 2026 13:10

surendran-246 and others added 2 commits July 23, 2026 18:44

Merge branch 'staging/tamil_tn_v1' into feat-ta-cardinal

0531bc2

Signed-off-by: surendran-246 <surendrans@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

026875c

for more information, see https://pre-commit.ci

github-actions Bot removed the Stale label Jul 24, 2026

Modified Cardinal

5561410

Signed-off-by: surendran-246 <surendrans@nvidia.com>

pre-commit-ci Bot and others added 2 commits August 19, 2026 07:23

[pre-commit.ci] auto fixes from pre-commit.com hooks

316e82e

for more information, see https://pre-commit.ci

Updated Jenkins file

f64f094

Signed-off-by: surendran-246 <surendrans@nvidia.com>

folivoramanh requested a review from mgrafu August 28, 2026 03:45

mgrafu reviewed Sep 1, 2026

View reviewed changes

surendran-246 and others added 2 commits September 3, 2026 10:23

Updated Cardinal class based on the PR review

7fd892b

Signed-off-by: surendran-246 <surendrans@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

13368b6

for more information, see https://pre-commit.ci

surendran-246 requested a review from mgrafu September 3, 2026 06:08

surendran-246 and others added 5 commits September 3, 2026 17:33

Added comma parsing + docstring

57ba53c

Signed-off-by: surendran-246 <surendrans@nvidia.com>

Added comma parsing + docstring

aad3c7a

Signed-off-by: surendran-246 <surendrans@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

316230f

for more information, see https://pre-commit.ci

Optimized cardinal file and added the init file

ddd96c2

Signed-off-by: surendran-246 <surendrans@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

a7df5e9

for more information, see https://pre-commit.ci

		@@ -0,0 +1,53 @@
		4 நான்குகள்~நான்கு நான்குகள்

Uh oh!

Conversation

surendran-246 commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Before your PR is "Ready for review"

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

github-actions Bot commented Aug 8, 2026

Uh oh!

surendran-246 commented Aug 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

surendran-246 Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

surendran-246 Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

surendran-246 Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

surendran-246 commented Jul 7, 2026 •

edited

Loading

surendran-246 Sep 3, 2026 •

edited

Loading

surendran-246 Sep 3, 2026 •

edited

Loading

surendran-246 Sep 3, 2026 •

edited

Loading