Skip to content

Avoid NaNs from roundoff in MeshGraphNets normalization - #759

Open
sylvesterkaczmarek wants to merge 1 commit into
google-deepmind:masterfrom
sylvesterkaczmarek:fix/meshgraphnets-normalizer-variance
Open

sylvesterkaczmarek wants to merge 1 commit into
google-deepmind:masterfrom
sylvesterkaczmarek:fix/meshgraphnets-normalizer-variance

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

Prevent MeshGraphNets' online normalizer from producing NaNs when floating-point roundoff makes the computed variance slightly negative.

The normalizer computes variance using E[x^2] - E[x]^2. For nearly constant features, finite-precision accumulation can make that expression a small negative number even though the mathematical variance is non-negative. The current implementation applies sqrt first and only then clamps the resulting standard deviation, so a negative intermediate value becomes NaN and the later maximum cannot recover it.

This change:

  • computes the variance explicitly
  • clamps variance to zero before taking the square root
  • preserves the existing std_epsilon floor
  • adds a regression test using accumulated statistics that reproduce a small negative roundoff variance

Fixes #346.

Validation

Added meshgraphnets/normalization_test.py with a focused regression case verifying that a small negative roundoff variance produces std_epsilon rather than NaN.

TensorFlow/Sonnet matching this legacy environment are not installed in the execution environment, so the regression test was not executed locally and no local test pass is claimed.

The final branch is based directly on current upstream master, contains one commit (9411f4e6bb1693c31013db1fe7c384b02b993313), and changes only meshgraphnets/normalization.py plus its regression test.

@polarbe

polarbe commented Aug 23, 2026 via email

Copy link
Copy Markdown

@sylvesterkaczmarek

Copy link
Copy Markdown
Author

Could a maintainer review the MeshGraphNets normalization roundoff fix and arrange a fresh approved workflow run? The previous security run ended with no jobs after 30 days, which looks like expired approval.

@polarbe

polarbe commented Oct 7, 2026 via email

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment