[BUG] Small difference between Python and C++

[mgovers]

Describe the bug

There seems to be a difference between calling e.g. calculate_power_flow with the same arguments between Python and C++ on the same dataset. This happens even for the linear method, so it is unlikely to be an issue within the PGM core itself. This should not be happening, as the Python side only uses zero-copy views of the C++ data in the whole process.

NOTE: by "whole process", I mean: JSON file -> Python JSON string -> C++ JSON string -> C++ deserialization -> C++ deserialized dataset -> Python input -> C++ input dataset -> C++ calculation -> C++ result dataset -> Python result dataset -> C++ serializable dataset -> C++ serialization -> C++ JSON string -> Python JSON string -> JSON file

Input Data Validity

• I checked the validity of input data using the helper function (examples here).

Data Dump

input data

{
  "version": "1.0",
  "type": "input",
  "is_batch": false,
  "attributes": {},
  "data": {
    "node": [
      {"id": 1, "u_rated": 10e3},
      {"id": 2, "u_rated": 10e3}
    ],
    "link": [
      {"id": 3, "from_node": 1, "to_node": 2, "from_status": 1, "to_status": 1}
    ],
    "source": [
      {"id": 4, "node": 1, "status": 1, "u_ref": 1.05, "sk": 1e12}
    ],
    "sym_load": [
      {"id": 5, "node": 2, "status": 1, "type": 1, "p_specified": 2.0e5, "q_specified": 1.0e5}
    ]
  }
}

params.json

{
  "calculation_method": ["linear"],
  "rtol": 1e-9,
  "atol": 1e-9
}

asym_output.json (taken from C++ result)

{
  "version": "1.0",
  "type": "asym_output",
  "is_batch": false,
  "attributes": {},
  "data": {
    "node": [
      {"id": 1, "energized": 1, "u_pu": [1.0499998746253056, 1.0499998746253056, 1.0499998746253056], "u": [6062.1771026399347, 6062.1771026399347, 6062.1771026399347], "u_angle": [-1.8905703971413632e-07, -2.0943952914502351, 2.0943949133361559], "p": [73499.990661431395, 73499.981651442053, 73499.994606849592], "q": [36749.983176509326, 36749.98083851079, 36749.983936708901]},
      {"id": 2, "energized": 1, "u_pu": [1.0499998741003056, 1.0499998741003056, 1.0499998741003058], "u": [6062.1770996088462, 6062.1770996088462, 6062.1770996088471], "u_angle": [-1.8755703975179635e-07, -2.0943952899502354, 2.094394914836156], "p": [-73499.990569556423, -73499.981559567081, -73499.99451497462], "q": [-36749.983268384312, -36749.980930385755, -36749.984028583902]}
    ],
    "link": [
      {"id": 3, "energized": 1, "loading": 0, "p_from": [73499.990661431395, 73499.981651442053, 73499.994606849592], "q_from": [36749.983176509326, 36749.98083851079, 36749.983936708901], "i_from": [13.555440711385531, 13.555439209554709, 13.555441349582171], "s_from": [82175.482296754548, 82175.473192390127, 82175.486165615614], "p_to": [-73499.990569556423, -73499.981559567081, -73499.99451497462], "q_to": [-36749.983268384312, -36749.980930385762, -36749.984028583902], "i_to": [13.555440711385531, 13.555439209554709, 13.555441349582171], "s_to": [82175.482255666808, 82175.473151302387, 82175.486124527859]}
    ],
    "source": [
      {"id": 4, "energized": 1, "p": [73499.990661431395, 73499.981651442053, 73499.994606849592], "q": [36749.983176509326, 36749.980838510804, 36749.983936708893], "i": [13.555440711385531, 13.555439209554711, 13.55544134958217], "s": [82175.482296754548, 82175.473192390142, 82175.486165615599], "pf": [0.89442725016211089, 0.89442723961397907, 0.89442725606415618]}
    ],
    "sym_load": [
      {"id": 5, "energized": 1, "p": [73499.98237404383, 73499.982374043815, 73499.982374043844], "q": [36749.991187021915, 36749.991187021908, 36749.991187021922], "i": [13.555440086367891, 13.555440086367893, 13.555440086367895], "s": [82175.478466699191, 82175.478466699176, 82175.478466699205], "pf": [0.89442719099991586, 0.89442719099991586, 0.89442719099991586]}
    ]
  }
}

To Reproduce

Run the C++ validation tests and Python validation tests separately on the provided dataset (platform used here: MacOS, but I have been able to reproduce this on Linux as well with another dataset)

C++ behavior

the test passes (in fact, the C++ test result was used as the benchmark

Python behavior

the test fails. The result i'm getting is

{
  "version": "1.0",
  "type": "asym_output",
  "is_batch": false,
  "attributes": {},
  "data": {
    "node": [
      {"id": 1, "energized": 1, "u_pu": [1.0499998746253056, 1.0499998746253054, 1.0499998746253054], "u": [6062.1771026399347, 6062.1771026399338, 6062.1771026399338], "u_angle": [-1.8905703971087225e-07, -2.0943952914502351, 2.0943949133361559], "p": [73499.990661431395, 73499.982955293919, 73499.984874771093], "q": [36749.983176509326, 36749.983096847507, 36749.99036235566]},
      {"id": 2, "energized": 1, "u_pu": [1.0499998741003056, 1.0499998741003056, 1.0499998741003056], "u": [6062.1770996088462, 6062.1770996088462, 6062.1770996088462], "u_angle": [-1.8755703977009874e-07, -2.0943952899502354, 2.094394914836156], "p": [-73499.990569556423, -73499.982863418947, -73499.984782896106], "q": [-36749.983268384305, -36749.983188722472, -36749.990454230639]}
    ],
    "link": [
      {"id": 3, "energized": 1, "loading": 0, "p_from": [73499.990661431395, 73499.982955293919, 73499.984874771093], "q_from": [36749.983176509326, 36749.983096847507, 36749.99036235566], "i_from": [13.555440711385531, 13.555439568527941, 13.555440387716526], "s_from": [82175.482296754548, 82175.475368549422, 82175.480334615713], "p_to": [-73499.990569556423, -73499.982863418947, -73499.984782896106], "q_to": [-36749.983268384312, -36749.983188722472, -36749.990454230639], "i_to": [13.555440711385531, 13.555439568527941, 13.555440387716526], "s_to": [82175.482255666808, 82175.475327461696, 82175.480293527973]}
    ],
    "source": [
      {"id": 4, "energized": 1, "p": [73499.990661431395, 73499.982955293905, 73499.984874771108], "q": [36749.983176509326, 36749.9830968475, 36749.99036235566], "i": [13.555440711385531, 13.555439568527939, 13.555440387716528], "s": [82175.482296754548, 82175.475368549422, 82175.480334615728], "pf": [0.89442725016211089, 0.89442723179456241, 0.89442720110039753]}
    ],
    "sym_load": [
      {"id": 5, "energized": 1, "p": [73499.98237404383, 73499.98237404383, 73499.98237404383], "q": [36749.991187021915, 36749.991187021915, 36749.991187021915], "i": [13.555440086367891, 13.555440086367895, 13.555440086367893], "s": [82175.478466699191, 82175.478466699191, 82175.478466699191], "pf": [0.89442719099991586, 0.89442719099991586, 0.89442719099991586]}
    ]
  }
}

diff

8,9c8,9
<       {"id": 1, "energized": 1, "u_pu": [1.0499998746253056, 1.0499998746253056, 1.0499998746253056], "u": [6062.1771026399347, 6062.1771026399347, 6062.1771026399347], "u_angle": [-1.8905703971413632e-07, -2.0943952914502351, 2.0943949133361559], "p": [73499.990661431395, 73499.981651442053, 73499.994606849592], "q": [36749.983176509326, 36749.98083851079, 36749.983936708901]},
<       {"id": 2, "energized": 1, "u_pu": [1.0499998741003056, 1.0499998741003056, 1.0499998741003058], "u": [6062.1770996088462, 6062.1770996088462, 6062.1770996088471], "u_angle": [-1.8755703975179635e-07, -2.0943952899502354, 2.094394914836156], "p": [-73499.990569556423, -73499.981559567081, -73499.99451497462], "q": [-36749.983268384312, -36749.980930385755, -36749.984028583902]}
---
>       {"id": 1, "energized": 1, "u_pu": [1.0499998746253056, 1.0499998746253054, 1.0499998746253054], "u": [6062.1771026399347, 6062.1771026399338, 6062.1771026399338], "u_angle": [-1.8905703971087225e-07, -2.0943952914502351, 2.0943949133361559], "p": [73499.990661431395, 73499.982955293919, 73499.984874771093], "q": [36749.983176509326, 36749.983096847507, 36749.99036235566]},
>       {"id": 2, "energized": 1, "u_pu": [1.0499998741003056, 1.0499998741003056, 1.0499998741003056], "u": [6062.1770996088462, 6062.1770996088462, 6062.1770996088462], "u_angle": [-1.8755703977009874e-07, -2.0943952899502354, 2.094394914836156], "p": [-73499.990569556423, -73499.982863418947, -73499.984782896106], "q": [-36749.983268384305, -36749.983188722472, -36749.990454230639]}
12c12
<       {"id": 3, "energized": 1, "loading": 0, "p_from": [73499.990661431395, 73499.981651442053, 73499.994606849592], "q_from": [36749.983176509326, 36749.98083851079, 36749.983936708901], "i_from": [13.555440711385531, 13.555439209554709, 13.555441349582171], "s_from": [82175.482296754548, 82175.473192390127, 82175.486165615614], "p_to": [-73499.990569556423, -73499.981559567081, -73499.99451497462], "q_to": [-36749.983268384312, -36749.980930385762, -36749.984028583902], "i_to": [13.555440711385531, 13.555439209554709, 13.555441349582171], "s_to": [82175.482255666808, 82175.473151302387, 82175.486124527859]}
---
>       {"id": 3, "energized": 1, "loading": 0, "p_from": [73499.990661431395, 73499.982955293919, 73499.984874771093], "q_from": [36749.983176509326, 36749.983096847507, 36749.99036235566], "i_from": [13.555440711385531, 13.555439568527941, 13.555440387716526], "s_from": [82175.482296754548, 82175.475368549422, 82175.480334615713], "p_to": [-73499.990569556423, -73499.982863418947, -73499.984782896106], "q_to": [-36749.983268384312, -36749.983188722472, -36749.990454230639], "i_to": [13.555440711385531, 13.555439568527941, 13.555440387716526], "s_to": [82175.482255666808, 82175.475327461696, 82175.480293527973]}
15c15
<       {"id": 4, "energized": 1, "p": [73499.990661431395, 73499.981651442053, 73499.994606849592], "q": [36749.983176509326, 36749.980838510804, 36749.983936708893], "i": [13.555440711385531, 13.555439209554711, 13.55544134958217], "s": [82175.482296754548, 82175.473192390142, 82175.486165615599], "pf": [0.89442725016211089, 0.89442723961397907, 0.89442725606415618]}
---
>       {"id": 4, "energized": 1, "p": [73499.990661431395, 73499.982955293905, 73499.984874771108], "q": [36749.983176509326, 36749.9830968475, 36749.99036235566], "i": [13.555440711385531, 13.555439568527939, 13.555440387716528], "s": [82175.482296754548, 82175.475368549422, 82175.480334615728], "pf": [0.89442725016211089, 0.89442723179456241, 0.89442720110039753]}
18c18
<       {"id": 5, "energized": 1, "p": [73499.98237404383, 73499.982374043815, 73499.982374043844], "q": [36749.991187021915, 36749.991187021908, 36749.991187021922], "i": [13.555440086367891, 13.555440086367893, 13.555440086367895], "s": [82175.478466699191, 82175.478466699176, 82175.478466699205], "pf": [0.89442719099991586, 0.89442719099991586, 0.89442719099991586]}
---
>       {"id": 5, "energized": 1, "p": [73499.98237404383, 73499.98237404383, 73499.98237404383], "q": [36749.991187021915, 36749.991187021915, 36749.991187021915], "i": [13.555440086367891, 13.555440086367895, 13.555440086367893], "s": [82175.478466699191, 82175.478466699191, 82175.478466699191], "pf": [0.89442719099991586, 0.89442719099991586, 0.89442719099991586]}

Expected behavior

The C++ test result and Python test result are the same (binary-equivalent). It is at this point in time not known which of the two is correct.

Screenshots

No response

Additional context

No response

Community Meeting

• I wish to discuss this bug in the bi-monthly PGM community meeting. Please join the PGM community meeting.

[TonyXiang8787]

@mgovers can you elaborate what exactly the problem is?

The difference seems to be within the default preset tolerance of 1e-8. So that's expected? You set comparison tolerance to 1e-9 which is higher than convergence criteria.

[mgovers]

@mgovers can you elaborate what exactly the problem is?

The difference seems to be within the default preset tolerance of 1e-8. So that's expected? You set comparison tolerance to 1e-9 which is higher than convergence criteria.

the issue is that if i use the exact same input data from Python and from C++ and i use a linear solver (no convergence criteria/iteration), then i get differences. This should not happen because they should be binary equivalent. I have not been able to figure out yet why that difference exists, but I have had larger grids in which the difference exceeds 1e-8, which is not great.

[TonyXiang8787]

@mgovers

If you do use NRPF and set tolerance to 1e-8, can you reproduce a deviation larger than 1e-8 for voltages?

[mgovers]

@mgovers

If you do use NRPF and set tolerance to 1e-8, can you reproduce a deviation larger than 1e-8 for voltages?

yes, I have.

But once again, that is not the point of this issue. This is not about iterations. This is about multiple code paths that inexplicably produce different output even though they are supposed to be binary-equivalent.

[TonyXiang8787]

I don't think floating point number calcualtions can garantee binary equivalent in the first place. It is always about reaching to certain expectation of deviations.

No hardware/software architecture garantees floating point binary equivalent output given the same input, as far as I know.

[mgovers]

On a given platform and architecture, following the same code path, the behavior is deterministic. The reason not to do floating-point binary-equivalence checks has to do with portability, not with whether it's deterministic or not.

Since both the python side and the C++ side do everything zero-copy, they should be binary-equivalent. Even if not, they should deviate by maybe one or a couple double-precision floating point epsilons (10^-15 ~ 10^-16) at most. Here, they deviate by a multiple of that.

At the very least, I want to investigate why this happens.

[TonyXiang8787]

On a given platform and architecture, following the same code path, the behavior is deterministic. The reason not to do floating-point binary-equivalence checks has to do with portability, not with whether it's deterministic or not.

Since both the python side and the C++ side do everything zero-copy, they should be binary-equivalent. Even if not, they should deviate by maybe one or a couple double-precision floating point epsilons (10^-15 ~ 10^-16) at most. Here, they deviate by a multiple of that.

At the very least, I want to investigate why this happens.

When you did your comparison, did you use the same produced built artifact (shared library)? If you compare the shared library built via some cmake preset to the shared library built by scikit-build-core, those two have been compiled using different flags which can affect floating point handling.

My advice to investigate further, is to compare the exact built shared library, do one of the following:

1. Copy the built shared library from cmake preset into site-packages.
2. Copy the built shared library in site-packages into your cmake preset target tree.

Then compare C++ and python run results.

[mgovers]

Yes that was also my intention. I just want to know why it happens. If it turns out to be for valid reasons, then we can leave it be. Above all, we just need to understand the reason.