Port to PowerIO 0.11

CHANGELOG.md

@@ -1,5 +1,85 @@
 # Changelog
 
+## Unreleased
+
+Tracks **PowerIO 0.11** (powerio C ABI 7). The `[compat]` bound is now
+`PowerIO = "0.11"`; the binding gates its ABI handshake on equality, so it and its
+binaries move together and the bound cannot be relaxed.
+
+PowerIO 0.11 replaces the accessor-and-JSON-payload layer with typed element
+tables, so this is a rewrite of the seam rather than a version bump. PowerDiff is a
+consumer of PowerIO: every electrical quantity PowerIO states is now read from it,
+and nothing that PowerIO computes is computed again here.
+
+### Changed
+
+- **Breaking:** `parse_file` and `parse_matpower` return a
+  `PowerIO.PioModule{PowerIO.BalancedNetwork}` rather than a bare network. The
+  module carries the reader's diagnostics, the source record and the history
+  alongside the case, and it is what `PowerIO.emit` writes back out. Every network
+  constructor accepts the module or the network inside it, so `DCNetwork(net)`,
+  `ACNetwork(net)`, `DCOPFProblem(net)`, `ACOPFProblem(net)`,
+  `DCPowerFlowState(net)` and `calc_demand_vector(net)` are unchanged at the call
+  site.
+- **Breaking:** a failure inside PowerIO reaches the caller as a
+  `PowerIO.PowerIOError` instead of being flattened into an `ArgumentError`. It
+  carries the diagnostic `code` and the records behind it, both of which wrapping
+  discarded. PowerDiff's own refusals stay `ArgumentError`.
+- **Breaking:** `parse_matpower_struct` is removed. It was a compatibility alias
+  for `parse_matpower` with no callers.
+- **Breaking:** `network_findings` is removed, one release after it was added and
+  before it ever shipped. It existed because PowerIO 0.9 reached its findings only
+  through a normalize pass PowerDiff had to own, deduplicate and label. In 0.11 they
+  are `m.diagnostics`, a property of the module `parse_file` already returns, and
+  a `Diagnostic` is a record with a `code` and a `severity` rather than a line of
+  text. Wrapping a property in an exported function is surface, not integration.
+- The series conductance and susceptance are read back from PowerIO's terminal
+  admittance coefficients rather than derived from `r` and `x`, and each branch
+  terminal's charging admittance is carried on its own side. PowerDiff previously
+  summed the two sides, split the total evenly and discarded the charging
+  conductance, which lost fidelity on every source that states the terminals
+  separately. Nothing in `src/` inverts a branch impedance any more.
+- The branch-by-bus incidence matrix is stated once and shared by both network
+  types, which previously assembled one each.
+- Out-of-service and isolated elements are selected out here rather than by
+  PowerIO. `to_powerdata` is unfiltered in 0.11: every row carries a `status` and
+  its source row number, which is exactly the `IDMapping` index, so the two-pass
+  reconciliation that used to recover those numbers is gone. Isolated (`type == 4`)
+  buses and everything standing on them are dropped, as before.
+- A generator's cost model is read off the element rather than inferred from the
+  shape of a converted row, so a piecewise linear cost is refused for what it is.
+- The format vocabulary is PowerIO's rather than a copy of it: an unrecognized
+  token goes to PowerIO, which answers with what the linked library actually reads,
+  and a reader PowerIO gains is reachable without a PowerDiff release. PowerDiff's
+  historical short spellings still resolve. A bare `json` is still refused as
+  ambiguous. The hand-maintained distribution-format blocklist is replaced by a
+  check on what the source actually parsed to, which also covers time series,
+  scenario sets and calculation instances.
+- `docs/powerio-integration.md` moves to `docs/src/` and is published; it stated the
+  ingest contract but sat outside the docs build, so it was never rendered.
+
+### Fixed
+
+- A branch whose source states no thermal limit takes the synthesized limit whether
+  the source spells that `0` or `Inf`. PowerIO 0.11 carries MATPOWER's `rate_a == 0`
+  out as `Inf`, which the previous `rate_a > 0` test accepted, so every unrated
+  branch would have reached the solver with an unbounded flow. Stock IEEE 300 leaves
+  all 411 branches unrated.
+- Generator reactive limits may be absent. PowerIO carries a bound the case does not
+  state as `±Inf` rather than refusing the case, and stock case9241pegase leaves them
+  off seven generators. PowerDiff leaves the bound off the solver model and its KKT
+  complementarity row reads `ρ = 0`, the multiplier of a constraint that is not
+  there and the value a solver reports for a bound it was never given. Without this
+  the upgrade would have turned a clean refusal into `0 * Inf` — a `NaN` in the
+  residual and an `Inf` in the Jacobian. The KKT sparsity pattern is unchanged.
+
+### Added
+
+- A test that the ingest still reproduces MATPOWER's network, comparing
+  `ptdf_matrix` against `PowerModels.calc_basic_ptdf_matrix` on three PGLib cases.
+  It shares no code with the ingest and pins the topology, the branch susceptances
+  and the reference bus together.
+
 ## 0.1.0
 
 First release on the PowerIO 0.6.x data layer, and the first cut through the

Project.toml

@@ -21,7 +21,7 @@ JuMP = "1"
 LazyArtifacts = "1"
 NLPModelsIpopt = "0.11.2"
 PowerModels = "0.21"
-PowerIO = "0.8.2"
+PowerIO = "0.11"
 julia = "1.10"
 
 [extras]

README.md

@@ -33,7 +33,7 @@ Pkg.add(url="https://github.com/grid-opt-alg-lab/PowerDiff.jl.git")
 ```julia
 using PowerDiff
 
-# Parse a supported PowerIO case into a PowerIO.BalancedNetwork
+# Parse a case into a PowerIO module
 net = parse_file("case14.m")
 dc_net = DCNetwork(net)
 d = calc_demand_vector(net)
@@ -61,13 +61,24 @@ See the [Getting Started guide](https://samueltalkington.com/research/powerdiff/
 
 ## Input Format
 
-PowerDiff reads files through PowerIO. `parse_file` supports MATPOWER `.m`,
-PSS/E `.raw`, PowerWorld `.aux`, PowerModels JSON, and Egret JSON. For streams,
-pass `from`; JSON streams need `from=:egret` or `from=:powermodels`.
+PowerDiff reads files through PowerIO, and `parse_file` reads every transmission
+format the linked PowerIO library does — MATPOWER `.m`, PSS/E `.raw`, PowerWorld,
+PowerModels JSON, Egret JSON, pandapower, PyPSA, PSLF, gridfm, GO Challenge 3 and
+the rest. The format tokens are PowerIO's, so a reader PowerIO gains works here at
+once.
+
+A path's format is inferred from its extension; a stream has no extension, so pass
+`from` (MATPOWER is assumed otherwise). A bare `json` names a container rather than
+a reader, so name the one you mean: `from=:powermodels`, `:egret`, `:pandapower`.
+
+`parse_file` returns a `PowerIO.PioModule`. Beyond the case itself it carries
+`m.diagnostics`, the reader's findings as records you can branch on by `code` and
+`severity`; `m.sources[1].format`, the reader that ran; and enough for
+`PowerIO.emit(m, "psse", path)` to write the case out again.
 
 ## Dependencies
 
-- [PowerIO.jl](https://github.com/eigenergy/PowerIO.jl) — Parser and data layer (see `docs/powerio-integration.md`)
+- [PowerIO.jl](https://github.com/eigenergy/PowerIO.jl) — Parser and data layer (see [PowerIO Integration](https://samueltalkington.com/research/powerdiff/powerio-integration/))
 - [JuMP.jl](https://github.com/jump-dev/JuMP.jl) — Optimization modeling
 - [ExaModels.jl](https://github.com/exanauts/ExaModels.jl) — Alternative optimization modeling for GPU parallelization
 - [Ipopt.jl](https://github.com/jump-dev/Ipopt.jl) — Default solver for DC and AC OPF

docs/make.jl

@@ -46,6 +46,7 @@ makedocs(
             "AC Power Flow" => "math/ac-power-flow.md",
             "AC Optimal Power Flow" => "math/ac-opf.md",
         ],
+        "PowerIO Integration" => "powerio-integration.md",
         "Advanced Topics" => "advanced.md",
         "API Reference" => "api.md",
     ],

docs/src/api.md

@@ -5,7 +5,6 @@
 ```@docs
 parse_file
 parse_matpower
-parse_matpower_struct
 get_path
 ```
 

docs/src/getting-started.md

@@ -7,14 +7,23 @@ This guide walks through the main workflows: DC power flow, DC OPF with LMP anal
 ```julia
 using PowerDiff
 
-# Parse a supported PowerIO case into a PowerIO.BalancedNetwork
+# Parse a case into a PowerIO module
 net = parse_file("case14.m")
 ```
 
-PowerDiff reads files through PowerIO. `parse_file` supports MATPOWER `.m`,
-PSS/E `.raw`, PowerWorld `.aux`, PowerModels JSON, and Egret JSON. For streams,
-pass `from`; JSON streams need `from=:egret` or `from=:powermodels`. The former
-PowerModels dictionary constructors were removed.
+PowerDiff reads files through PowerIO, and `parse_file` reads every transmission
+format the linked PowerIO library does — MATPOWER `.m`, PSS/E `.raw`, PowerWorld,
+PowerModels JSON, Egret JSON, pandapower, PyPSA, PSLF, gridfm, GO Challenge 3 and the
+rest. The format tokens are PowerIO's, so a reader PowerIO gains works here at once.
+
+A path's format is inferred from its extension; a stream has no extension, so pass
+`from` (MATPOWER is assumed otherwise). A bare `json` names a container rather than a
+reader, so name the one you mean: `from=:powermodels`, `:egret`, `:pandapower`.
+The former PowerModels dictionary constructors were removed.
+
+`parse_file` returns a `PowerIO.PioModule`, which carries the reader's findings on
+`m.diagnostics` alongside the case. See
+[PowerIO Integration](powerio-integration.md) for the whole seam.
 
 ## Interactive Exploration
 

docs/src/index.md

@@ -25,7 +25,7 @@ Pkg.add(url="https://github.com/grid-opt-alg-lab/PowerDiff.jl.git")
 ```julia
 using PowerDiff
 
-# Parse a supported PowerIO case into a PowerIO.BalancedNetwork
+# Parse a case into a PowerIO module
 net = parse_file("case14.m")
 dc_net = DCNetwork(net)
 d = calc_demand_vector(net)

docs/src/powerio-integration.md

@@ -0,0 +1,90 @@
+# PowerIO Integration
+
+PowerIO is PowerDiff's parser and data layer, and the only one. PowerDiff has no
+parser backend switch and no second representation of a case, and it derives no
+electrical quantity that PowerIO already states.
+
+## Parsing
+
+`PowerDiff.parse_file(path)` returns a `PowerIO.PioModule{PowerIO.BalancedNetwork}`.
+The module carries more than the network: `m.value` is the case, `m.diagnostics`
+holds the reader's findings, `m.sources[1].format` names the reader that ran, and
+`m.history` records what has been done to it. Pass the module, or the network
+inside it, to [`DCNetwork`](@ref) or [`ACNetwork`](@ref).
+
+For paths the extension selects the reader unless `from` is given. For streams
+pass `from`, because a stream has no extension; MATPOWER is assumed when neither
+is given.
+
+The format vocabulary is PowerIO's rather than a copy of it. PowerDiff normalizes
+its own historical short spellings (`:m`, `:raw`, `:aux`, `:pm`, `:powermodels`,
+`:egret`) and hands every other token straight over, so a reader PowerIO gains is
+reachable without a PowerDiff release, and an unknown token is answered by PowerIO
+with what the linked library actually reads. Two refusals stay local because
+PowerIO cannot make them: a bare `json` names a container rather than a reader, and
+a source that parses to anything other than a balanced transmission network is
+refused by naming what it holds.
+
+A failure inside PowerIO reaches the caller as a `PowerIO.PowerIOError`, which
+carries the diagnostic `code` and the records behind it. PowerDiff does not wrap
+it; wrapping would discard both.
+
+## What PowerIO states and what PowerDiff adds
+
+The network constructors read `PowerIO.to_powerdata`. It supplies per-unit powers
+and ratings, radian branch angles and angle bounds, per-bus aggregated load and
+shunt, right-aligned per-unit polynomial cost coefficients, each branch terminal's
+own charging admittance, and the four terminal admittance coefficients from which
+the series conductance and susceptance are read back exactly.
+
+It is **unfiltered**: every generator and branch row carries a `status`, isolated
+buses are present, and each row's `i` is its source row number. PowerDiff selects
+what it models -- rows whose status is set, on buses that are not isolated -- and
+that source row number becomes the [`IDMapping`](@ref) index, so an out-of-service
+row leaves a gap instead of renumbering the rows that remain.
+
+On top of that selection sit the four modeling decisions PowerIO leaves to its
+consumer:
+
+- polynomial cost interpretation, reading the cost model off the generator and
+  refusing piecewise linear costs. Higher-than-quadratic polynomials are refused by
+  `to_powerdata` itself.
+- a finite thermal limit when the source states none. MATPOWER spells that `0` and
+  PowerIO carries it out as `Inf`; both take the same synthesized limit, the largest
+  flow the endpoint voltage limits and the angle window physically admit.
+- default angle-difference bounds, mapping MATPOWER's `±360` and `0, 0` spellings to
+  a `±60°` window.
+- refusal of storage and HVDC records, which PowerDiff does not model.
+
+## What PowerDiff computes and why it is not a duplicate
+
+Two objects look like something PowerIO offers and are not.
+
+`admittance_matrix(net, sw)` assembles `Y` from `g`, `b`, `tap`, `shift` and the
+four terminal terms on every call. `PowerIO.calc_admittance_matrix` tabulates `Y`
+for one network at its parsed values with a boolean in-service flag; PowerDiff
+needs `Y` as a function of parameters a caller perturbs, because `sw` is continuous
+in `[0, 1]` and `calc_sensitivity(state, :vm, :b)` and its nine siblings
+differentiate through `g` and `b`. The same argument applies to the susceptance
+Laplacian `B = AᵀWA` with `W = -b .* sw`, and to the branch flow matrix.
+
+The branch-by-bus incidence matrix is stated once, in `_incidence_matrix`, and
+shared by both network types. `PowerIO.calc_incidence_matrix` covers in-service
+branches in table order against all buses and returns a bare sparse matrix with no
+branch or bus index map, so relabeling it into PowerDiff's sorted-source-id space
+costs more than stating it.
+
+## Reading what PowerIO found
+
+`m.diagnostics` is a `Vector{PowerIO.Diagnostic}`, available as soon as a case is
+parsed and without emitting it again. A `Diagnostic` is a record, not a line of
+text: branch on `d.code` and `d.severity`, and treat `d.message` as prose that
+carries no stability promise.
+
+```julia
+m = parse_file("case14.m")
+any(d -> d.severity === :warning, m.diagnostics)
+```
+
+PowerDiff logs none of it. Writing a case back out in another format goes through
+PowerIO directly, `PowerIO.emit(m, "psse", path)`.

examples/sens_examples/post_event_restoration.jl

@@ -24,9 +24,8 @@ const BASE_MVA = 100.0
 
 """Solve the IEEE300 DC OPF at supplied continuous branch switching states."""
 function solve_at_switching(network_data, demand, switching)
-    network = DCNetwork(network_data)
-    network.sw .= switching
-    problem = DCOPFProblem(network, demand)
+    problem = DCOPFProblem(DCNetwork(network_data), demand)
+    update_switching!(problem, switching)
     solution = solve!(problem)
     return problem, solution
 end

examples/sens_examples/post_event_restoration.md

@@ -10,9 +10,8 @@ julia --project=. -e 'using Pkg; Pkg.instantiate()'
 julia --project=. examples/sens_examples/post_event_restoration.jl
 ```
 
-PowerDiff currently requires PowerIO 0.8.2 or later within the compatible 0.8
-release series. If an older untracked `Manifest.toml` pins PowerIO 0.1.x, it
-must be regenerated from the current `Project.toml` before running the example.
+The example resolves whatever `Project.toml` pins. If an untracked `Manifest.toml`
+holds an older PowerIO, regenerate it from `Project.toml` before running.
 
 ## Example Problem Setup
 

src/PowerDiff.jl

@@ -103,7 +103,7 @@ export Sensitivity, silence
 export operand_symbols, parameter_symbols
 export jvp, vjp, jvp!, vjp!, dict_to_vec, vec_to_dict
 export kkt_layout, kkt_dims, kkt_indices
-export parse_file, parse_matpower, parse_matpower_struct, get_path
+export parse_file, parse_matpower, get_path
 
 # DC Power Flow Types
 export DCNetwork, DCPowerFlowState

src/prob/ac_opf_solve.jl

@@ -137,8 +137,11 @@ function _extract_ac_opf_solution(prob::ACOPFProblem{JuMPBackend})
     # Extract dual variables - generation bounds (inequality)
     ρ_pg_lb = [dual(LowerBoundRef(prob.pg[i])) for i in 1:k]
     ρ_pg_ub = [dual(UpperBoundRef(prob.pg[i])) for i in 1:k]
-    ρ_qg_lb = [dual(LowerBoundRef(prob.qg[i])) for i in 1:k]
-    ρ_qg_ub = [dual(UpperBoundRef(prob.qg[i])) for i in 1:k]
+    # An absent reactive limit has no bound reference and no multiplier. Zero is not a
+    # stand-in there: it is the value the KKT system pins the multiplier of an absent
+    # constraint to, so the solved duals and the residual agree by construction.
+    ρ_qg_lb = [has_lower_bound(prob.qg[i]) ? dual(LowerBoundRef(prob.qg[i])) : 0.0 for i in 1:k]
+    ρ_qg_ub = [has_upper_bound(prob.qg[i]) ? dual(UpperBoundRef(prob.qg[i])) : 0.0 for i in 1:k]
 
     # Extract dual variables - flow variable bounds (inequality)
     σ_p_fr_lb = zeros(m)