diff --git a/README.md b/README.md index 5717084a..1ae56445 100644 --- a/README.md +++ b/README.md @@ -344,12 +344,12 @@ field, standard dense output and I control, and stops VMEC orbits at the LCFS. and retains its existing electric-field, stopping and exterior-continuation interfaces. The generic `essos.rk.solve_rk` also accepts an arbitrary array RHS. -For Boozer, use `trace_boozer(..., integrator="essos_dopri8", -native_execution="cuda")`. Its CUDA adapter supports fixed cosine Fourier -fields with K=0; the portable path also supports sine spectra, angle tables -and PI. GPU arrays are private solver views of the original field data; -the public field classes are unchanged. This addition is forward-only: -split collisions and accepted-step reverse AD belong to separate changes. +For Boozer, use `trace_boozer(..., integrator="essos_dopri8")`. Boozer runs on +the portable path only (`native_execution="cuda"` raises): a fused kernel was +slower than the vmapped one. If the installed JAX cannot lower the fused kernel, +`prepare_solver(..., execution="cuda")` warns and uses the portable path. GPU +arrays are private solver views of the original field data; the public field +classes are unchanged. ### Forward comparison @@ -365,8 +365,6 @@ VMEC/Boozer use reactor-scale QA at 4 keV and 20 us. Compare backends within row | Field / model | Stepping | Native / Diffrax warm ms | Native / Diffrax buffers MiB | Measured | |---|---|---:|---:|---| -| boozer / gc | adaptive | 1.659 / 3.769 | 0.026 / 0.151 | 2026-10-26 | -| boozer / gc | fixed | 18.201 / 50.142 | 0.026 / 0.154 | 2026-10-26 | | coils / gc | adaptive | 84.426 / 160.362 | 0.026 / 5.980 | 2026-10-24 | | coils / gc | fixed | 419.872 / 676.766 | 0.026 / 5.970 | 2026-10-24 | | coils / lorentz | adaptive | 29.429 / 70.426 | 0.070 / 0.731 | 2026-10-24 | @@ -381,60 +379,23 @@ Boozer; velocity errors include both absolute and birth-speed-normalized values against a tighter Diffrax reference. The measured gains include the private field adapters and describe the complete forward implementation. -Generate all eight comparison workloads and their files on a CUDA machine: +Time the shared and CUDA forward workloads on a CUDA machine: ```sh -python -m examples.benchmarks.benchmark_native_orbits --suite \ - --execution cuda --output output/native_forward +for field in coils vmec; do + python -m examples.benchmarks.benchmark_native --suite orbit --field $field \ + --execution both --out output/native_forward/$field +done ``` -Choose a fresh output folder. The script creates per-workload `comparison.json` -and `comparison.csv`, plus aggregate `forward.csv` and `forward.md`. Generated -results are kept under `output/` and are not versioned. Three blocked warm -samples alternate backend order after startup; setup, compilation and the tighter -reference are excluded. Compilation may use a persistent cache and is not -reported as a cold measurement. Reports include inputs, source hashes, step -counts, failures, error metrics and compiler buffers. - -Defaults reproduce the table's 64 particles, 4 keV, seed 42, eight saves, -rtol=atol=1e-9 and 20 us GC workloads. Fixed VMEC/Boozer use 160 steps, coil GC -224, and coil Lorentz 1792 over 2 us. For one workload, omit `--suite` and select -`--field vmec`, `--field boozer` or `--field coils --model gc/lorentz`, with -`--stepping adaptive/fixed`. A matching Boozer transform can be supplied with -`--field-cache`; otherwise the optional `essos[booz]` dependency computes it. - -### QA Boozer alpha workload - -Previously recorded full workload: 8192 alphas, 3.52 MeV, birth s=0.3, -seed 42, 20 ms, 101 saves, 16 direct Fourier modes. The preceding profiling -run identified an A100/JAX 0.7.2; the full console logs identify a CUDA device. -Compilation and warm execution are separated. This is a CATAPULT-sized -workload, not a new measurement against CATAPULT. - -| Method | Control | Compile s | Warm s | Accepted / rejected | Lost / failed | -|---|---|---:|---:|---:|---:| -| Native Dopri5 | I, rtol=atol=1e-10 | 4.247 | 21.554 | 895113025 / 63143676 | 110 / 0 | -| Native Dopri8 | I, rtol=atol=1e-10 | 5.069 | 18.448 | 351025279 / 62226975 | 111 / 0 | -| Native Dopri8 | I, rtol=atol=1e-8 | 4.985 | 9.223 | 189259503 / 7909253 | 106 / 0 | -| Existing RK4 | Fixed dt=1.25e-7 s | 2.728 | 20.244 | 1310720000 grid slots / 0 | 108 / 0 | - -These full-workload measurements predate the current extraction and have not -been rerun after the VMEC and initialization updates. RK4 checks loss at endpoints; -native adaptive methods refine the LCFS crossing. The differing loss counts are -not an accuracy proof. Regenerate the workload with the script below when ready. - -Run on the machine's normal JAX backend (the alpha example selects CUDA -execution automatically when available): - -```sh -python -m examples.benchmarks.benchmark_native_boozer_alpha \ - --integrator essos_dopri8 --particles 8192 --horizon .02 \ - --rtol 1e-10 --atol 1e-10 --output output/native_boozer_alpha -``` - -The example saves its Boozer transform, births, trajectories, statistics, -timings and a diagnostic plot. Reuse its `boozer_inputs.npz` with -`--field-cache`; select `essos_rk45` or `rk4` for the corresponding baseline. +Choose a fresh output folder per workload. The script writes `result.json` (compile time, +first execution, warm median, particles per second, compiler buffers, peak memory, lost and +failed counts) and, with `--save-states`, `states.npz`. Generated results are kept under +`output/` and are not versioned. The Diffrax comparison and error columns of the table above +came from an earlier script that has been removed. Defaults are 64 particles, 4 keV, seed 42, +eight saves and rtol=atol=1e-9; fixed steps default to 224 over 20 us (GC) and 1792 over 2 us +(`--model lorentz`). Select `--stepping adaptive/fixed`. The Boozer field needs the optional +`essos[booz]` dependency and always uses the portable JAX path. ## Native split orbit and collisions @@ -450,37 +411,10 @@ result = trace_split(field, particles, species, tmax=tmax, timestep=dt) # check The interval and tolerance controls are local; refine `timestep` and `collision_dtmax` to check the splitting error. VMEC fields use their own geometry, Boozer fields need -`spatial_geometry=BoozerGeometry.from_booz_xform(booz)`. `execution="independent"` runs one -persistent GPU kernel per particle (float64, Dopri8, zero electric field); `interpret=True` runs it -on CPU. `tests/test_native_split.py` checks Spitzer slowing down, the Lorentz eigenvalues, +`spatial_geometry=BoozerGeometry.from_booz_xform(booz)`. `tests/test_native_split.py` checks Spitzer slowing down, the Lorentz eigenvalues, Ito/Stratonovich consistency and the absent-bath and lost-particle limits; `examples/particle_tracing_collisions/native_split_slowing_down.py` prints the Spitzer comparison. - -## Accepted-map native reverse AD - -`solve_rk(..., adjoint="accepted")` and `trace_split(..., adjoint="accepted")` give gradients of the -accepted-step map: the forward solve keeps accepted intervals, Wiener increments, rejection decisions -and event fractions, and reverse differentiates the stages, dense saves and collision maps -([essos/native_ad.py](essos/native_ad.py)). First-hit times and the adaptive mesh are not differentiated, -so a finite difference that reruns adaptation measures a different map. - -```python -def objective(dofs): - out = trace_split(field(dofs), particles, species, tmax=tmax, timestep=dt, adjoint="accepted", - adjoint_max_accepted_steps=4096, adjoint_max_orbit_steps=4096) - return loss(out.ys) -gradient = jax.grad(objective)(dofs) -``` - -Collisionless orbits use `Tracing(..., native_adjoint="accepted")`. The tape capacities -`adjoint_max_accepted_steps` and `adjoint_max_orbit_steps` raise during reverse when exceeded; -`adjoint_checkpoint_steps` (default 16) trades replay workspace against stored checkpoints and does not -change the gradient. Direct coil fields can use the persistent GPU reverse kernel with -`execution="independent"`; VMEC and Boozer fields use checkpointed JAX. `tests/test_native_ad.py` checks -gradients against forward mode and finite differences through the frozen schedule, the adjoint identity, -checkpoint independence and the persistent kernel against shared reverse. -`examples/coil_optimization/optimize_coils_native_ad.py` minimizes the final minor radius of alpha -guiding centers over the coil dofs. `examples/benchmarks/benchmark_native_ad.py` times value-and-gradient against forward. +`examples/benchmarks/benchmark_native.py` times the orbit and collision suites (`--help`). ## Testing diff --git a/essos/dynamics.py b/essos/dynamics.py index 177e5877..fa09e9e1 100644 --- a/essos/dynamics.py +++ b/essos/dynamics.py @@ -1055,13 +1055,7 @@ def __init__(self, trajectories_input=None, initial_conditions=None, times_to_tr max_steps=1_000_000, exterior_field=None, wall=None, max_returns=16, reentry_depth=None, particle_batch_size=None, integrator="diffrax", native_error_control="physics", orbit_length_scale=1., orbit_speed_scale=None, - native_controller="i", native_pcoeff=.4, native_icoeff=.3, - native_adjoint="none", native_adjoint_checkpoint_steps=16, - native_adjoint_max_accepted_steps=None, native_adjoint_parameter_accumulation="segment", adjoint=None): - - self.native_adjoint, self.native_adjoint_checkpoint_steps = native_adjoint, native_adjoint_checkpoint_steps - self.native_adjoint_max_accepted_steps = native_adjoint_max_accepted_steps - self.native_adjoint_parameter_accumulation = native_adjoint_parameter_accumulation + native_controller="i", native_pcoeff=.4, native_icoeff=.3, adjoint=None): from essos.rk import _configure_tracing as configure configure(self, integrator, model, solver, native_error_control, orbit_length_scale, diff --git a/essos/native_ad.py b/essos/native_ad.py deleted file mode 100644 index 5d0eba29..00000000 --- a/essos/native_ad.py +++ /dev/null @@ -1,1577 +0,0 @@ -"""Checkpointed reverse AD of accepted native orbit and collision maps. - -Controllers, rejected trials, event fractions and recorded Wiener increments -are frozen. Stage physics, collision coefficients and spatial kicks remain -differentiable. Only one checkpoint segment's stage residuals is live. -""" -from typing import NamedTuple -import equinox as eqx -import jax -import jax.numpy as jnp -import numpy as np -from jax import lax, tree_util, ad_checkpoint -from jax._src.interpreters import partial_eval -from jax.extend.core import ClosedJaxpr, Literal -from jax.custom_derivatives import SymbolicZero, custom_vjp_primal_tree_values -from .rk import (solve_rk, _trial_step, _C, _A, _ERROR, - _C8, _A8, _ERROR8, _rk45_dense, _dopri8_dense, _RKTrial, RKStatus, - _evaluate_inline, _bind_primitive) -from .split import solve_split, SplitStatus - -def gpu_jit_options(): - """Compile guarded GPU reverse without CUDA command-buffer capture. - - Apply to the outer JIT; the solver and its replay remain on the GPU. - """ - return ({'compiler_options':{'xla_gpu_enable_command_buffer':''}} - if jax.default_backend()=='gpu' else {}) - -def _tableau(method): - """Nodes, stage rows and error weights, with the matching dense output.""" - if method == "dopri5": - return (_C, _A, _ERROR), _rk45_dense - return (_C8, _A8, _ERROR8), _dopri8_dense - -def _contains_custom_rule(graph): - """Find opaque derivative rules, including calls nested in control flow.""" - for equation in graph.eqns: - if equation.primitive.name.startswith("custom_"): - return True - for value in tree_util.tree_leaves(equation.params): - nested = getattr(value, "jaxpr", value) - if hasattr(nested, "eqns") and _contains_custom_rule(nested): - return True - return False - - -def _partition_graph(graph, unknowns): - # Partial evaluation can replace a nested custom-JVP call by its primal - # graph. In that case only partition whole primitive calls by dependency; - # never transform their bodies. Parameter-only calls still move to setup, - # avoiding pullbacks of fixed discretization data inside the stage VJP. - if _contains_custom_rule(graph.jaxpr): - if not all(unknowns): - raise ValueError("opaque RHS partition expects state/time inputs") - examples = [jax.ShapeDtypeStruct(var.aval.shape, var.aval.dtype) for var in graph.jaxpr.invars] - original = graph - graph = jax.make_jaxpr(lambda *values: _evaluate_inline( - original.jaxpr, original.consts, *values))(*examples) - body = graph.jaxpr - unknown = set(body.invars) - setup_equations, stage_equations = [], [] - for equation in body.eqns: - dependent = bool(equation.effects) or any( - not isinstance(var, Literal) and var in unknown for var in equation.invars) - (stage_equations if dependent else setup_equations).append(equation) - if dependent: - unknown.update(equation.outvars) - residuals, seen = [], set() - for var in [var for equation in stage_equations for var in equation.invars]+list(body.outvars): - if not isinstance(var, Literal) and var not in unknown and var not in seen: - residuals.append(var) - seen.add(var) - setup = body.replace(invars=[], outvars=residuals, eqns=setup_equations, - effects=set().union(*(eq.effects for eq in setup_equations))) - stage = body.replace(constvars=[], invars=residuals+list(body.invars), - eqns=stage_equations, effects=set().union(*(eq.effects for eq in stage_equations))) - return ClosedJaxpr(setup, graph.consts), ClosedJaxpr(stage, ()) - setup, stage, _, _ = partial_eval.partial_eval_jaxpr_nounits( - graph, unknowns, instantiate=True) - return setup, stage - - -def _partition_rhs(rhs, t0, y0, args): - """Split solve-invariant setup from time/state-dependent stage arithmetic.""" - graph = jax.make_jaxpr(lambda t, y: rhs(t, y, args))(t0, y0) - return _partition_graph(graph, [True, True]) - - -class _RKCheckpointTape(NamedTuple): - """Accepted step sizes and segment starts; no per-step state/save tape.""" - step_sizes: jax.Array - times: jax.Array - states: jax.Array - derivatives: jax.Array - fraction: jax.Array - final_time: jax.Array - initial_saves: jax.Array - count: jax.Array - status: jax.Array - - -class _RKSchedule(NamedTuple): - """Accepted schedule and optional sparse checkpoints for a containing split solve.""" - times: jax.Array - fraction: jax.Array - count: jax.Array - states: object = None - derivatives: object = None - - -def _geometry_pullback(values,seed): - """Transpose every prepared input, including independent tangent aliases. - - The original setup pullback combines the tangent views and handles coil - symmetry/current scaling. Diagonal placeholders retain their own bars. - Coordinate-wise matrix products fuse with the geometry operations here. - """ - point,gamma,ly,lz,lx,weights,z0,tangent,z1,z2 = values - u,matrix = seed - r = (point-gamma).T - cross_tangent = jnp.stack([lx,ly,lz]) - radius = jnp.linalg.norm(r,axis=0) - inverse3 = 1./radius**3 - inverse5 = inverse3/radius**2 - inverse7 = inverse5/radius**2 - cross = jnp.cross(cross_tangent,r,axisa=0,axisb=0,axisc=0) - mr = jnp.stack([matrix[i,0]*r[0]+matrix[i,1]*r[1]+matrix[i,2]*r[2] for i in range(3)]) - mtc = jnp.stack([matrix[0,i]*cross[0]+matrix[1,i]*cross[1]+matrix[2,i]*cross[2] for i in range(3)]) - skew = jnp.array([matrix[2,1]-matrix[1,2],matrix[0,2]-matrix[2,0],matrix[1,0]-matrix[0,1]])[:,None,None] - u = u[:,None,None] - k = jnp.sum(u*cross+skew*tangent,axis=0) - d = jnp.sum(cross*mr,axis=0) - dr = (jnp.cross(u,cross_tangent,axisa=0,axisb=0,axisc=0)*inverse3 - -3*(k*r+jnp.cross(mr,cross_tangent,axisa=0,axisb=0,axisc=0)+mtc)*inverse5 - +15*d*r*inverse7) - dl = (jnp.cross(r,u,axisa=0,axisb=0,axisc=0)*inverse3 - -3*jnp.cross(r,mr,axisa=0,axisb=0,axisc=0)*inverse5) - factor = weights/gamma.shape[1] - dr,dl = dr*factor,dl*factor - bars = (jnp.sum(dr,axis=(1,2)),-dr.T,dl[1],dl[2],dl[0], - jnp.sum(k*inverse3-3*d*inverse5,axis=0)/gamma.shape[1], - matrix[0,0].reshape((1,)),skew*inverse3*factor, - matrix[1,1].reshape((1,)),matrix[2,2].reshape((1,))) - return tuple(bar.astype(value.dtype) for bar,value in zip(bars,values)) - - -def _supported_field(closed, jacobian): - from .fields import BiotSavart - body = closed.jaxpr - name = "B_and_dB_by_dX" if jacobian else "B" - original = getattr(getattr(BiotSavart, name, None), "__wrapped__", None) - if original is None: - return False - source = original.__code__ - expected = f"{name} at {source.co_filename}:{source.co_firstlineno}" - if (getattr(getattr(body,"debug_info",None),"func_src_info",None) != expected - or body.effects or len(body.invars) != (10 if jacobian else 6)): - return False - shapes = [var.aval.shape for var in body.invars] - if len(shapes[1]) != 3 or shapes[1][-1] != 3: - return False - coils,nodes,_ = shapes[1] - if coils < 1 or nodes < 1: - return False - expected_shapes = [(3,),(coils,nodes,3),(nodes,coils),(nodes,coils),(nodes,coils),(coils,)] - if jacobian: - expected_shapes += [(1,),(3,nodes,coils),(1,),(1,)] - if shapes != expected_shapes or [var.aval.shape for var in body.outvars] != ([(3,),(3,3)] if jacobian else [(3,)]): - return False - if any(var.aval.dtype != jnp.dtype('float64') for var in body.invars): - return False - cross = [eq for eq in body.eqns if eq.primitive.name == 'jit' and eq.params.get('name') == 'cross'] - return len(cross) == 1 and list(cross[0].invars[1:]) == list(body.invars[2:5]) - - -def _B_pullback(values, seed): - """Exact field-only transpose, retaining all six independent input bars. - - Coordinate products expose the small cross products to fusion. The setup - pullback still combines tangent views and maps geometry/current weights to - the original coil DOFs; no physical inputs are frozen here. - """ - point, gamma, ly, lz, lx, weights = values - r = (point-gamma).T - inverse3 = 1./jnp.linalg.norm(r, axis=0)**3 - inverse5 = inverse3/jnp.sum(r*r, axis=0) - factor = weights/gamma.shape[1] - u = seed[0][:, None, None] - c0, c1, c2 = ly*r[2]-lz*r[1], lz*r[0]-lx*r[2], lx*r[1]-ly*r[0] - k = u[0]*c0+u[1]*c1+u[2]*c2 - dr = jnp.stack(( - factor*((u[1]*lz-u[2]*ly)*inverse3-3*k*r[0]*inverse5), - factor*((u[2]*lx-u[0]*lz)*inverse3-3*k*r[1]*inverse5), - factor*((u[0]*ly-u[1]*lx)*inverse3-3*k*r[2]*inverse5))) - dl = jnp.stack((r[1]*u[2]-r[2]*u[1], r[2]*u[0]-r[0]*u[2], - r[0]*u[1]-r[1]*u[0]))*inverse3*factor - bars = (jnp.sum(dr, axis=(1, 2)), -dr.T, dl[1], dl[2], dl[0], - jnp.sum(k*inverse3, axis=0)/gamma.shape[1]) - return tuple(bar.astype(value.dtype) for bar, value in zip(bars, values)) - - -def has_prepared_biot_savart(graph): - """Recognize ordinary JIT calls without entering custom/control-flow bodies.""" - for equation in graph.eqns: - if equation.primitive.name == "jit": - nested = equation.params["jaxpr"] - name = equation.params.get("name") - if name in ("B", "B_and_dB_by_dX") and _supported_field(nested, name != "B"): - return True - if has_prepared_biot_savart(nested.jaxpr): - return True - return False - - -def _field_call(closed, values, jacobian=True): - """Keep original values and use the exact transpose to define a JVP. - - Transposing the linear pullback supplies the tangent without freezing its - geometry coefficients. Ordinary JAX AD still provides higher derivatives. - """ - pullback = _geometry_pullback if jacobian else _B_pullback - @jax.custom_jvp - def evaluate(*values): - return tuple(jax.core.eval_jaxpr(closed.jaxpr,closed.consts,*values)) - @evaluate.defjvp - def tangent(primals,tangents): - value = evaluate(*primals) - seed_shape = tree_util.tree_map(jnp.zeros_like,value) - transpose = jax.linear_transpose(lambda seed:pullback(primals,seed),seed_shape) - return value,transpose(tangents)[0] - # Expose only the small field outputs to the outer checkpoint policy. - # A custom JVP otherwise hides its internal field reductions from it. - return tuple(ad_checkpoint.checkpoint_name(value, "native_biot_savart_output") - for value in evaluate(*values)) - - -def _rewrite(graph,constants,*arguments): - env = dict(zip(graph.constvars,constants)) - env.update(zip(graph.invars,arguments)) - def read(var): - return var.val if isinstance(var,Literal) else env[var] - for eq in graph.eqns: - values = [read(var) for var in eq.invars] - with eq.ctx.manager: - if eq.primitive.name == 'jit': - closed = eq.params['jaxpr'] - name = eq.params.get('name') - if name in ('B', 'B_and_dB_by_dX') and _supported_field(closed, name != 'B'): - result = _field_call(closed, values, name != 'B') - else: - result = _rewrite(closed.jaxpr,closed.consts,*values) - else: - result = _bind_primitive(eq.primitive, values, eq.params) - if not eq.primitive.multiple_results: - result = [result] - env.update(zip(eq.outvars,result)) - return [read(var) for var in graph.outvars] - - -def prepare_biot_savart_rhs(prepare,evaluate,t,y,raw): - """Specialize only the prepared reverse RHS; preserve the original setup.""" - parameters = prepare(raw) - graph = jax.make_jaxpr(evaluate)(t,y,parameters) - if not has_prepared_biot_savart(graph.jaxpr): - return None - def rhs(t,y,parameters): - return _rewrite(graph.jaxpr,graph.consts,t,y,*tree_util.tree_leaves(parameters))[0] - # Materialize the transformed graph now so the selection is deterministic. - changed = jax.make_jaxpr(rhs)(t,y,parameters) - def prepared_rhs(t,y,parameters): - return jax.core.eval_jaxpr(changed.jaxpr,changed.consts,t,y,*tree_util.tree_leaves(parameters))[0] - prepared_rhs._native_biot_savart_transpose = True - return prepare,prepared_rhs - - -def prepare_reverse_rhs(prepare, evaluate, t, y, raw, *, specialize_biot_savart=True): - """Repartition the same RHS after exposing jit boundaries to reverse AD. - - This is a reverse-only representation: the adaptive forward driver keeps - its original prepared RHS. Field objects and their own AD rules are unchanged. - Stock prepared Biot--Savart B and B/J calls use exact geometry transposes. - RHSs containing custom derivative primitives retain their original - preparation, including rules nested in jit or control-flow calls. - """ - original = jax.make_jaxpr(lambda t,y,p: evaluate(t,y,prepare(p)))(t,y,raw) - if _contains_custom_rule(original.jaxpr): - return prepare, evaluate - if specialize_biot_savart: - if has_prepared_biot_savart(original.jaxpr): - specialized = prepare_biot_savart_rhs(prepare, evaluate, t, y, raw) - if specialized is not None: - return specialized - if not any(eq.primitive.name == "jit" for eq in original.jaxpr.eqns): - return prepare, evaluate - graph = jax.make_jaxpr(lambda t,y,p: _evaluate_inline(original.jaxpr, original.consts, - t,y,*tree_util.tree_leaves(p))[0])(t,y,raw) - setup, stage, _, _ = partial_eval.partial_eval_jaxpr_nounits( - graph, [True, True]+[False]*len(tree_util.tree_leaves(raw)), instantiate=True) - - def prepare_values(raw): - return tuple(jax.core.eval_jaxpr(setup.jaxpr, setup.consts, *tree_util.tree_leaves(raw))) - - def rhs(t,y,parameters): - return jax.core.eval_jaxpr(stage.jaxpr, stage.consts, *parameters,t,y)[0] - - return prepare_values, rhs - - -def prepare_rhs_linearization(rhs, t, y, parameters): - """Return ``(evaluate, pullback, initial_cache)`` or None for cheap RHSs. - - ``evaluate(t, y)`` returns the original RHS and compact residuals. - ``pullback(cache, seed)`` returns state and prepared-parameter cotangents. - The initial cache is used only for the first accepted step's initial FSAL - derivative. Other steps reuse the preceding endpoint derivative in AD. - Named small Biot--Savart outputs remain visible even when their reductions - are hidden behind a custom JVP. Large geometry arrays are rematerialized. - """ - selected = [] - - def save_compact_value(primitive, *avals, **params): - if primitive.name == "name" and params.get("name") == "native_biot_savart_output": - keep = len(avals) == 1 and np.prod(avals[0].shape) <= 12 - if keep: - selected.append(True) - return bool(keep) - if primitive.name != "reduce_sum": - return False - shape = avals[0].shape - output_size = np.prod([size for axis, size in enumerate(shape) if axis not in params["axes"]]) - keep = np.prod(shape) > 9 and output_size <= 9 - if keep: - selected.append(True) - return bool(keep) - - remat_rhs = jax.checkpoint(rhs, policy=save_compact_value, prevent_cse=False) - transpose_graph = [] - - def linearize(t, y, values): - value, pullback = jax.vjp(lambda y, p: remat_rhs(t, y, p), y, values) - graph, shape = jax.make_jaxpr(pullback, return_shape=True)(jnp.zeros_like(value)) - transpose_graph[:] = [(graph.jaxpr, tree_util.tree_structure(shape))] - return value, tuple(graph.consts) - - graph = jax.make_jaxpr(linearize)(t, y, parameters) - if not selected: - return None - # This is value dependence for storing residuals, not a stop-gradient on - # physical parameters: the transpose still returns every parameter bar. - setup, stage, unknown, _ = partial_eval.partial_eval_jaxpr_nounits( - graph, [True, True]+[False]*len(tree_util.tree_leaves(parameters)), - instantiate=[True]+[False]*(len(graph.out_avals)-1)) - values = jax.core.eval_jaxpr(setup.jaxpr, setup.consts, *tree_util.tree_leaves(parameters)) - fixed_count = sum(not flag for flag in unknown) - constants, bindings = values[:fixed_count], values[fixed_count:] - - descriptors, groups = [], {} - for aval in stage.out_avals[1:]: - dtype = aval.dtype - size = int(np.prod(aval.shape)) - group = len(groups) if dtype not in groups else list(groups).index(dtype) - offset = groups.get(dtype, 0) - groups[dtype] = offset+size - descriptors.append((group, offset, size, aval.shape)) - - def evaluate(t, y): - out = jax.core.eval_jaxpr(stage.jaxpr, stage.consts, *bindings, t, y) - buffers = [[] for _ in groups] - for value, (group, _, _, _) in zip(out[1:], descriptors): - buffers[group].append(jnp.ravel(value)) - return out[0], tuple(jnp.concatenate(values) for values in buffers) - - transpose, structure = transpose_graph[0] - - def pullback(packed, seed): - residuals = [packed[group][offset:offset+size].reshape(shape) - for group, offset, size, shape in descriptors] - fixed, variable = iter(constants), iter(residuals) - values = [next(variable) if flag else next(fixed) for flag in unknown[1:]] - out = jax.core.eval_jaxpr(transpose, values, seed) - return tree_util.tree_unflatten(structure, out) - - return evaluate, pullback, evaluate(t, y)[1] - - -def _zeros(tree): - def zero(x): - x = jnp.asarray(x) - dtype = x.dtype if jnp.issubdtype(x.dtype, jnp.inexact) else jax.dtypes.float0 - return jnp.zeros(x.shape, dtype=dtype) - return tree_util.tree_map(zero, tree) - - -def _add(left, right): - return tree_util.tree_map(lambda a, b: a if a.dtype == jax.dtypes.float0 else a+b, left, right) - - -class _ParameterActivity(eqx.Module): - """Static JAX activity flags, distinct from the frozen time schedule.""" - flags: tuple = eqx.field(static=True) - - -def _convert_all(function, *examples): - """Expose every graph constant, including outer jit tracers, as an input.""" - graph, shape = jax.make_jaxpr(function, return_shape=True)(*examples) - structure = tree_util.tree_structure(shape) - def evaluate(arguments, constants): - flat = jax.core.eval_jaxpr(graph.jaxpr, constants, *tree_util.tree_leaves(arguments)) - return tree_util.tree_unflatten(structure, flat) - return evaluate, graph.consts - - -def _prepared_activity(prepare, active): - """Exclude fixed tables from parameter transposes before tracing stages.""" - graph=jax.make_jaxpr(prepare)(active) - live=set(graph.jaxpr.invars) - for equation in graph.jaxpr.eqns: - if equation.primitive.name!='stop_gradient' and any( - not isinstance(v,Literal) and v in live for v in equation.invars): - live.update(equation.outvars) - return tuple(not isinstance(v,Literal) and v in live for v in graph.jaxpr.outvars) - - -def _freeze_fixed(values, flags): - leaves,structure=tree_util.tree_flatten(values) - return tree_util.tree_unflatten(structure, - [value if active else lax.stop_gradient(value) for value,active in zip(leaves,flags)]) - - -def _step_pullback(rhs, y, t, h, endpoint, increments, parameters, - fraction, sample_fractions, owned, by, bf, saved_bar, method, - initial_fsal=False, cached_pullback=None): - """Transpose the RK recurrence from reconstructed primal stage values. - - Segment reconstruction supplies the primal stage values. Linear RK/dense - algebra is transposed separately; each RHS VJP retains the physical - derivatives at the original stage time/state. Stock prepared Biot--Savart - calls can use the exact native field linearization. Only one RHS transpose's temporary arrays are - live at a time, rather than every stage's field intermediates. The reverse - recurrence includes all stage dependencies; stages are not frozen in AD. - """ - (nodes, rows, _), dense = _tableau(method) - def outputs(y, endpoint, increments, parameters): - terminal = dense(y, endpoint, increments, fraction) - samples = jax.vmap(lambda s: dense(y, endpoint, increments, s))(sample_fractions) - return terminal, samples - _, output_vjp = jax.vjp(outputs, y, endpoint, increments, parameters) - dy, dy1, dk, dense_parameters = output_vjp((by, jnp.where(owned[:, None], saved_bar, 0.))) - dp = dense_parameters - - dy = dy+dy1 - bank = increments[:-1] - dk = dk.at[:-1].add(jnp.asarray(rows[-1], y.dtype)[:, None]*dy1) - coefficients = np.zeros((len(increments), len(bank))) - for i, row in enumerate(rows, start=1): - coefficients[i, :len(row)] = row - coefficients = jnp.asarray(coefficients, y.dtype) - times = jnp.asarray((0.,)+nodes, y.dtype) - - def stage(reverse_index, carry): - dy, dk, dp = carry - i = len(increments)-1-reverse_index - weights = coefficients[i] - point = y+jnp.tensordot(weights, bank, axes=1) - point = jnp.where(i == len(increments)-1, endpoint, jnp.where(i == 0, y, point)) - # The endpoint FSAL derivative is unscaled; its RK increment also - # contributes to dense output. Stage zero is visited only on the first - # accepted step, where the initial FSAL value is an actual RHS call. - seed = h*dk[i]+jnp.where(i == len(increments)-1, bf, 0.) - if cached_pullback is None: - _, rhs_vjp = jax.vjp(lambda state,p: rhs(t+times[i]*h,state,p), point, parameters) - dpoint, dparams = rhs_vjp(seed) - else: - dpoint, dparams = cached_pullback(i, seed) - return dy+dpoint, dk.at[:-1].add(weights[:, None]*dpoint), _add(dp, dparams) - bound = len(increments)-1+jnp.asarray(initial_fsal, jnp.int32) - dy, dk, dp = lax.fori_loop(jnp.asarray(0, jnp.int32), bound, stage, (dy, dk, dp)) - return dy, jnp.where(initial_fsal, 0., h*dk[0]), dp - - -def _trial_with_residuals(linearize, t, y, h, f0, constants, initial_cache): - """Replay the unchanged RK stages and keep compact RHS residuals. - - Used only in backward reconstruction; no embedded error is needed here. - Stage zero's cache belongs to the initial trajectory RHS. It is consumed - only when ``initial_fsal`` is true in the first accepted step's pullback. - """ - nodes, rows, error_weights = constants - n = len(error_weights) - coefficients = np.zeros((n-2, n-1)) - for i, row in enumerate(rows[:-1]): - coefficients[i, :len(row)] = row - coefficients = jnp.asarray(coefficients, y.dtype) - times = jnp.asarray(nodes[:-1], y.dtype) - bank = jnp.concatenate([(h*f0)[None], jnp.zeros((n-2,)+y.shape, y.dtype)]) - cache = tree_util.tree_map(lambda r: jnp.zeros((n,)+r.shape, r.dtype).at[0].set(r), initial_cache) - - def stage(i, values): - bank, cache = values - point = y+jnp.tensordot(coefficients[i-1], bank, axes=1) - f, residuals = linearize(t+times[i-1]*h, point) - return lax.dynamic_update_index_in_dim(bank, h*f, i, axis=0), tree_util.tree_map( - lambda buffer, r: lax.dynamic_update_index_in_dim(buffer, r, i, axis=0), cache, residuals) - - bank, cache = lax.fori_loop(1, n-1, stage, (bank, cache)) - endpoint = y+jnp.tensordot(jnp.asarray(rows[-1], y.dtype), bank, axes=1) - final, residuals = linearize(t+h, endpoint) - cache = tree_util.tree_map(lambda buffer, r: buffer.at[-1].set(r), cache, residuals) - trial = _RKTrial(endpoint, jnp.zeros_like(y), jnp.concatenate([bank, (h*final)[None]]), final) - return trial, cache - - -def _reverse_segment(rhs, parameters, linearization, method, steps, extras, start, stop, y, f, t0, by, bf, dp, - pull, initial_cache, segment, first): - """Rebuild the RK stages of accepted steps ``[start, stop)`` from one checkpoint, then transpose them. - - ``steps(i, t)`` gives the step's ``(time, size)`` (``t`` is the running time, starting at ``t0``) and - ``extras(i, t, h)`` its ``(fraction, sample_fractions, owned, saved_bar)``. ``pull`` maps a - step's prepared-parameter bar into the accumulator ``dp``. Stage vectors and compact RHS - residuals of one segment are live at a time; ``first`` marks the step that owns the initial FSAL. - """ - constants = _tableau(method)[0] - n, stages = y.size, len(constants[2]) - banks = (jnp.zeros((segment, n), y.dtype), jnp.zeros((segment, n), y.dtype), - jnp.zeros((segment, stages, n), y.dtype), jnp.zeros((segment, 2), y.dtype), - tree_util.tree_map(lambda r: jnp.zeros((segment, stages)+r.shape, r.dtype), initial_cache)) - - def reconstruct(c): - i, t, y, f, (states, endpoints, increments, times, cache) = c - slot = i-start - t, h = steps(i, t) - if linearization is None: - trial, residuals = _trial_step(rhs, t, y, h, parameters, f, *constants), () - else: - trial, residuals = _trial_with_residuals(linearization[0], t, y, h, f, constants, initial_cache) - cache = tree_util.tree_map(lambda buffer, r: buffer.at[slot].set(r), cache, residuals) - # An event can only end the last accepted step; no later slot consumes its full endpoint. - return i+1, t+h, trial.y, trial.f1, (states.at[slot].set(y), endpoints.at[slot].set(trial.y), - increments.at[slot].set(trial.increments), times.at[slot].set(jnp.stack([t, h])), cache) - - *_, (states, endpoints, increments, times, cache) = lax.while_loop( - lambda c: c[0] < stop, reconstruct, (start, t0, y, f, banks)) - - def reverse_step(c): - i, by, bf, dp = c - i -= 1 - slot = i-start - t, h = times[slot] - fraction, sample_fractions, owned, saved_bar = extras(i, t, h) - cached = None if linearization is None else (lambda stage, seed: linearization[1]( - tree_util.tree_map(lambda buffer: buffer[slot, stage], cache), seed)) - dy, df, contribution = _step_pullback(rhs, states[slot], t, h, endpoints[slot], increments[slot], parameters, - fraction, sample_fractions, owned, by, bf, saved_bar, method, initial_fsal=(i == first), cached_pullback=cached) - return i, dy, df, _add(dp, pull(contribution)) - - _, by, bf, dp = lax.while_loop(lambda c: c[0] > start, reverse_step, (stop, by, bf, dp)) - return by, bf, dp - - -def solve_accepted(rhs, y0, args=None, **options): - """Return the original forward result with an accepted-step reverse rule. - - Closure conversion exposes RHS field/coil parameters as explicit VJP inputs. - A second conversion exposes forward-only controls (including event/norm - closures) separately: their cotangents are zero under this AD contract. - """ - if options.get("event") is not None: - # The converted driver uses prepared RHS parameters internally; event - # callbacks must still receive their original public args object. - options["event"] = tree_util.tree_map( - lambda function:lambda t,y,unused:function(t,y,args), options["event"]) - y0 = jnp.asarray(y0, dtype=jnp.result_type(y0, float)) - t0 = lax.stop_gradient(jnp.asarray(options.get("t0", 0.), dtype=y0.dtype)) - ts = lax.stop_gradient(jnp.asarray(options.get("ts", ()), dtype=y0.dtype)) - _validate_ad_sizes(adjoint_checkpoint_steps=options.get('adjoint_checkpoint_steps',16), - adjoint_max_accepted_steps=options.get('adjoint_max_accepted_steps')) - checkpoint_steps = int(options.pop("adjoint_checkpoint_steps", 16)) - parameter_accumulation = options.pop("adjoint_parameter_accumulation", "segment") - if parameter_accumulation not in ("step", "segment"): - raise ValueError("adjoint_parameter_accumulation must be 'step' or 'segment'") - # Bind arbitrary ESSOS args before conversion. Particles intentionally is - # an ordinary object, and need not become a new field/particle PyTree just - # to cross the custom VJP boundary. Its actually used traced attributes, - # and those of the field, are exposed by the converted numerical graph. - setup, stage = _partition_rhs(rhs, t0, y0, args) - raw_parameters = (setup.consts, stage.consts) - - def prepare(raw): - return jax.core.eval_jaxpr(setup.jaxpr, raw[0]), raw[1] - - def evaluate(t, y, parameters): - prepared, stage_constants = parameters - return jax.core.eval_jaxpr(stage.jaxpr, stage_constants, *prepared, t, y)[0] - - def forward(initial, parameters): - return solve_rk(evaluate, initial, parameters, **options, - _recorder=_OrbitRecorder(options.get("adjoint_max_accepted_steps") or options.get("max_steps", 1_000_000), checkpoint_steps)) - - converted_forward, driver_constants = _convert_all(forward, y0, prepare(raw_parameters)) - controls = (driver_constants, t0, ts) - method = options.get("method", "dopri5") - - @jax.custom_vjp - def integrate(initial, parameters, controls): - return converted_forward((initial, prepare(parameters)), controls[0])[0] - - def fwd(initial, parameters, controls): - activity = _ParameterActivity(tuple(leaf.perturbed for leaf in tree_util.tree_leaves(parameters))) - initial, raw, controls = custom_vjp_primal_tree_values((initial, parameters, controls)) - parameters = prepare(raw) - out, tape = converted_forward((initial, parameters), controls[0]) - return out, (initial, raw, controls, tape, activity) - - def bwd(residual, bar): - initial, raw, controls, tape, activity = residual - tape = eqx.error_if(tape, tape.count > tape.step_sizes.shape[0], - "accepted-step AD capacity exceeded; increase adjoint_max_accepted_steps") - raw_leaves, raw_structure = tree_util.tree_flatten(raw) - active = tuple(value for value, flag in zip(raw_leaves, activity.flags) if flag) - backward_prepare, backward_rhs = prepare_reverse_rhs(prepare, evaluate, controls[1], initial, raw) - def prepare_active(values): - iterator = iter(values) - full = tree_util.tree_unflatten(raw_structure, - [next(iterator) if flag else value for value, flag in zip(raw_leaves, activity.flags)]) - return backward_prepare(full) - parameters, setup_pullback = jax.vjp(prepare_active, active) - parameter_flags=_prepared_activity(prepare_active,active) - original_rhs=backward_rhs - backward_rhs=lambda t,y,p:original_rhs(t,y,_freeze_fixed(p,parameter_flags)) - _, t0, ts = controls - linearization = prepare_rhs_linearization(backward_rhs, t0, initial, parameters) - initial_cache = () if linearization is None else linearization[2] - sample_indices = jnp.arange(len(ts)) - terminal_bar = jnp.zeros_like(initial) if isinstance(bar.y, SymbolicZero) else bar.y - saved_bar = jnp.zeros((len(ts), initial.size), initial.dtype) if isinstance(bar.ys, SymbolicZero) else bar.ys - - def reverse_segment(carry): - end, by, bf, bp = carry - segment = jnp.maximum((end-1) // checkpoint_steps, 0) - start = segment*checkpoint_steps - def extras(i, t, h): - last = i == tape.count-1 - owned = (ts > t) & (ts <= jnp.where(last, tape.final_time, t+h)) - return jnp.where(last, tape.fraction, 1.), jnp.where(owned, (ts-t)/h, 0.), owned, saved_bar - pull = (lambda dp: setup_pullback(dp)[0]) if parameter_accumulation == "step" else (lambda dp: dp) - # Rejections and adaptation are absent from this RK reverse replay. - by, bf, bp = _reverse_segment(backward_rhs, parameters, linearization, method, - lambda i, t: (t, tape.step_sizes[i]), extras, start, end, tape.states[segment], - tape.derivatives[segment], tape.times[segment], by, bf, bp, pull, initial_cache, - checkpoint_steps, 0) - return start, by, bf, bp - - def reverse(carry): - if parameter_accumulation == "step": - return reverse_segment(carry) - # Preparation's pullback is linear in these cotangents. Sum them - # locally before applying it; the outer carry still stores only - # active raw-parameter bars, never a full-trajectory field tape. - i, by, bf, bp = carry - i, dy, df, dp = reverse_segment((i, by, bf, _zeros(parameters))) - return i, dy, df, _add(bp, setup_pullback(dp)[0]) - - _, by, bf, bp = lax.while_loop(lambda carry: carry[0] > 0, reverse, - (tape.count, terminal_bar, jnp.zeros_like(initial), _zeros(active))) - by = by+jnp.sum(jnp.where((sample_indices < tape.initial_saves)[:, None], saved_bar, 0.), axis=0) - iterator = iter(bp) - raw_bar = tree_util.tree_unflatten(raw_structure, - [next(iterator) if flag else _zeros(value) for value, flag in zip(raw_leaves, activity.flags)]) - derivatives = (by, raw_bar, _zeros(controls)) - return eqx.error_if(derivatives, tape.status > RKStatus.EVENT, - "accepted-step AD requires a successful/event solve; inspect status and max_steps") - - integrate.defvjp(fwd, bwd, symbolic_zeros=True) - return integrate(y0, raw_parameters, controls) - - -def replay_accepted(rhs, y0, args, tape, ts, *, method='dopri8'): - """Independent JVP/FD oracle: propagate new inputs on the frozen grid.""" - tape = tree_util.tree_map(lax.stop_gradient, tape) - constants, dense = _tableau(method) - ts = lax.stop_gradient(jnp.asarray(ts)) - y0 = jnp.asarray(y0) - t0 = tape.times[0] - saved = jnp.where((jnp.arange(len(ts)) < tape.initial_saves)[:,None],y0,jnp.inf) - def step(c): - i,t,y,f,saved=c - h=tape.step_sizes[i] - trial=_trial_step(rhs,t,y,h,args,f,*constants) - end=jnp.where(i==tape.count-1,tape.final_time,t+h) - fraction=jnp.where(i==tape.count-1,tape.fraction,1.) - owned=(ts>t)&(ts<=end) - fractions=jnp.where(owned,(ts-t)/h,0.) - values=jax.vmap(lambda s:dense(y,trial.y,trial.increments,s))(fractions) - return i+1,t+h,dense(y,trial.y,trial.increments,fraction),trial.f1,jnp.where(owned[:,None],values,saved) - _,_,y,_,saved=lax.while_loop(lambda c:c[0] SplitStatus.EVENT, - "accepted split AD requires a successful/event solve; inspect status and work limits"), - (tape.count > tape.times.shape[0], - "accepted split AD capacity exceeded; increase adjoint_max_accepted_steps"), - (tape.orbit_count > tape.orbit_times.shape[0], - "accepted orbit AD capacity exceeded; increase adjoint_max_orbit_steps")): - checked = eqx.error_if(checked, condition, message) - return tape._replace(count=checked) if compact else checked - - -def _partition(function, *examples): - """Expose closure parameters and hoist parameter-only kernel preparation.""" - graph, shape = jax.make_jaxpr(function, return_shape=True)(*examples) - structure = tree_util.tree_structure(shape) - setup, stage = _partition_graph(graph, [True] * len(examples)) - - def prepare(raw): - return jax.core.eval_jaxpr(setup.jaxpr, raw[0]), raw[1] - - def evaluate(arguments, parameters): - residuals, constants = parameters - out = jax.core.eval_jaxpr(stage.jaxpr, constants, *residuals, *arguments) - return tree_util.tree_unflatten(structure, out) - - return (setup.consts, stage.consts), prepare, evaluate - - -def _state_projection(prepare, evaluate, raw, *examples): - """Prune collision error-only inputs before transposing parameter setup. - - The forward driver still evaluates the complete production CollisionStep. - Reverse replay uses its state result only. Selecting live prepared inputs - here avoids differentiating error-only preparation, including expressions - whose derivatives can be singular even when the physical kick is regular. - """ - parameters = prepare(raw) - leaves = tree_util.tree_leaves(parameters) - graph = jax.make_jaxpr(lambda arguments,p:evaluate(arguments,p).y)(examples,parameters) - nargs = len(tree_util.tree_leaves(examples)) - state_graph, used = partial_eval.dce_jaxpr(graph.jaxpr,[True], - instantiate=[True]*nargs+[False]*len(leaves)) - live = used[nargs:] - - def prepare_state(values): - return tuple(value for value,flag in zip(tree_util.tree_leaves(prepare(values)),live) if flag) - - def state(arguments,values): - return jax.core.eval_jaxpr(state_graph,graph.consts,*tree_util.tree_leaves(arguments),*values)[0] - - return prepare_state,state - - -def _orbit_replay(rhs, initial, parameters, tape, index, method): - """Forward-only oracle; propagate new physical inputs on the frozen grid.""" - constants, dense = _tableau(method) - start, end = tape.offsets[index], tape.offsets[index+1] - f = rhs(tape.orbit_times[start, 0], initial, parameters) - - def step(local): - i, y, f = local - t, h = tape.orbit_times[i] - trial = _trial_step(rhs, t, y, h, parameters, f, *constants) - fraction = jnp.where(i == end-1, tape.fractions[index], 1.) - terminal = dense(y, trial.y, trial.increments, fraction) - return i+1, terminal, trial.f1 - - _, final, _ = lax.while_loop( - lambda local: local[0] < end, step, (start, initial, f)) - return final - - -def replay_accepted_split(rhs, collision, initial, args, tape, ts, *, method="dopri8"): - """Forward-only frozen-schedule/noise oracle for finite-difference checks. - - New parameters propagate through every orbit and collision map. Recorded - checkpoint states are deliberately ignored. No adaptation or noise drawing - occurs. Returns (saved_states, terminal_state); ordinary JVPs are supported. - """ - tape = _checked_tape(tree_util.tree_map(lax.stop_gradient, tape)) - ts = lax.stop_gradient(jnp.asarray(ts)) - rhs_bound = lambda t, y, unused: rhs(t, y, args) - - def advance(local): - index, y, saved = local - y = _orbit_replay(rhs_bound, y, None, tape, index, method) - t, end = tape.times[index] - y = lax.cond(tape.kicks[index], - lambda y: collision(end, y, end-t, tape.increments[index], args).y, - lambda y: y, y) - owned = (ts > t) & (ts <= end) - return index+1, y, jnp.where(owned[:, None], y, saved) - - _, final, saved = lax.while_loop(lambda local: local[0] < tape.count, advance, - (jnp.array(0), initial, jnp.broadcast_to(initial, (len(ts), initial.size)))) - saved = jnp.where((jnp.arange(len(ts)) >= tape.saved_count)[:, None], final, saved) - return saved, final - - -def solve_accepted_split(rhs, collision, y0, args=None, **options): - """Attach a discrete reverse rule to the unchanged production split solve.""" - forward_override = options.pop('_forward_override',None) - # Kernels receive prepared numerical parameters internally. Control - # callbacks retain the user's original args ABI, while their numerical - # decisions are frozen by the outer custom VJP. - if options.get("event") is not None: - options["event"] = tree_util.tree_map( - lambda function: lambda t,y,unused,**kw: function(t,y,args,**kw), options["event"]) - if options.get("step_limit") is not None: - step_limit = options["step_limit"] - options["step_limit"] = lambda t,y,unused: step_limit(t,y,args) - y0 = jnp.asarray(y0, dtype=jnp.result_type(y0, float)) - t0 = lax.stop_gradient(jnp.asarray(options.get("t0", 0.), y0.dtype)) - ts = lax.stop_gradient(jnp.asarray(options["ts"], y0.dtype)) - h0 = jnp.asarray(options["dt0"], y0.dtype) - noise = jnp.zeros((options["noise_dim"],), y0.dtype) - raw_rhs, prepare_rhs, evaluate_rhs = _partition(lambda t, y: rhs(t, y, args), t0, y0) - raw_collision, prepare_collision, evaluate_collision = _partition( - lambda t, y, h, dw: collision(t, y, h, dw, args), t0, y0, h0, noise) - prepare_collision_state, collision_state = _state_projection( - prepare_collision,evaluate_collision,raw_collision,t0,y0,h0,noise) - - def prepare(raw): - return prepare_rhs(raw[0]), prepare_collision(raw[1]) - - def orbit_rhs(t, y, parameters): - return evaluate_rhs((t, y), parameters) - - # Components omitted from a stored endpoint must be copied unchanged by - # the collision map. This metadata selects compact full-orbit storage; - # generic maps and GC retain their previous layout. - collision_indices = options.get("_collision_state_indices") - forward_options = dict(options) - if collision_indices is not None: - # _partition already hoisted all parameter-only RHS preparation. - forward_options["_orbit_rhs_prepared"] = True - - def forward(initial, parameters): - if forward_override is not None: - # The independent recorder supplies the same accepted tape ABI. - # Its numerical inputs are closure-converted into frozen controls; - # physical bars still flow through raw_rhs/raw_collision below. - return forward_override(initial,parameters,record=True) - return solve_split(lambda t, y, p: orbit_rhs(t, y, p[0]), - lambda t, y, h, dw, p: evaluate_collision((t, y, h, dw), p[1]), - initial, parameters, **forward_options, _recorder=_SplitRecorder(initial, forward_options)) - - raw = (raw_rhs, raw_collision) - converted_forward, driver_constants = _convert_all(forward, y0, prepare(raw)) - controls = (driver_constants, ts) - outer_segment = int(options.get("adjoint_checkpoint_steps", 16)) - inner_segment = int(options.get("adjoint_orbit_checkpoint_steps", 16)) - workspace = min(options.get("max_orbit_steps", 10_000), - options.get("adjoint_max_orbit_steps") or options.get("max_steps", 100_000)) - method = options.get("method", "dopri8") - - @jax.custom_vjp - def integrate(initial, raw, controls): - return converted_forward((initial, prepare(raw)), controls[0])[0] - - def fwd(initial, raw, controls): - activity = _ParameterActivity(tuple(leaf.perturbed for leaf in tree_util.tree_leaves(raw))) - initial, raw, controls = custom_vjp_primal_tree_values((initial, raw, controls)) - out, tape = converted_forward((initial, prepare(raw)), controls[0]) - saved_controls = (None,controls[1]) if forward_override is not None else controls - return out, (initial, raw, saved_controls, tape, activity, - out.y if collision_indices is not None else None) - - def bwd(residual, bar): - initial, raw, controls, tape, activity, final = residual - tape = _checked_tape(tape, compact=collision_indices is not None) - raw_leaves, raw_structure = tree_util.tree_flatten(raw) - active = tuple(value for value, flag in zip(raw_leaves, activity.flags) if flag) - - def active_raw(values): - iterator = iter(values) - return tree_util.tree_unflatten(raw_structure, - [next(iterator) if flag else value for value, flag in zip(raw_leaves, activity.flags)]) - - backward_prepare, backward_rhs = prepare_reverse_rhs( - prepare_rhs, orbit_rhs, t0, initial, raw[0]) - rp, rhs_setup_pullback = jax.vjp( - lambda values: backward_prepare(active_raw(values)[0]), active) - rhs_flags=_prepared_activity(lambda values:backward_prepare(active_raw(values)[0]),active) - original_rhs=backward_rhs - backward_rhs=lambda t,y,p:original_rhs(t,y,_freeze_fixed(p,rhs_flags)) - linearization = prepare_rhs_linearization(backward_rhs, t0, initial, rp) - csp, collision_setup_pullback = jax.vjp( - lambda values:prepare_collision_state(active_raw(values)[1]),active) - collision_flags=_prepared_activity(lambda values:prepare_collision_state(active_raw(values)[1]),active) - backward_collision_state=lambda arguments,p:collision_state(arguments,_freeze_fixed(p,collision_flags)) - ts = controls[1] - saved_bar = jnp.zeros((len(ts), initial.size), initial.dtype) if isinstance(bar.ys, SymbolicZero) else bar.ys - terminal_bar = jnp.zeros_like(initial) if isinstance(bar.y, SymbolicZero) else bar.y - terminal_bar += jnp.sum(jnp.where((jnp.arange(len(ts)) >= tape.saved_count)[:, None], saved_bar, 0.), axis=0) - - def reverse_map(carry, initial): - index, by, bp, post = carry - index -= 1 - offset = tape.checkpoint_offsets[index] - # The next map's initial state is this map's post-collision state. - # Restore only changed components to obtain the pre-collision state. - orbit_end = (tape.orbit_endpoints[index] if collision_indices is None else - post.at[jnp.asarray(collision_indices)].set(tape.orbit_endpoints[index])) - count = (workspace+inner_segment-1)//inner_segment - rows = offset+jnp.arange(count) - # Padded rows are never read by the bounded inner reverse loop. - if collision_indices is None: - checkpoints = (tape.orbit_states[rows],tape.orbit_derivatives[rows]) - else: - # One first derivative per preceding interval is omitted. - derivative_rows = offset-index+jnp.arange(max(count-1,0)) - checkpoints = (tape.orbit_states[rows],tape.orbit_derivatives[derivative_rows]) - t, end = tape.times[index] - by += jnp.sum(jnp.where(((ts > t) & (ts <= end))[:, None], saved_bar, 0.), axis=0) - - def kick_reverse(values): - by, bp = values - _, pullback = jax.vjp(lambda y, values: backward_collision_state( - (end, y, end-t, tape.increments[index]),values),orbit_end,csp) - dy, dp = pullback(by) - return dy, (bp[0],_add(bp[1],dp)) - - by, bp = lax.cond(tape.kicks[index], kick_reverse, lambda values: values, (by, bp)) - orbit_start, orbit_stop = tape.offsets[index], tape.offsets[index+1] - states, derivatives = checkpoints - # Each collision starts a new orbit: its initial FSAL cache must - # use this map's state, not the trajectory's original birth state. - if collision_indices is None: - map_initial_cache = (linearization[0](tape.orbit_times[orbit_start, 0], initial)[1] - if linearization is not None else ()) - elif linearization is not None: - map_initial_value, map_initial_cache = linearization[0](tape.orbit_times[orbit_start, 0], initial) - else: - map_initial_value = backward_rhs(tape.orbit_times[orbit_start, 0], initial, rp) - map_initial_cache = () - - def reverse_orbit_segment(local): - stop, by, bf, bp = local - segment = jnp.maximum((stop-orbit_start-1) // inner_segment, 0) - start = orbit_start + segment*inner_segment - if collision_indices is None: - f = derivatives[segment] - else: - f = (jnp.where(segment == 0, map_initial_value, derivatives[jnp.maximum(segment-1, 0)]) - if derivatives.shape[0] else map_initial_value) - extras = lambda i, t, h: (jnp.where(i == orbit_stop-1, tape.fractions[index], 1.), - jnp.empty((0,), initial.dtype), jnp.empty((0,), bool), jnp.empty((0, initial.size), initial.dtype)) - # Transpose parameter preparation once, after summing the segment's prepared bars. - by, bf, dp = _reverse_segment(backward_rhs, rp, linearization, method, - lambda i, t: tuple(tape.orbit_times[i]), extras, start, stop, states[segment], f, - jnp.zeros((), initial.dtype), by, bf, _zeros(rp), lambda dp: dp, map_initial_cache, - inner_segment, orbit_start) - return start, by, bf, (_add(bp[0], rhs_setup_pullback(dp)[0]), bp[1]) - - _, by, _, bp = lax.while_loop(lambda values: values[0] > orbit_start, - reverse_orbit_segment, (orbit_stop, by, jnp.zeros_like(initial), bp)) - return index, by, bp, initial if collision_indices is not None else None - - def reverse(carry): - stop = carry[0] - segment = jnp.maximum((stop-1) // outer_segment, 0) - start = segment*outer_segment - def reverse_recorded(values): - index = values[0]-1 - initial = (jnp.where(tape.offsets[index+1]>tape.offsets[index], - tape.orbit_states[tape.checkpoint_offsets[index]],tape.orbit_endpoints[index]) - if collision_indices is None else tape.orbit_states[tape.checkpoint_offsets[index]]) - return reverse_map(values,initial) - return lax.while_loop(lambda values: values[0]>start,reverse_recorded,carry) - - _, by, bp, _ = lax.while_loop(lambda values: values[0] > 0, reverse, - (tape.count, terminal_bar, (_zeros(active),_zeros(csp)), final)) - has_kick = jnp.any((jnp.arange(tape.kicks.size) < tape.count) & tape.kicks) - collision_bar = lax.cond(has_kick,lambda _:collision_setup_pullback(bp[1])[0], - lambda _:_zeros(active),None) - bp = _add(bp[0],collision_bar) - by += jnp.sum(jnp.where((jnp.arange(len(ts)) < tape.initial_saves)[:, None], saved_bar, 0.), axis=0) - iterator = iter(bp) - raw_bar = tree_util.tree_unflatten(raw_structure, - [next(iterator) if flag else _zeros(value) for value, flag in zip(raw_leaves, activity.flags)]) - # Independent GPU preparation is frozen controller/forward data. - # A symbolic zero prunes its entire setup transpose. Materialized - # zero arrays can otherwise retain a second coil-geometry pullback. - return by, raw_bar, None if forward_override is not None else _zeros(controls) - - integrate.defvjp(fwd, bwd, symbolic_zeros=True) - return integrate(y0, raw, controls) - - -class _OrbitRecorder: - """Sparse states/FSAL plus accepted time sizes; never record a rejection.""" - def __init__(self, capacity, segment, schedule=False): - self.capacity, self.segment, self.schedule = capacity, segment, schedule - _validate_ad_sizes(capacity=capacity, segment=segment) - - def initialize(self, state): - y = state['y']; nodes = (self.capacity+self.segment-1)//self.segment - state.update(ad_states=jnp.zeros((nodes,y.size),y.dtype), - ad_derivatives=jnp.zeros((nodes,y.size),y.dtype), - ad_times=jnp.zeros((nodes,),y.dtype), ad_steps=jnp.zeros((self.capacity,),y.dtype), - ad_fraction=jnp.ones((),y.dtype)) - if self.schedule: - state['ad_schedule']=jnp.zeros((self.capacity,2),y.dtype) - return state - - def record(self, previous, updated, t, y, h, fraction, accept): - i=previous['accepted'] - def store(state): - def checkpoint(state): - slot=i//self.segment - return dict(state,ad_states=state['ad_states'].at[slot].set(y), - ad_derivatives=state['ad_derivatives'].at[slot].set(previous['f']), - ad_times=state['ad_times'].at[slot].set(t)) - state=lax.cond(i%self.segment==0,checkpoint,lambda s:s,state) - state=dict(state,ad_steps=state['ad_steps'].at[i].set(h),ad_fraction=fraction) - if self.schedule: - state['ad_schedule']=state['ad_schedule'].at[i].set(jnp.stack([t,h])) - return state - return lax.cond(accept&(i=saved_count[...], saved(i), zero8) - gp = tuple(jnp.zeros_like(value) for value in geom) - cp = jnp.zeros((1 << (ncoeff-1).bit_length(),), jnp.float64) - fp = zero - row_indices = jnp.arange(16)[:, None] - def reverse_map(carry): - index, by, gp, cp, fp, post = carry - index -= 1 - t, end = tref[index, 0], tref[index, 1] - for i in range(nsave): - by += jnp.where((tsref[i]>t)&(tsref[i]<=end), saved(i), zero8) - offset = nodes[index] - map_initial = read_state(states, offset) - endpoint = (read_state(endpoints, index) if gc else - pack(tuple(scalar(post, i) for i in range(3))+ - tuple(endpoints[index, i] for i in range(3))+(zero, zero))) - def kick_reverse(values): - by, gp, cp, fp = values - dw = tuple(dwref[index, j] for j in range(noise_dim)) - result = evaluate(kick_pb.jaxpr, kick_constants, *at(endpoint), *geom, - *cvalues, floor_value, end-t, *dw, *at(by), scalarize=True) - by = pack(result[:8]) - gp = tuple(a+b for a, b in zip(gp, result[8:8+nfield])) - first_constant=8+nfield - cp += pack(tuple(jnp.asarray(v).reshape(()) for v in result[first_constant:first_constant+ncoeff]) - +(zero,)*(cp.shape[0]-ncoeff)) - fp += result[first_constant+ncoeff].reshape(()) - return by, gp, cp, fp - by, gp, cp, fp = jax.lax.cond(kicks[index], kick_reverse, lambda values: values, (by, gp, cp, fp)) - orbit_start, orbit_stop = offsets[index], offsets[index+1] - def reverse_segment(local): - stop, by, bf, gp = local - segment = (stop-orbit_start-1)//segment_steps - start = orbit_start+segment*segment_steps - y = read_state(states, offset+segment) - f, cache = rhs(y) - def reconstruct(values): - i, y, f, cache = values - h = otime[i, 1] - bank = jnp.where(row_indices==0, h*f[None, :], jnp.zeros((16, 8), jnp.float64)) - cache_bank = jnp.where(row_indices==0, cache[None,:], jnp.zeros((16,cache_width),jnp.float64)) - def stage(j, values): - bank, f, cache_bank, cache = values - point = y+jnp.sum(jnp.where(row_indicesstart, reverse_step, (stop, by, bf, gp)) - _, by, _, gp = jax.lax.while_loop(lambda values: values[0]>orbit_start, - reverse_segment, (orbit_stop, by, zero8, gp)) - return index, by, gp, cp, fp, map_initial - post = yref[...] - _, by, gp, cp, fp, _ = jax.lax.while_loop(lambda values: values[0]>0, reverse_map, - (countref[...], by, gp, cp, fp, post)) - for i in range(nsave): - by += jnp.where(i 0), value, jnp.nan) - - original = jax.make_jaxpr(physical)(y0) - flat = jax.make_jaxpr(lambda y: _evaluate_inline(original.jaxpr, original.consts, y)[0])(y0) - - def evaluate(y): - graph = flat.jaxpr - env = dict(zip(graph.constvars, flat.consts)) - env.update(zip(graph.invars, (y,))) - def read(v): - return v.val if isinstance(v, Literal) else env[v] - for equation in graph.eqns: - values = [read(v) for v in equation.invars] - name = equation.primitive.name - if name == "slice": - value = values[0] - for axis, (start, end, size) in enumerate(zip(equation.params["start_indices"], - equation.params["limit_indices"], value.shape)): - if start != 0 or end != size: - if end-start != 1: - raise ValueError("Unsupported kernel-local RHS slice") - value = _select_one(value, start, axis) - result = [value] - elif name == "dynamic_slice": - value = values[0] - for axis, (index, size, wanted) in enumerate(zip(values[1:], value.shape, - equation.params["slice_sizes"])): - if size != wanted: - if wanted != 1: - raise ValueError("Unsupported kernel-local dynamic RHS slice") - value = _select_one(value, jnp.clip(index, 0, size-1), axis) - result = [value] - elif name == "concatenate" and len(values) > 2: - value = jnp.zeros((len(values),), values[0].dtype) - for i, v in enumerate(values): - value = jnp.where(jnp.arange(len(values)) == i, jnp.reshape(v, ()), value) - result = [value] - elif name == "le_to": - # Positive finite knots followed by +inf; NaN queries sort last. - result = [jnp.less_equal(values[0], values[1]) | jnp.isnan(values[1])] - elif name == "is_finite": - result = [_kernel_is_finite(values[0])] - else: - result = _bind_primitive(equation.primitive, values, equation.params) - if not equation.primitive.multiple_results: - result = [result] - env.update(zip(equation.outvars, result)) - return read(graph.outvars[0]) - return evaluate - - - -def _bind_primitive(primitive, values, parameters): - """Bind an equation using either supported JAX get_bind_params ABI. - - Older JAX returns (subfunctions, keyword_parameters); newer JAX returns - only keyword_parameters. Keep legacy subfunctions for opaque AD/call - primitives rather than discarding them while adapting ordinary equations. - This Python dispatch runs during graph preparation, not each RK stage. - """ - binding = primitive.get_bind_params(parameters) - if isinstance(binding, tuple): - subfunctions, binding = binding - return primitive.bind(*subfunctions, *values, **binding) - return primitive.bind(*values, **binding) - - -def _evaluate_inline(graph, constants, *arguments): - """Expose jit boundaries without entering custom AD/control-flow calls.""" - from jax.extend.core import Literal - environment = dict(zip(graph.constvars, constants)) - environment.update(zip(graph.invars, arguments)) - - def read(var): - return var.val if isinstance(var, Literal) else environment[var] - - for equation in graph.eqns: - values = [read(var) for var in equation.invars] - if equation.primitive.name in ("jit", "pjit"): - nested = equation.params["jaxpr"] - result = _evaluate_inline(nested.jaxpr, nested.consts, *values) - else: - result = _bind_primitive(equation.primitive, values, equation.params) - if not equation.primitive.multiple_results: - result = [result] - environment.update(zip(equation.outvars, result)) - return [read(var) for var in graph.outvars] - # Private Biot-Savart GPU view def _cross(a,b): @@ -1096,53 +941,116 @@ def estimate(endpoint,error,speed): # Independent GPU particle loops -def _prepare_gpu_solver(arrays, factory, speed, times, dt0, rtol, atol, max_steps, *, - state_size=4, method="dopri8", adaptive=True, num_warps=1, interpret=False): - """Prepare a CUDA solver outside jit; return solve(y0, invariant). +# Persistent-kernel setup - Fixed and adaptive Dopri5/8 share FSAL stages and standard dense output. - Adaptive acceptance uses the adapter's error blocks and I control. Save - times do not constrain steps. Each particle stops independently; there is - no particle chunking or compaction. Numerical equivalence is subject to - ordinary floating-point changes from a different compiler execution path. - """ - from jax.experimental import pallas as pl - from jax.experimental.pallas import triton as plt - if method not in ("dopri5", "dopri8"): - raise ValueError("The fused orbit method must be dopri5 or dopri8") +def require_cuda(interpret): + """Reject non-CUDA backends unless the Pallas interpreter is requested.""" if not interpret and (jax.default_backend() != "gpu" or "cuda" not in jax.devices()[0].client.platform_version.lower()): raise ValueError("The experimental fused orbit solver requires CUDA") + + +def check_save_times(times): + """Return save times as float64; they must be ordered, start at zero and end positive.""" times = np.asarray(times, dtype=np.float64) if (times.ndim != 1 or len(times) < 2 or not np.isfinite(times).all() or times[0] != 0 or times[-1] <= 0 or np.any(np.diff(times) < 0)): raise ValueError("Save times must be ordered, start at zero and end at a positive time") - if not (np.isfinite(dt0) and dt0 > 0 and np.isfinite(speed) and speed > 0 and - np.isfinite(rtol) and rtol >= 0 and np.isfinite(atol) and atol > 0): - raise ValueError("Require positive finite dt0, speed, atol and finite rtol >= 0") - if (isinstance(max_steps, (bool, np.bool_)) or not isinstance(max_steps, (int, np.integer)) or - not 0 < max_steps <= np.iinfo(np.int32).max): - raise ValueError("max_steps must be a positive int32 work bound") + return times + + +def check_work_bound(name, value, limit=np.iinfo(np.int32).max): + """Require a positive int (not bool) no larger than ``limit``.""" + if isinstance(value, (bool, np.bool_)) or not isinstance(value, (int, np.integer)) or not 0 < value <= limit: + raise ValueError(f"{name} must be a positive int32 work bound") + + +class KernelTableau(NamedTuple): + """Padded Runge-Kutta banks: stage matrix, error and dense-output weights.""" + matrix: jnp.ndarray + error: jnp.ndarray + dense: jnp.ndarray + stages: int + bank_size: int + order: int + + +def kernel_tableau(method): + """Pad Dopri5/Dopri8 coefficients to a power-of-two stage bank.""" + if method not in ("dopri5", "dopri8"): + raise ValueError("The fused orbit method must be dopri5 or dopri8") rows, weights, order = (_A, _ERROR, 5) if method == "dopri5" else (_A8, _ERROR8, 8) stages = len(weights) bank_size = 1 << (stages-1).bit_length() matrix = np.zeros((bank_size, bank_size)) for i, row in enumerate(rows, 1): matrix[i, :len(row)] = row - coefficients = jnp.asarray(matrix) - error_weights = jnp.asarray(tuple(weights)+(0.,)*(bank_size-stages)) - if method == "dopri5": - dense_matrix = np.zeros((1, bank_size)) - dense_matrix[0, :stages] = np.asarray(_MID) - else: - dense_matrix = np.zeros((len(_DENSE8[0]), bank_size)) - dense_matrix[:, :stages] = np.asarray(_DENSE8).T - dense_coefficients = jnp.asarray(dense_matrix) + dense = np.zeros((1 if method == "dopri5" else len(_DENSE8[0]), bank_size)) + dense[:, :stages] = np.asarray(_MID)[None] if method == "dopri5" else np.asarray(_DENSE8).T + return KernelTableau(jnp.asarray(matrix), jnp.asarray(tuple(weights)+(0.,)*(bank_size-stages)), + jnp.asarray(dense), stages, bank_size, order) + + +def save_loop(condition, write, index): + """Run ``write`` while ``condition(index)``; refs are read in the body, not the condition. + + JAX 0.6 cannot discharge Pallas ref reads in while conditions, so the + next condition is carried through the loop. + """ + def advance(carry): + index = write(carry[0]) + return index, condition(index) + return jax.lax.while_loop(lambda c: c[1], advance, (index, condition(index)))[0] + + +@functools.lru_cache(maxsize=None) +def _kernel_lowers(): + """True if this JAX can lower the Pallas idioms the fused kernels use (checked once).""" + def kernel(x, out): + def write(i): + out[i] = x[i] + return i + 1 + save_loop(lambda i: i < 2, write, 0) + out[3] = _kernel_is_finite(x[2]).astype(x.dtype) + try: + jax.jit(lambda x: launch_kernel(kernel, jax.ShapeDtypeStruct((4,), x.dtype), (x,), warps=1, + interpret=False)).lower(jnp.ones(4)) + except Exception: + return False + return True + + +def launch_kernel(kernel, shapes, arguments, *, warps, interpret): + """Launch a gridless Pallas kernel with ``warps`` warps per program.""" + from jax.experimental import pallas as pl + from jax.experimental.pallas import triton as plt + return pl.pallas_call(kernel, out_shape=shapes, grid=(), interpret=interpret, + compiler_params=plt.CompilerParams(num_warps=warps))(*arguments) + + +def _prepare_gpu_solver(arrays, factory, speed, times, dt0, rtol, atol, max_steps, *, + state_size=4, method="dopri8", adaptive=True, num_warps=1, interpret=False): + """Prepare a CUDA solver outside jit; return solve(y0, invariant). + + Fixed and adaptive Dopri5/8 share FSAL stages and standard dense output. + Adaptive acceptance uses the adapter's error blocks and I control. Save + times do not constrain steps. Each particle stops independently; there is + no particle chunking or compaction. Numerical equivalence is subject to + ordinary floating-point changes from a different compiler execution path. + """ + require_cuda(interpret) + times = check_save_times(times) + if not (np.isfinite(dt0) and dt0 > 0 and np.isfinite(speed) and speed > 0 and + np.isfinite(rtol) and rtol >= 0 and np.isfinite(atol) and atol > 0): + raise ValueError("Require positive finite dt0, speed, atol and finite rtol >= 0") + check_work_bound("max_steps", max_steps) + coefficients, error_weights, dense_coefficients, stages, bank_size, order = kernel_tableau(method) tmax = float(times[-1]) # Load float64 scalars explicitly. Nested Pallas loops must not infer # their device scalar precision from weak Python constant initializers. scalars = jnp.asarray([0., dt0, tmax, 8*np.finfo(np.float64).eps*tmax, speed, rtol+atol], jnp.float64) inputs = tuple(arrays)+(coefficients, error_weights, dense_coefficients, jnp.asarray(times), scalars) + dense_rows = dense_coefficients.shape[0] nsave = len(times) def solve(y0, moment): @@ -1164,14 +1072,6 @@ def kernel(yref, mref, initial_status_ref, *refs): def finite(value): return jnp.min(_kernel_is_finite(value).astype(jnp.int32)) > 0 - def save_loop(condition, write, index): - # JAX 0.6 cannot discharge Pallas ref reads in while conditions. - def advance(carry): - index = write(carry[0]) - return index, condition(index) - return jax.lax.while_loop(lambda c: c[1], advance, - (index, condition(index)))[0] - def dense(y, endpoint, bank, fraction): if method == "dopri5": # Same quartic extension as rk._rk45_dense; the padded @@ -1186,7 +1086,7 @@ def dense(y, endpoint, bank, fraction): value = y+fraction*(first+fraction*(c+fraction*(b+fraction*a))) return jnp.where(fraction == 0, y, jnp.where(fraction == 1, endpoint, value)) weights = jnp.zeros((bank_size,), point.dtype) - for power in range(dense_matrix.shape[0]): + for power in range(dense_rows): weights = weights*fraction+dref[power, :] value = y+jnp.sum((weights*fraction)[:, None]*bank, axis=0) return jnp.where(fraction == 0, y, jnp.where(fraction == 1, endpoint, value)) @@ -1297,36 +1197,14 @@ def save(i): jax.ShapeDtypeStruct((state_size,), y0.dtype), jax.ShapeDtypeStruct((), y0.dtype), jax.ShapeDtypeStruct((), y0.dtype), jax.ShapeDtypeStruct((3,), jnp.int32), jax.ShapeDtypeStruct((), jnp.int32)) - ys, saved, y, t, h, counts, status = pl.pallas_call(kernel, out_shape=shapes, grid=(), - interpret=interpret, compiler_params=plt.CompilerParams(num_warps=num_warps))(y0, moment, initial_status, *inputs) + ys, saved, y, t, h, counts, status = launch_kernel(kernel, shapes, (y0, moment, initial_status, *inputs), + warps=num_warps, interpret=interpret) return RKSolution(saved, ys, t, y, status, status == 1, RKStats(counts[0], counts[1], counts[2]), jnp.zeros((0,), y0.dtype), h) solve.method = method return solve -# Boozer GPU preparation - -def _prepare_boozer_gpu(field, mass, charge, speed, times, dt0, rtol, atol, max_steps, *, - method="dopri8", adaptive=True, interpret=False): - if field.table is not None or field.sine_coef is not None: - raise ValueError("The CUDA adapter requires a fixed cosine Fourier Boozer field") - radial, b = _padded_spline(field.r_knots, field.b_coef) - flux, profiles = _padded_spline(field.s_knots, field.profile_coef) - modes = b.shape[1] - arrays = (radial, b, flux, profiles, jnp.pad(field.xm, (0,modes-len(field.xm))), - jnp.pad(field.xn, (0,modes-len(field.xn)))) - scalar = lambda y,i: _select_one(y,i,0).reshape(()) - def factory(refs, point, moment): - model = _KernelField(refs, float(field.psi0)) - evaluate = _kernel_rhs(model, point, moment, mass, charge) - def estimate(endpoint, error, speed): - a,b,c,d = (scalar(error,i) for i in range(4)) - return jnp.maximum(jnp.sqrt(a*a+b*b),jnp.maximum(jnp.abs(c),jnp.abs(d)/speed)) - return evaluate, estimate, lambda y: 1.-scalar(y,0)**2-scalar(y,1)**2 - return _prepare_gpu_solver(arrays,factory,speed,times,dt0,rtol,atol,max_steps, - method=method,adaptive=adaptive,interpret=interpret) - # Existing Tracing integration class _ParticleParameters(NamedTuple): @@ -1360,6 +1238,11 @@ def prepare_solver(field, *, mass, charge, energy, times, dt0, model="gc", metho if vmec and model!='gc': raise ValueError("VMEC native integration supports guiding centers") size=5 if vmec else 4 if model=='gc' else 6 + if execution=='cuda' and not interpret: + require_cuda(interpret) + if not _kernel_lowers(): + warnings.warn("This JAX version cannot lower the fused Pallas kernel; using the portable path.") + execution='jax' if execution=='cuda': if controller!='i': raise ValueError("CUDA execution supports the I controller") @@ -1442,10 +1325,7 @@ def _solve_tracing(trace, terms, controller, t0, y0, args, event, throw, exterio root_find=event.root_finder is not None, max_steps=trace.max_steps, method="dopri8" if trace.integrator == "essos_dopri8" else "dopri5", error_norm=_trace_error_norm(trace, exterior=exterior, flag=flag), - controller=trace.native_controller, pcoeff=trace.native_pcoeff, icoeff=trace.native_icoeff, - adjoint=trace.native_adjoint, adjoint_checkpoint_steps=trace.native_adjoint_checkpoint_steps, - adjoint_max_accepted_steps=trace.native_adjoint_max_accepted_steps, - adjoint_parameter_accumulation=trace.native_adjoint_parameter_accumulation) + controller=trace.native_controller, pcoeff=trace.native_pcoeff, icoeff=trace.native_icoeff) ys = solution.ys if throw: import equinox as eqx @@ -1466,28 +1346,20 @@ def _boozer_error_norm(error, speed, rtol, atol): def trace_boozer_orbits(field, y0, mu0, mass, charge, speed, times, dt0, rtol, atol, method, adaptive, controller, pcoeff, icoeff, max_steps, *, execution="jax"): - """Select portable JAX or fixed-field CUDA execution without changing physics. + """Trace Boozer orbits on the portable JAX path. - CUDA supports fixed steps and adaptive I control for cosine Fourier fields. The portable - path also supports fixed steps, PI control, sine modes and angle tables. - The choice is explicit: no backend, field layout or controller is changed. + A fused CUDA kernel for Boozer fields was slower than this path, so + ``execution="cuda"`` is rejected instead of silently changed. """ - if execution not in ("jax", "cuda"): - raise ValueError("native_execution must be 'jax' or 'cuda'") - solver = None - if execution == "cuda": - if controller != "i": - raise ValueError("CUDA native orbits require I control") - prepare_orbit_solver = _prepare_boozer_gpu - solver = prepare_orbit_solver(field, mass, charge, float(speed), times, - dt0, rtol, atol, max_steps, method=method, adaptive=adaptive) + if execution != "jax": + raise ValueError("Boozer native_execution must be 'jax'; the CUDA kernel was slower") return _boozer_batch(field, y0, mu0, mass, charge, speed, times, dt0, rtol, atol, - method, adaptive, controller, pcoeff, icoeff, max_steps, solver) + method, adaptive, controller, pcoeff, icoeff, max_steps) -@partial(jax.jit, static_argnames=("method", "adaptive", "controller", "max_steps", "orbit_solver")) +@partial(jax.jit, static_argnames=("method", "adaptive", "controller", "max_steps")) def _boozer_batch(field, y0, mu0, mass, charge, speed, times, dt0, rtol, atol, - method, adaptive, controller, pcoeff, icoeff, max_steps, orbit_solver=None): + method, adaptive, controller, pcoeff, icoeff, max_steps): """One shared native RK solve per orbit, independent of the save grid.""" from .boozer import _chart, guiding_center_rhs @@ -1510,13 +1382,10 @@ def error_norm(y0, y1, error): # a parallel turning point; these are chart errors, not metres. return _boozer_error_norm(error, speed, rtol, atol) - if orbit_solver is None: - solution = solve_rk(rhs, y0, t1=times[-1], dt0=dt0, ts=times, - method=method, adaptive=adaptive, rtol=rtol, atol=atol, - error_norm=error_norm, controller=controller, pcoeff=pcoeff, icoeff=icoeff, - max_steps=max_steps, event=lambda t, y, args: 1. - y[0]**2 - y[1]**2) - else: - solution = orbit_solver(y0, mu) + solution = solve_rk(rhs, y0, t1=times[-1], dt0=dt0, ts=times, + method=method, adaptive=adaptive, rtol=rtol, atol=atol, + error_norm=error_norm, controller=controller, pcoeff=pcoeff, icoeff=icoeff, + max_steps=max_steps, event=lambda t, y, args: 1. - y[0]**2 - y[1]**2) def output(y): s, _, theta, *_ = _chart(y) diff --git a/essos/split.py b/essos/split.py index 4aa58ac2..72e9667c 100644 --- a/essos/split.py +++ b/essos/split.py @@ -2,19 +2,15 @@ An independently controlled native RK orbit solve precedes each Milstein collision map. Brownian rejection memory preserves every sampled interval. -Private GPU adapters leave public field classes unchanged. Optional accepted-map -recording connects to the checkpointed reverse rules in ``essos.native_ad``. """ from enum import IntEnum -from functools import lru_cache from typing import NamedTuple import jax import jax.numpy as jnp import numpy as np from jax import lax, tree_util from .background_species import nu_D_ab, nu_par_ab, d_nu_par_ab, nu_s_ab -from .rk import RKStatus, solve_rk, _concrete_check, _prepare_rhs, OrbitErrorNorm - +from .rk import RKStatus, solve_rk, _concrete_check, _prepare_rhs, OrbitErrorNorm, _prepare_vmec_data def _ratio(numerator, denominator): @@ -246,10 +242,7 @@ class SplitSolution(NamedTuple): stats: SplitStats step_ts: jax.Array -def solve_split(rhs, collision, y0, args=None, *, t0=0.0, t1, dt0, ts, key, noise_dim, adaptive=True, dtmin=0.0, dtmax=jnp.inf, method='dopri8', orbit_dt0=None, orbit_adaptive=True, orbit_stage_execution='loop', reuse_orbit_step=True, rtol=1e-09, atol=1e-09, error_norm=None, controller='i', pcoeff=0.4, icoeff=0.3, event=None, max_steps=100000, max_orbit_steps=10000, max_pending=64, record_steps=False, step_limit=None, align_orbit=False, event_dtmax=None, _orbit_rhs_prepared=False, - adjoint='none', adjoint_checkpoint_steps=16, adjoint_orbit_checkpoint_steps=16, - adjoint_max_accepted_steps=None, adjoint_max_orbit_steps=None, - _record_ad=False, _collision_state_indices=None, _recorder=None): +def solve_split(rhs, collision, y0, args=None, *, t0=0.0, t1, dt0, ts, key, noise_dim, adaptive=True, dtmin=0.0, dtmax=jnp.inf, method='dopri8', orbit_dt0=None, orbit_adaptive=True, orbit_stage_execution='loop', reuse_orbit_step=True, rtol=1e-09, atol=1e-09, error_norm=None, controller='i', pcoeff=0.4, icoeff=0.3, event=None, max_steps=100000, max_orbit_steps=10000, max_pending=64, record_steps=False, step_limit=None, align_orbit=False, event_dtmax=None, _orbit_rhs_prepared=False): """Orbit advancement followed by an endpoint collision kick (Lie splitting). ``collision(t, y, h, dW, args)`` returns a ``CollisionStep`` with normalized @@ -273,16 +266,6 @@ def solve_split(rhs, collision, y0, args=None, *, t0=0.0, t1, dt0, ts, key, nois small. Stopped or failed outputs hold the last finite state; inspect ``status``. Work and Brownian memory are bounded and no inaccurate step is forced through ``dtmin`` or a memory limit.""" - if adjoint not in ('none','accepted'): - raise ValueError("adjoint must be 'none' or 'accepted'") - if adjoint == 'accepted' or _record_ad: - options=locals().copy() - from .native_ad import solve_accepted_split, _SplitRecorder - for name in ('rhs','collision','y0','args','adjoint','_record_ad','_recorder'): - options.pop(name) - if adjoint == 'accepted': - return solve_accepted_split(rhs,collision,y0,args,**options) - _recorder=_SplitRecorder(jnp.asarray(y0),options) if align_orbit and (not adaptive or not orbit_adaptive or not reuse_orbit_step): raise ValueError('orbit endpoint alignment requires adaptive orbit and collision clocks and orbit-step reuse') for name, value in (('max_steps', max_steps), ('max_orbit_steps', max_orbit_steps)): @@ -345,16 +328,9 @@ def one(s): return dict(s, ys=s['ys'].at[s['index']].set(s['y']), index=s['index'] + 1) return lax.while_loop(lambda s: (s['index'] < len(ts)) & (ts[jnp.minimum(s['index'], len(ts) - 1)] <= s['t']), one, s) s = save(s) - initial_saves = s['index'] - orbit_recorder = None - if _recorder is not None: - from .native_ad import _OrbitRecorder - s = _recorder.initialize(s) - orbit_recorder = _OrbitRecorder(max_orbit_steps, adjoint_orbit_checkpoint_steps, schedule=True) def integrate(t, y, end, guess): - return solve_rk(orbit_rhs, y, args, t0=t, t1=end, dt0=jnp.minimum(guess, end-t), ts=jnp.empty((0,), y0.dtype), method=method, adaptive=orbit_adaptive, rtol=rtol, atol=atol, stage_execution=orbit_stage_execution, error_norm=error_norm, controller=controller, pcoeff=pcoeff, icoeff=icoeff, event=event, root_find=True, max_steps=max_orbit_steps, - _recorder=orbit_recorder) + return solve_rk(orbit_rhs, y, args, t0=t, t1=end, dt0=jnp.minimum(guess, end-t), ts=jnp.empty((0,), y0.dtype), method=method, adaptive=orbit_adaptive, rtol=rtol, atol=atol, stage_execution=orbit_stage_execution, error_norm=error_norm, controller=controller, pcoeff=pcoeff, icoeff=icoeff, event=event, root_find=True, max_steps=max_orbit_steps) def kick(t, y, h, dw): result = collision(t, y, h, dw, args) @@ -394,8 +370,7 @@ def advance_with_limit(s, limit): def attempt(_): guess = s['orbit_h'] if reuse_orbit_step and orbit_adaptive else orbit_dt0 - result = integrate(s['t'], s['y'], end, guess) - orbit, schedule = result if _recorder is not None else (result, None) + orbit = integrate(s['t'], s['y'], end, guess) trial = lax.cond(orbit.status == RKStatus.SUCCESS, lambda _: kick(end, orbit.y, h, dw), lambda _: CollisionStep(orbit.y, jnp.zeros((), y0.dtype), jnp.zeros((), y0.dtype)), None) v = values(end, trial.y) finite = valid_trial(trial) & finite_values(v) @@ -426,8 +401,6 @@ def attempt(_): if record_steps: state['history'] = lax.cond(accepted, lambda history: history.at[state['accepted']].set(end), lambda history: history, state['history']) state = save(state) - if _recorder is not None: - state = _recorder.record(s, state, accepted, orbit_event, orbit, schedule, dw) return state return lax.cond(usable, attempt, lambda _: dict(s, memory=m, status=jnp.where(overflow, SplitStatus.BROWNIAN_CAPACITY, SplitStatus.MIN_STEP)), None) @@ -443,12 +416,10 @@ def advance(s): ys = jnp.where((jnp.arange(len(ts)) >= s['index'])[:, None], s['y'], s['ys']) stats = SplitStats(s['steps'], s['accepted'], s['rejected'], s['orbit_steps'], s['orbit_rejected'], s['memory'].peak, s['max_error'], s['accepted_orbit_steps'], s['max_orbit_steps_per_collision']) solution = SplitSolution(ts, ys, s['t'], s['y'], status, stats, s['history']) - return solution if _recorder is None else _recorder.finish(s, solution, initial_saves) + return solution -def trace_split(field, particles, species, *, model='GuidingCenter', tmax, timestep, times=None, n_save=100, integrator='essos_dopri8', collision_tolerance=0.001, adaptive=True, collision_dtmax=None, orbit_dt0=None, orbit_adaptive=True, orbit_stage_execution=None, reuse_orbit_step=True, collision_controller='moments', moment_tolerance=.005, moment_reference_time=.005, rtol=1e-09, atol=1e-09, controller='i', pcoeff=0.4, icoeff=0.3, speed_floor=None, spatial_diffusion=None, spatial_geometry=None, orbit=True, electric_field=None, event=None, max_steps=100000, max_orbit_steps=10000, max_pending=64, record_steps=False, execution='shared', interpret=False, step_limit=None, - adjoint='none', adjoint_checkpoint_steps=16, adjoint_orbit_checkpoint_steps=16, - adjoint_max_accepted_steps=None, adjoint_max_orbit_steps=None, _record_ad=False, _replay_tapes=None, reverse_execution='auto'): - """Batched split tracing, optionally with accepted-map reverse differentiation, using ESSOS field and particle objects. +def trace_split(field, particles, species, *, model='GuidingCenter', tmax, timestep, times=None, n_save=100, integrator='essos_dopri8', collision_tolerance=0.001, adaptive=True, collision_dtmax=None, orbit_dt0=None, orbit_adaptive=True, orbit_stage_execution=None, reuse_orbit_step=True, collision_controller='moments', moment_tolerance=.005, moment_reference_time=.005, rtol=1e-09, atol=1e-09, controller='i', pcoeff=0.4, icoeff=0.3, speed_floor=None, spatial_diffusion=None, spatial_geometry=None, orbit=True, electric_field=None, event=None, max_steps=100000, max_orbit_steps=10000, max_pending=64, record_steps=False, step_limit=None): + """Batched split tracing using ESSOS field and particle objects. Returns a ``SplitSolution`` with shared ts and particle-leading arrays. GC states are ``(position[3], v_parallel, mu)`` with **physical** mu=m*v_perp^2/(2 B) @@ -469,21 +440,6 @@ def trace_split(field, particles, species, *, model='GuidingCenter', tmax, times the collision checks at ``collision_tolerance`` and orbit checks at ``rtol/atol``; none of these is a global error bound. ``"conservative"`` keeps only the collision checks. An explicit ``step_limit(t, y, args)`` [s] replaces the moment predictor. - - ``execution="independent"`` runs one persistent GPU kernel per particle with the same - collision maps and predictor. It needs GPU float64, Threefry keys, zero electric - field, Dopri8/I and matching orbit/collision adaptation; unsupported options raise. - ``"shared"`` is ordinary JAX execution. ``interpret=True`` runs the independent - kernel under the Pallas interpreter on any backend (slow; for testing). - - ``adjoint="accepted"`` records the accepted outer/inner schedules and Wiener - increments; reverse differentiates every accepted orbit and collision map - (including spatial kicks) and freezes controller, rejection and event decisions. - ``adjoint_orbit_checkpoint_steps`` bounds the live replay segment and the two - ``adjoint_max_*`` options bound tape storage (overflow raises in reverse). With - ``reverse_execution="auto"``, uniform alpha/direct-coil cases under independent - forward execution use persistent GPU reverse; everything else uses checkpointed - JAX reverse on the selected device. ``"independent"`` rejects unsupported cases. """ from .boozer import BoozerField, guiding_center_rhs from .coils import Coils @@ -492,19 +448,6 @@ def trace_split(field, particles, species, *, model='GuidingCenter', tmax, times from .electric_field import Electric_field_zero from .constants import ELEMENTARY_CHARGE - if adjoint not in ('none','accepted'): - raise ValueError("adjoint must be 'none' or 'accepted'") - if reverse_execution not in ('auto','shared','independent'): - raise ValueError("reverse_execution must be 'auto', 'shared' or 'independent'") - if (_record_ad and _replay_tapes is not None) or (adjoint!='none' and (_record_ad or _replay_tapes is not None)): - raise ValueError('record/replay and accepted AD are separate operations') - if adjoint=='accepted' and reverse_execution=='independent' and execution!='independent': - raise ValueError('independent reverse requires independent forward execution') - if adjoint=='accepted' or _record_ad: - from .native_ad import _validate_ad_sizes - _validate_ad_sizes(adjoint_checkpoint_steps=adjoint_checkpoint_steps, - adjoint_orbit_checkpoint_steps=adjoint_orbit_checkpoint_steps, - adjoint_max_accepted_steps=adjoint_max_accepted_steps,adjoint_max_orbit_steps=adjoint_max_orbit_steps) if integrator not in ('essos_rk45', 'essos_dopri8'): raise ValueError("split integrator must be 'essos_rk45' or 'essos_dopri8'") if model not in ('GuidingCenter', 'FullOrbit'): @@ -521,10 +464,8 @@ def trace_split(field, particles, species, *, model='GuidingCenter', tmax, times raise ValueError('BackgroundSpecies must contain at least one species') field = BiotSavart(field) if isinstance(field, Coils) else field electric_field = Electric_field_zero() if electric_field is None else electric_field - if execution not in ('shared', 'independent'): - raise ValueError("split execution must be 'shared' or 'independent'") boozer, vmec = (isinstance(field, BoozerField), isinstance(field, Vmec)) - if step_limit is not None and (execution != 'shared' or boozer or vmec): + if step_limit is not None and (boozer or vmec): raise ValueError('step_limit currently requires shared Cartesian tracing') if orbit_stage_execution is None: orbit_stage_execution = 'unrolled' if integrator == 'essos_dopri8' and model == 'GuidingCenter' and isinstance(field, BiotSavart) and (not adaptive) and (not orbit_adaptive) and (jax.default_backend() == 'gpu') else 'loop' @@ -546,11 +487,8 @@ def trace_split(field, particles, species, *, model='GuidingCenter', tmax, times times = jnp.asarray(times) if times.ndim != 1: raise ValueError('times must be one-dimensional') - if execution == 'independent': - _validate_independent(field, particles, species, model=model, integrator=integrator, adaptive=adaptive, orbit_adaptive=orbit_adaptive, controller=controller, electric_field=electric_field, spatial_diffusion=spatial_diffusion, orbit=orbit, record_steps=record_steps, reuse_orbit_step=reuse_orbit_step, event=event, times=times, interpret=interpret) - field_view = _split_field_data(field) if execution == 'independent' and isinstance(field,(BiotSavart,Vmec,BoozerField)) else None moment_vmec_data=None - if vmec and moment_control and execution=='shared': + if vmec and moment_control: with jax.ensure_compile_time_eval(): moment_vmec_data=_prepare_vmec_data(field) @@ -597,7 +535,6 @@ def rhs(t, y, args,source=field): if gc: return gc_drift(t, y, args,source) return jnp.concatenate([y[3:], charge / mass * (electric_field.E_covariant(y[:3]) + jnp.cross(y[3:], source.B_contravariant(y[:3])))]) - physical_rhs=rhs if gc: points = jnp.asarray(particles.initial_xyz) b0 = jax.vmap(magnitude)(points) @@ -614,9 +551,9 @@ def rhs(t, y, args,source=field): if particles.initial_xyz_fullorbit is None or particles.initial_vxvyvz is None: raise ValueError('FullOrbit requires initial full-orbit positions and velocities') y0 = jnp.concatenate([jnp.asarray(particles.initial_xyz_fullorbit), jnp.asarray(particles.initial_vxvyvz)], axis=1) - if execution == "shared" and y0.shape[0]: + if y0.shape[0]: rhs = _prepare_rhs(rhs, jnp.zeros((), y0.dtype), y0[0], None) - def one(initial, key, tape=None): + def one(initial, key): birth = public(initial) birth_speed = jnp.sqrt(birth[3] ** 2 + 2 * birth[4] * magnitude(birth[:3]) / mass) if gc else jnp.linalg.norm(birth[3:]) if speed_floor is None: @@ -639,8 +576,7 @@ def kick(t, y, h, dw, args,source=field): displacement=guiding_center_spatial_increment(state[3]/speed,dw[:3],c, spatial_direction(y,B,source),charge*B/mass) if vmec or boozer: - y=move_in_cartesian(y,displacement,lambda q:spatial_position(q,source), - implicit=adjoint=='accepted' and source is field) + y=move_in_cartesian(y,displacement,lambda q:spatial_position(q,source)) point=public(y)[:3] else: point=point+displacement;y=y.at[:3].set(point) @@ -658,7 +594,7 @@ def position(y): return field.to_xyz(point) if vmec else point active_limit=step_limit - if moment_control and execution=='shared': + if moment_control: moment_position=position if boozer: moment_position=lambda y:y[:3] @@ -683,1006 +619,16 @@ def norm(a, b, error): if event is not None: events.extend([f if not (vmec or boozer) else lambda t, y, args, f=f: f(t, public(y), args) for f in tree_util.tree_leaves(event)]) - if _replay_tapes is not None: - from .native_ad import replay_accepted_split - saved,final=replay_accepted_split(rhs,kick,initial,None,tape,times, - method='dopri5' if integrator=='essos_rk45' else 'dopri8') - return jax.vmap(public)(saved),public(final) - solve = solve_split - if execution == 'independent': - from functools import partial - def physics(source): - drift=lambda t,y,a:physical_rhs(t,y,a,source) - collision=lambda t,y,h,dw,a:kick(t,y,h,dw,a,source) - xyz=(lambda y:source.to_xyz(public(y)[:3])) if vmec else (lambda y:y[:3]) - measure=(lambda a,b,e:jnp.maximum(jnp.linalg.norm(e[:2]),jnp.maximum(jnp.abs(e[2]),jnp.abs(e[3])/birth_speed))/(rtol+atol)) if boozer else OrbitErrorNorm(rtol,atol,birth_speed,velocity_components=1 if gc else 3,position_map=xyz if vmec else None) - limit=native_step_limit(source,species,mass,charge,initial,drift,public,xyz,lambda p:magnitude(p,source), - gc=gc,coordinate=vmec or boozer,boozer=boozer, - electric_field=electric_field,tmax=tmax,dtmax=timestep if collision_dtmax is None else collision_dtmax, - tolerance=moment_tolerance,reference_time=moment_reference_time,orbit=orbit,reference_magnitude=magnitude) if moment_control else lambda t,y,a:jnp.asarray(jnp.inf) - level=lambda t,y,a:jnp.min(jnp.asarray([f(t,y,a) for f in events])) if events else jnp.asarray(1.,y.dtype) - return drift,collision,measure,level,limit - solve = partial(_solve_independent, field=field, species=species, gc=gc, birth_speed=birth_speed, speed_floor=floor, collision_tolerance=collision_tolerance, field_view=field_view,physics=physics,mass=mass,charge=charge,spatial_diffusion=spatial_diffusion,spatial_geometry=spatial_geometry,interpret=interpret,_reverse_execution=reverse_execution) - solution = solve(rhs, kick, initial, t1=tmax, dt0=timestep, ts=times, key=key, noise_dim=(5 if spatial_diffusion else 2) if gc else 3, dtmax=timestep if collision_dtmax is None else collision_dtmax, adaptive=adaptive, orbit_dt0=orbit_dt0, orbit_adaptive=orbit_adaptive, orbit_stage_execution=orbit_stage_execution, reuse_orbit_step=reuse_orbit_step, method='dopri5' if integrator == 'essos_rk45' else 'dopri8', rtol=rtol, atol=atol, controller=controller, pcoeff=pcoeff, icoeff=icoeff, error_norm=norm, step_limit=active_limit, align_orbit=moment_control and orbit_adaptive and reuse_orbit_step, event=tuple(events), max_steps=max_steps, max_orbit_steps=max_orbit_steps, max_pending=max_pending, record_steps=record_steps, _orbit_rhs_prepared=execution == "shared", - adjoint=adjoint,adjoint_checkpoint_steps=adjoint_checkpoint_steps, - adjoint_orbit_checkpoint_steps=adjoint_orbit_checkpoint_steps, - adjoint_max_accepted_steps=adjoint_max_accepted_steps,adjoint_max_orbit_steps=adjoint_max_orbit_steps, - _record_ad=_record_ad,_collision_state_indices=tuple(range(3,initial.size)) if not gc else None) - if _record_ad: - solution,tape=solution - return solution._replace(ys=jax.vmap(public)(solution.ys),y=public(solution.y)),tape + solution = solve_split(rhs, kick, initial, t1=tmax, dt0=timestep, ts=times, key=key, noise_dim=(5 if spatial_diffusion else 2) if gc else 3, dtmax=timestep if collision_dtmax is None else collision_dtmax, adaptive=adaptive, orbit_dt0=orbit_dt0, orbit_adaptive=orbit_adaptive, orbit_stage_execution=orbit_stage_execution, reuse_orbit_step=reuse_orbit_step, method='dopri5' if integrator == 'essos_rk45' else 'dopri8', rtol=rtol, atol=atol, controller=controller, pcoeff=pcoeff, icoeff=icoeff, error_norm=norm, step_limit=active_limit, align_orbit=moment_control and orbit_adaptive and reuse_orbit_step, event=tuple(events), max_steps=max_steps, max_orbit_steps=max_orbit_steps, max_pending=max_pending, record_steps=record_steps, _orbit_rhs_prepared=True) return solution._replace(ys=jax.vmap(public)(solution.ys), y=public(solution.y)) if speed_floor is not None: _concrete_check(speed_floor, lambda x: np.ndim(x) == 0 and np.isfinite(x) and (x > 0), 'speed_floor must be positive and finite') - if _replay_tapes is not None: - return jax.jit(jax.vmap(one))(y0,particles.random_keys,_replay_tapes) - if _record_ad: - solution,tapes=jax.jit(jax.vmap(one))(y0,particles.random_keys) - return solution._replace(ts=times),tapes return jax.jit(jax.vmap(one))(y0, particles.random_keys)._replace(ts=times) - -# Independent GPU execution -from jax.extend.core import Literal -from .rk import (_select_one, _kernel_is_finite, _evaluate_inline, _bind_primitive, - _A8, _ERROR8, _DENSE8, _pack, _cross, _dot, _coil_geometry, - _coil_field_values, _coil_rhs_values, _prepare_vmec_data, _vmec_chart, - _vmec_series, _vmec_rhs_values, _vmec_xyz_values, _padded_spline, _KernelField) -from .constants import ALPHA_PARTICLE_MASS as M, ALPHA_PARTICLE_CHARGE as Q -from jax._src import random as jax_random -threefry = getattr(jax_random, "threefry2x32", None) or jax_random.prng - -def scalar(value, i): - return _select_one(value, i, 0).reshape(()) - -pack = _pack - -@lru_cache(maxsize=1) -def _kernel_erf(): - from jax.extend.core import Primitive - from jax._src.pallas.triton import lowering - from jax._src.interpreters import mlir - primitive = Primitive("essos_split_erf") - primitive.def_abstract_eval(lambda value: value) - primitive.def_impl(lax.erf) - mlir.register_lowering(primitive, mlir.lower_fun(lax.erf, multiple_results=False)) - lowering.register_lowering(primitive)(lowering._make_dispatch_table( - "erf", cuda=[lowering._Extern([jnp.float64], "__nv_erf", jnp.float64), - lowering._Extern([jnp.float32], "__nv_erff", jnp.float32)])) - return primitive - - -def _split_select(value, index, axis): - if value.dtype == jnp.bool_: - shape = [1] * value.ndim - shape[axis] = value.shape[axis] - return jnp.max((value & (jnp.arange(value.shape[axis]).reshape(shape) == index)).astype(jnp.int32), axis=axis, keepdims=True) > 0 - return _select_one(value, index, axis) - -def _flatten(function, *examples): - from jax._src.core import DropVar - from jax._src.interpreters import partial_eval - graph = jax.make_jaxpr(function)(*examples) - graph = jax.make_jaxpr(lambda *x: _evaluate_inline(graph.jaxpr, graph.consts, *x))(*examples) - aliases,expressions,kept={},{},[] - def freeze(value): - if isinstance(value,dict):return tuple((k,freeze(v)) for k,v in sorted(value.items())) - if isinstance(value,(tuple,list)):return tuple(freeze(v) for v in value) - if isinstance(value,np.ndarray):return value.dtype.str,value.shape,value.tobytes() - hash(value) - return value - def operand(value): - if not isinstance(value,Literal):return aliases.get(value,value) - data=np.asarray(value.val) - return ('literal',data.dtype.str,data.shape,data.tobytes(),value.aval.weak_type) - for eq in graph.jaxpr.eqns: - inputs=[v if isinstance(v,Literal) else aliases.get(v,v) for v in eq.invars] - key=None - if not eq.effects: - try:key=eq.primitive,tuple(operand(v) for v in inputs),freeze(eq.params) - except TypeError:pass - if key is not None and key in expressions: - index=expressions[key];previous=kept[index] - # Retain newly used outputs when an earlier identical call dropped - # them. A DropVar must never become an input alias. - outputs=[new if isinstance(old,DropVar) else old for old,new in zip(previous.outvars,eq.outvars)] - kept[index]=previous.replace(outvars=outputs) - aliases.update((new,old) for new,old in zip(eq.outvars,outputs) if not isinstance(new,DropVar)) - else: - kept.append(eq.replace(invars=inputs)) - if key is not None:expressions[key]=len(kept)-1 - graph=type(graph)(graph.jaxpr.replace(eqns=kept,outvars=[v if isinstance(v,Literal) else aliases.get(v,v) for v in graph.jaxpr.outvars]),graph.consts) - live,_=partial_eval.dce_jaxpr(graph.jaxpr,[True]*len(graph.jaxpr.outvars),instantiate=True) - return type(graph)(live,graph.consts) - -def _kernel_integer_power(value, exponent): - """Use the same power-by-squaring order with literal-safe GPU operations.""" - if exponent == 0: - return jnp.ones_like(value) - reciprocal = exponent < 0 - exponent = abs(exponent) - result = None - while exponent: - exponent, odd = divmod(exponent, 2) - if odd: - result = value if result is None else result * value - if exponent: - value = value * value - return jnp.ones_like(result) / result if reciprocal else result - - -def _evaluate_kernel(graph, constants, *arguments, scalarize=False): - literals={} - def local_constant(value): - if not isinstance(value,np.ndarray) or not value.ndim:return value - key=value.dtype.str,value.shape,value.tobytes() - if key not in literals: - first=value.flat[0] - if np.issubdtype(value.dtype,np.number) and value.tobytes()==np.arange(value.size,dtype=value.dtype).tobytes(): - result=jnp.arange(value.size,dtype=value.dtype).reshape(value.shape) - else: - result=jnp.full(value.shape,first.item(),dtype=value.dtype) - index=jnp.arange(value.size).reshape(value.shape) - for i,item in enumerate(value.flat): - if item.tobytes()!=first.tobytes(): - result=jnp.where(index==i,jnp.asarray(item.item(),dtype=value.dtype),result) - literals[key]=result - return literals[key] - def shrink(value): - return value.reshape(()) if scalarize and getattr(value, 'size', 0) == 1 else value - constants = tuple((shrink(v) for v in constants)) - env = dict(zip(graph.constvars, constants)) - env.update(zip(graph.invars, arguments)) - - def read(v): - return np.asarray(v.val, dtype=v.aval.dtype) if isinstance(v, Literal) else env[v] - for eq in graph.eqns: - vals = [read(v) for v in eq.invars] - name, p = (eq.primitive.name, eq.params) - if scalarize and len(eq.outvars) == 1 and (eq.outvars[0].aval.size == 1): - if name in ('broadcast_in_dim', 'reshape', 'squeeze', 'slice', 'dynamic_slice', 'gather', 'reduce_sum', 'reduce_or', 'reduce_and', 'stack'): - if vals[0].size == 1: - env[eq.outvars[0]] = shrink(vals[0]) - continue - if name == 'dynamic_update_slice' and vals[0].size == vals[1].size == 1: - env[eq.outvars[0]] = shrink(vals[1]) - continue - numpy_ops = {'add': np.add, 'add_any': np.add, 'sub': np.subtract, 'mul': np.multiply, 'div': np.divide, 'neg': np.negative, 'abs': np.abs, 'sqrt': np.sqrt, 'log': np.log, 'exp': np.exp, 'pow': np.power, 'square': np.square, 'max': np.maximum, 'min': np.minimum, 'lt': np.less, 'le': np.less_equal, 'gt': np.greater, 'ge': np.greater_equal, 'eq': np.equal, 'ne': np.not_equal, 'and': np.bitwise_and, 'or': np.bitwise_or} - if all((not isinstance(v, jax.core.Tracer) for v in vals)): - vals=[np.asarray(value) for value in vals] - folded = None - if name in numpy_ops: - folded = [numpy_ops[name](*vals)] - elif name == 'integer_pow': - folded = [np.power(vals[0], p['y'])] - elif name == 'convert_element_type': - folded = [np.asarray(vals[0], dtype=p['new_dtype'])] - elif name == 'broadcast_in_dim': - shape=[1]*len(p['shape']) - for i,axis in enumerate(p['broadcast_dimensions']):shape[axis]=vals[0].shape[i] - folded=[np.broadcast_to(np.reshape(vals[0],shape),p['shape'])] - elif name in ('reshape','squeeze','transpose'): - value=vals[0] - dimensions=p.get('dimensions') if name=='reshape' else p.get('permutation') if name=='transpose' else None - if dimensions is not None:value=np.transpose(value,dimensions) - folded=[np.reshape(value,eq.outvars[0].aval.shape)] - elif name=='iota': - shape=[1]*len(p['shape']);shape[p['dimension']]=p['shape'][p['dimension']] - folded=[np.broadcast_to(np.arange(shape[p['dimension']],dtype=p['dtype']).reshape(shape),p['shape'])] - elif name in ('concatenate','stack'): - folded=[(np.concatenate if name=='concatenate' else np.stack)(vals,axis=p.get('dimension',p.get('axis')))] - if folded is not None: - env.update(((out, np.asarray(value, dtype=out.aval.dtype)) for out, value in zip(eq.outvars, folded))) - continue - vals=[local_constant(value) for value in vals] - if name == 'slice': - value = vals[0] - for axis, (start, end, size) in enumerate(zip(p['start_indices'], p['limit_indices'], value.shape)): - if start != 0 or end != size: - if end - start != 1: - raise ValueError(('slice', p)) - value = _split_select(value, start, axis) - result = [value] - elif name == 'dynamic_slice': - value = vals[0] - for axis, (index, size, wanted) in enumerate(zip(vals[1:], value.shape, p['slice_sizes'])): - if size != wanted: - if wanted != 1: - raise ValueError(('dynamic slice', p)) - value = _split_select(value, jnp.clip(index, 0, size - 1), axis) - result = [value] - elif name == 'gather' and vals[0].ndim == 1 and (p['slice_sizes'] == (1,)): - operand, indices = vals - if indices.size != 1: - raise ValueError(('gather indices', indices.shape, p)) - index = jnp.clip(indices.reshape(()), 0, len(operand) - 1) - result = [_split_select(operand, index, 0).reshape(eq.outvars[0].aval.shape)] - elif name == 'stack' and len(vals) == 1: - result = [jnp.expand_dims(vals[0], p['axis'])] - elif name == 'unstack': - result = [_split_select(vals[0], i, p['axis']).reshape(out.aval.shape) for i, out in enumerate(eq.outvars)] - elif name == 'split' and vals[0].ndim == 1 and all((out.aval.size == 1 for out in eq.outvars)): - result = [_split_select(vals[0], i, 0).reshape(out.aval.shape) for i, out in enumerate(eq.outvars)] - elif name == 'reduce_or': - result = [jnp.max(vals[0].astype(jnp.int32), axis=p['axes']) > 0] - elif name == 'reduce_and': - result = [jnp.min(vals[0].astype(jnp.int32), axis=p['axes']) > 0] - elif name == 'add_any': - result = [vals[0] + vals[1]] - elif name == 'pad' and vals[0].size == 1 and (len(p['padding_config']) == 1): - low, _, _ = p['padding_config'][0] - result = [jnp.where(jnp.arange(eq.outvars[0].aval.size) == low, vals[0].reshape(()), vals[1])] - elif name == 'erf': - result = [_kernel_erf().bind(vals[0])] - elif name == 'integer_pow': - result = [_kernel_integer_power(vals[0], p['y'])] - elif name == 'le_to': - result = [jnp.less_equal(vals[0], vals[1]) | jnp.isnan(vals[1])] - elif name in ('concatenate', 'stack') and (len(vals) > 2 or (scalarize and all((v.size == 1 for v in vals)))): - outshape = eq.outvars[0].aval.shape - if any((v.size != 1 for v in vals)): - raise ValueError(('concatenate', p)) - result = jnp.zeros((len(vals),), vals[0].dtype) - for i, v in enumerate(vals): - result = jnp.where(jnp.arange(len(vals)) == i, v.reshape(()), result) - result = [result.reshape(outshape)] - elif name == 'is_finite': - result = [_kernel_is_finite(vals[0]) if isinstance(vals[0], jax.core.Tracer) else np.isfinite(vals[0])] - else: - parameters = dict(p) - if name == 'convert_element_type' and parameters.get('weak_type'): - parameters['weak_type'] = False - result = _bind_primitive(eq.primitive, vals, parameters) - if not eq.primitive.multiple_results: - result = [result] - env.update(((v, shrink(value)) for v, value in zip(eq.outvars, result))) - return tuple((read(v) for v in graph.outvars)) - -def _evaluate_surface(graph, constants, state): - """Scalarize short state-vector operations in a spatial level set. - - A physical five/six-vector is represented as scalar components, avoiding - non-power-of-two local arrays. This covers scalar indexing, vector norms - and dot products used by the toroidal/spherical safety surfaces. - """ - word = lambda value, i: _select_one(value, i, 0).reshape(()) - env = dict(zip(graph.constvars, constants)) - env[graph.invars[0]] = tuple((word(state, i) for i in range(graph.invars[0].aval.size))) - - def read(var): - return np.asarray(var.val, dtype=var.aval.dtype) if isinstance(var, Literal) else env[var] - for eq in graph.eqns: - values = [read(var) for var in eq.invars] - name, p = (eq.primitive.name, eq.params) - vector = lambda x: isinstance(x, tuple) - if name == 'slice' and vector(values[0]): - result = [values[0][p['start_indices'][0]:p['limit_indices'][0]:None if p.get('strides') is None else p['strides'][0]]] - elif name == 'squeeze' and vector(values[0]) and (len(values[0]) == 1): - result = [values[0][0]] - elif name == 'unstack': - result = list(values[0]) - elif name in ('stack', 'concatenate'): - result = [tuple((item for value in values for item in (value if vector(value) else (value,))))] - elif name in ('reshape', 'broadcast_in_dim'): - size = eq.outvars[0].aval.size - result = [values[0] if vector(values[0]) else (values[0],) * size] - if not eq.outvars[0].aval.shape: - result = [result[0][0]] - elif name in ('reduce_sum', 'reduce_max', 'reduce_min') and vector(values[0]): - value = values[0][0] - combine = {'reduce_sum': lambda a, b: a + b, 'reduce_max': jnp.maximum, 'reduce_min': jnp.minimum}[name] - for item in values[0][1:]: - value = combine(value, item) - result = [value] - elif name == 'dot_general' and all((vector(value) for value in values)): - result = [sum((a * b for a, b in zip(*values)))] - elif name == 'gather' and vector(values[0]) and (len(values[1]) == 1): - index = values[1][0] - value = values[0][0] - for i, item in enumerate(values[0]): - value = jnp.where(index == i, item, value) - result = [(value,) if eq.outvars[0].aval.shape else value] - else: - parameters = dict(p) - if name == 'convert_element_type': - parameters['weak_type'] = False - if any((vector(value) for value in values)): - width = max((len(value) for value in values if vector(value))) - elements = [] - for i in range(width): - elements.append(_bind_primitive(eq.primitive, [value[i] if vector(value) else value for value in values], parameters)) - result = [tuple(elements)] - else: - value = _bind_primitive(eq.primitive, values, parameters) - result = value if eq.primitive.multiple_results else [value] - env.update(zip(eq.outvars, result)) - result = read(graph.outvars[0]) - return result[0] if isinstance(result, tuple) else result - -def _draw_normal(key, dim): - if not jax.config.jax_threefry_partitionable: - raise ValueError('The exact adapter currently requires partitionable Threefry') - word = lambda a, i: _select_one(a, i, 0).reshape(()) - k0, k1 = (word(key, 0), word(key, 1)) - a, b = threefry._threefry2x32_lowering(k0, k1, jnp.asarray(0, jnp.uint32), jnp.arange(2, dtype=jnp.uint32), use_rolled_loops=False) - next_key = jnp.where(jnp.arange(2) == 0, word(a, 0), word(b, 0)) - d0, d1 = (word(a, 1), word(b, 1)) - padded = 1 << (dim - 1).bit_length() - hi, lo = threefry._threefry2x32_lowering(d0, d1, jnp.asarray(0, jnp.uint32), jnp.arange(padded, dtype=jnp.uint32), use_rolled_loops=False) - bits = hi.astype(jnp.uint64) << jnp.uint64(32) | lo.astype(jnp.uint64) - unit = jax.lax.bitcast_convert_type(bits >> jnp.uint64(12) | jnp.uint64(4607182418800017408), jnp.float64) - 1.0 - lower = np.nextafter(np.float64(-1.0), np.float64(0.0)) - uniform = jnp.maximum(lower, unit * (1.0 - lower) + lower) - return (next_key, jnp.sqrt(jnp.asarray(2.0, jnp.float64)) * jax.lax.erf_inv(uniform)) - -@lru_cache(None) -def _register_kernel_placement(): - """Lower unspecified JAX placement as identity on already resident values.""" - from jax._src.dispatch import device_put_p - from jax._src.pallas.triton import lowering - if device_put_p in lowering.triton_lowering_rules: - return lowering.triton_lowering_rules[device_put_p] - @lowering.register_lowering(device_put_p) - def placement(ctx,*values,**parameters): - if any(v is not None for name in ('devices','srcs') for v in parameters.get(name,())): - raise NotImplementedError('explicit device transfers are unsupported inside native GPU kernels') - return values - return placement - - -def _prepare_split_kernel(arrays, factory, times, *, adaptive, dt0, dtmax, speed, tolerance=1e-09, noise_dim=2, max_steps=100000, max_orbit_steps=10000, max_pending=64, warps=1, state_size=8, bank_components=8, orbit_dt0=None, final_time=None, align_orbit=False, interpret=False, - record=False, adjoint_checkpoint_steps=16, adjoint_orbit_checkpoint_steps=16, - adjoint_max_accepted_steps=None, adjoint_max_orbit_steps=None, collision_indices=None): - """Persistent particle loop; factory also supplies the optional predictor.""" - from jax.experimental import pallas as pl - from jax.experimental.pallas import triton as plt - if not interpret:_register_kernel_placement() - times = jnp.asarray(times, jnp.float64) - nsave, stages, bank_size = (len(times), len(_ERROR8), 16) - matrix = np.zeros((bank_size, bank_size)) - for i, row in enumerate(_A8, 1): - matrix[i, :len(row)] = row - dense = np.zeros((len(_DENSE8[0]), bank_size)) - dense[:, :stages] = np.asarray(_DENSE8).T - inputs = tuple(arrays) + (jnp.asarray(matrix), jnp.asarray(dense), jnp.asarray(tuple(_ERROR8) + (0.0,) * (bank_size - stages)), jnp.asarray(times), jnp.asarray([0.0, dt0, dtmax, times[-1] if final_time is None else final_time, speed, 2 * tolerance, dt0 if orbit_dt0 is None else orbit_dt0], jnp.float64)) - capacity = max_pending if adaptive else 1 - noise_size = 1 << (noise_dim - 1).bit_length() - - if record: - from .native_ad import _SplitCheckpointTape, _TapeView - outer_capacity = min(max_steps,adjoint_max_accepted_steps or max_steps) - inner_capacity = adjoint_max_orbit_steps or max_steps - inner_segment = adjoint_orbit_checkpoint_steps - outer_segment = adjoint_checkpoint_steps - if collision_indices is None: - node_capacity = (inner_capacity+inner_segment-1)//inner_segment+outer_capacity-1 - derivative_capacity = node_capacity - else: - nonempty_maps = min(outer_capacity,inner_capacity) - node_capacity = nonempty_maps+(inner_capacity-nonempty_maps)//inner_segment - derivative_capacity = (inner_capacity-1)//inner_segment - endpoint_size = 8 if collision_indices is None else len(collision_indices) - shape = lambda dims,dtype=jnp.float64:jax.ShapeDtypeStruct(dims,dtype) - tape_shapes = _SplitCheckpointTape( - shape((outer_capacity,2)),shape((outer_capacity,noise_dim)),shape((outer_capacity,),jnp.bool_), - shape((outer_capacity+1,),jnp.int32),shape((inner_capacity,2)),shape((outer_capacity,)), - shape(((outer_capacity+outer_segment-1)//outer_segment,8)), - shape((),jnp.int32),shape((),jnp.int32),shape((),jnp.int32),shape((),jnp.int32),shape((),jnp.int32), - shape((outer_capacity+1,),jnp.int32),shape((node_capacity,8)), - shape((derivative_capacity,8)),shape((outer_capacity,endpoint_size)),shape((),jnp.int32)) - padded_tape_shapes = tuple(shape(tuple(1< 0 if isinstance(x,jax.core.Tracer) else jnp.asarray(np.isfinite(x).all()) - bi = jnp.arange(bank_size)[:, None] - - def polynomial(y, endpoint, bank, fraction): - weights = jnp.zeros((bank_size,), y.dtype) - for power in range(dense.shape[0]): - weights = weights * fraction + dref[power, :] - result = y + expand(jnp.sum((weights * fraction)[:, None] * bank, axis=0)) - return jnp.where(fraction == 0, y, jnp.where(fraction == 1, endpoint, result)) - - def orbit(y, t, end, guess, outer_index, inner_index, node_index): - old = level(y) - start = (t, y, dynamic_rhs(y), jnp.minimum(guess, end - t), old, integer(0), integer(0), integer(0)) + ((zero+1,) if record else ()) - - def step(c): - t, y, f0, guess, old, status, trials, rejected = c[:8] - last_fraction = c[8] if record else zero+1 - remaining = end - t - h = jnp.minimum(guess, remaining) - close = remaining - h <= 8 * jnp.finfo(y.dtype).eps * jnp.maximum(jnp.abs(t), jnp.abs(end)) - h = jnp.where(close, remaining, h) - stop = jnp.where(h == remaining, end, t + h) - bank = jnp.where(bi == 0, h * f0[None, :], jnp.zeros((bank_size, bank_components), y.dtype)) - - def stage(i, carry): - bank, _, checked = carry - contribution = jnp.where(bi < i, aref[i, :][:, None] * bank, 0.0) - yi = y + expand(jnp.sum(contribution, axis=0)) - fi = dynamic_rhs(yi) - increment = h * fi - return (jnp.where(bi == i, increment[None, :], bank), fi, checked & finite(increment) & finite(fi)) - bank, f1, stage_finite = jax.lax.fori_loop(1, stages, stage, (bank, f0, finite(h * f0) & finite(f0))) - endpoint = y + expand(jnp.sum(jnp.where(bi < stages - 1, aref[stages - 1, :][:, None] * bank, 0.0), axis=0)) - embedded = expand(jnp.sum(jnp.where(bi < stages, eref[...][:, None] * bank, 0.0), axis=0)) - error = norm(endpoint, embedded, birth_speed, budget) if adaptive else zero - new = level(endpoint) - valid = finite(endpoint) & stage_finite & finite(f1) & finite(embedded) & finite(error) & finite(new) & (error >= 0) - accept = valid & ((error <= 1) | (not adaptive)) & (stop > t) - crossing = accept & (((old > 0) != (new > 0)) | (new == 0)) - factor = jnp.where(valid, jnp.clip(0.9 * jnp.maximum(error, 1e-16) ** (-1 / 8), 0.2, 5.0), 0.2) - proposal = h * jnp.where(accept, factor, jnp.minimum(factor, 0.9)) if adaptive else first_orbit_h - - def bisect(c): - lo, hi, i, ok = c - mid = (lo + hi) / 2 - value = level(polynomial(y, endpoint, bank, mid)) - same = (value > 0) == (old > 0) - return (jnp.where(same, mid, lo), jnp.where(same, hi, mid), i + 1, ok & finite(value)) - lo, hi, _, ok = jax.lax.while_loop(lambda c: crossing & c[3] & (c[2] < 52), bisect, (zero, zero + 1, integer(0), jnp.asarray(True))) - root = (lo + hi) / 2 - failed_root = crossing & ~ok - accept &= ~failed_root - proposal = jnp.where(failed_root, 0.2 * h, proposal) - stop_y = jax.lax.cond(crossing, lambda _: polynomial(y, endpoint, bank, root), lambda _: endpoint, None) - stuck = (stop <= t) | ~accept & ((h <= 0) | (t + proposal <= t)) - status = jnp.where(crossing & ok, 1, jnp.where(stuck, jnp.where(valid, 3, 4), 0)).astype(jnp.int32) - if not adaptive: - status = jnp.where(~valid, 4, status) - if record: - local_index = trials-rejected - slot = inner_index+local_index - node = node_index+local_index//inner_segment - def save_step(_): - tape_refs['orbit_times'][slot,0] = t - tape_refs['orbit_times'][slot,1] = h - def checkpoint(_): - tape_refs['orbit_states'][node,:] = y - if collision_indices is None: - tape_refs['orbit_derivatives'][node,:] = expand(f0) - elif derivative_capacity: - def derivative(_): - tape_refs['orbit_derivatives'][node-outer_index-1,:] = expand(f0) - jax.lax.cond(local_index>0,derivative,lambda _:None,None) - jax.lax.cond(local_index%inner_segment==0,checkpoint,lambda _:None,None) - jax.lax.cond(accept&(outer_index 0) & finite(first_h) & (ceiling > 0) & ~jnp.isnan(ceiling) & (first_orbit_h > 0) & finite(first_orbit_h) & (birth_speed > 0) & finite(birth_speed) & (budget > 0) & finite(budget) & (final_t >= 0) & finite(final_t) - previous = zero - for i in range(nsave): - stamp = tsref[i] - valid_inputs &= finite(stamp) & (stamp >= previous) & (stamp <= final_t) - previous = stamp - initial_status = jnp.where(valid_inputs, jnp.where(finite(y0) & finite(level(y0)), jnp.where(level(y0) <= 0, 1, 0), 4), 5).astype(jnp.int32) - index = save(y0, zero, integer(0)) - if record: - tape_refs['initial_saves'][...] = index - start = (zero, y0, jnp.minimum(first_h, ceiling), first_orbit_h, kref[...], integer(0), integer(0), index, integer(0), initial_status, integer(0), integer(0), integer(0), integer(0), integer(0), zero,integer(0),integer(0)) + ((integer(0),)*3 if record else ()) - - def advance(c): - t, y, h, oh, key, size, peak, index, grid, status, trials, accepted, rejected, ot, orr, maxerr,path_steps,path_max = c[:18] - ad_count,ad_inner,ad_nodes = c[18:] if record else (integer(0),)*3 - barrier = jnp.where(index < nsave, tsref[jnp.minimum(index, nsave - 1)], final_t) - limit=step_limit(t,y) if adaptive and step_limit is not None else zero+jnp.inf - valid_limit=(limit>0)&~jnp.isnan(limit) - if adaptive: - wanted = jnp.minimum(t + jnp.minimum(h,limit), barrier) - if align_orbit: - length=wanted-t - near=finite(oh)&(oh>0)&(wanted=.9*length)&(oh0,lambda _:ends[size-1],lambda _:wanted,None) - draw=(size==0)|((wanted 0) & (end > t) - - def attempt(_): - stop, endpoint, next_oh, os, orbit_trials, orbit_rejected, fraction = orbit(y, t, end, jnp.minimum(oh, h) if adaptive else jnp.minimum(first_orbit_h, h), ad_count, ad_inner, ad_nodes) - - def collide(_): - return kick(endpoint, h, increments[size - 1, :]) - trial, ed, es = jax.lax.cond(os == 0, collide, lambda _: (endpoint, zero, zero), None) - valid = finite(trial) & finite(level(trial)) & finite(ed) & finite(es) & (ed >= 0) & (es >= 0) - error = jnp.maximum(ed, es) - accept = (os == 0) & valid & ((error <= 1) | (not adaptive)) - reject = (os == 0) & valid & ~accept - event = os == 1 - ns = jnp.where(os != 0, os, jnp.where(~valid, 4, jnp.where(accept & (level(trial) <= 0), 1, 0))).astype(jnp.int32) - if adaptive: - factor = 0.9 * jnp.minimum(jnp.maximum(ed, 1e-30) ** (-0.5), jnp.maximum(es, 1e-30) ** (-2 / 3)) - upper = jnp.where(accept, zero + 1.5, zero + 0.9) - factor = jnp.maximum(zero + 0.1, jnp.minimum(factor, upper)) - nh = jnp.minimum(h * factor, ceiling) - else: - nh = first_h - nt = jnp.where(event, stop, jnp.where(accept, end, t)) - ny = jnp.where(event, endpoint, jnp.where(accept, trial, y)) - ni = save(ny, nt, index) - committed = accept|event - inner_count = orbit_trials-orbit_rejected - if record: - nodes = (inner_count+inner_segment-1)//inner_segment - def commit(_): - tape_refs['times'][ad_count,0],tape_refs['times'][ad_count,1] = t,nt - for component in range(noise_dim): - tape_refs['increments'][ad_count,component] = increments[size-1,component] - tape_refs['kicks'][ad_count] = accept - tape_refs['offsets'][ad_count+1] = ad_inner+inner_count - tape_refs['checkpoint_offsets'][ad_count+1] = ad_nodes+nodes - tape_refs['fractions'][ad_count] = fraction - endpoint_components = range(8) if collision_indices is None else collision_indices - for component,index_ in enumerate(endpoint_components): - tape_refs['orbit_endpoints'][ad_count,component] = scalar(endpoint,index_) - def outer_checkpoint(_): - tape_refs['states'][ad_count//outer_segment,:] = y - jax.lax.cond(ad_count%outer_segment==0,outer_checkpoint,lambda _:None,None) - jax.lax.cond(committed&(ad_count0)&(value<=axis[index]),index-1,index) - index=jnp.where((indexaxis[index+1]),index+1,index) - else: - def search(_, bounds): - low, high = bounds - middle = (low+high)//2 - below = axis[jnp.minimum(middle,n-1)] < value - return jnp.where(below,middle+1,low),jnp.where(below,high,middle) - insertion, _ = lax.fori_loop(0,n.bit_length(),search,(jnp.int32(0),jnp.int32(n))) - index = jnp.clip(insertion-1,0,n-2) - indices.append(index) - fractions.append((value-axis[index])/(axis[index+1]-axis[index])) - outside |= (valueaxis[n-1]) - result = jnp.asarray(0., y.dtype) - for corner in range(8): - index, weight = jnp.int32(0), jnp.asarray(1., y.dtype) - for axis, (start, fraction, n) in enumerate(zip(indices,fractions,shape)): - high = (corner>>(2-axis))&1 - index = index*n+start+high - weight *= fraction if high else 1-fraction - result += weight*refs[3][jnp.clip(index,0,int(np.prod(shape))-1)] - distance=jnp.where(outside,refs[4][()],result) - return jnp.maximum(0.,distance+refs[5][()]) if stopping else distance - return arrays, level - - -def _prepare_split_field(field, species, *, gc, collision_tolerance, speed_floor, - event=None, state_size=5, field_view=None,mass=M,charge=Q,spatial_diffusion=False): - """Reuse the orbit PR's private field views, carrying physical mu per kick.""" - from .fields import BiotSavart, Vmec - from .boozer import guiding_center_rhs - coil, vmec = isinstance(field, BiotSavart), isinstance(field, Vmec) - if spatial_diffusion and (not gc or not coil): - raise ValueError('the native coordinate adapter does not yet transform GC spatial diffusion') - arrays, metadata = _split_field_data(field) if field_view is None else field_view - if coil: - nc, nq = metadata - nfield = len(arrays) - cg = _coefficient_graph(species, not gc,mass,charge) - # Keep the per-particle floor as explicit data, not a captured map constant. - def gc_map(v, xi, h, dw, floor, *c): - out = guiding_center_collision_step(v, xi, h, dw, CollisionCoefficients(*c), - tolerance=collision_tolerance, speed_floor=floor) - return out.y, out.drift_error, out.diffusion_error - gg = _flatten(gc_map, jnp.asarray(4e5), jnp.asarray(.5), jnp.asarray(1e-7), - jnp.zeros(2),jnp.asarray(0.), *(jnp.asarray(1.),)*6) if gc else None - functions = tree_util.tree_leaves(event) - tables = [function._kernel_view() for function in functions if isinstance(function,ClassifierStop)] - functions = [function for function in functions if not isinstance(function,ClassifierStop)] - eg = _flatten(lambda y: jnp.min(jnp.asarray([f(0., y, None) for f in functions])), - jnp.zeros(state_size)) if functions else None - event_arrays = [] - if eg is not None: - for value in eg.consts: - if value.ndim > 1: - raise ValueError("independent stopping surfaces require scalar/vector constants") - event_arrays.append(value if value.ndim == 0 else - jnp.pad(value, (0, (1 << (value.size-1).bit_length())-value.size))) - all_arrays = (tuple(arrays)+tuple(cg.consts)+(tuple(gg.consts)+(jnp.asarray(speed_floor),) if gc else ())+tuple(event_arrays) - +tuple(value for values,_ in tables for value in values)) - - def factory(refs, y0): - frefs = refs[:nfield] - ci = nfield+len(cg.consts) - cc = [ref[...] for ref in refs[nfield:ci]] - mc = [ref[...] for ref in refs[ci:ci+len(gg.consts)+1]] if gc else () - event_start = ci+(len(gg.consts)+1 if gc else 0) - erefs = refs[event_start:event_start+len(event_arrays)] - ec = ([ref[...] if value.ndim == 0 else tuple(scalar(ref[...], i) for i in range(value.size)) - for ref, value in zip(erefs, eg.consts)] if eg is not None else ()) - zero = scalar(y0, 7) - def orbit_state(y): - return pack(tuple(scalar(y, i) for i in (0, 1, 2, 3, 5))+(zero, zero, zero)) - if coil: - def rhs(y): - value = _coil_rhs_values(frefs, y, nc, nq, "gc" if gc else "lorentz", 0., mass, charge,scalar(y,4) if gc else None) - return pack(tuple(scalar(value, i) for i in range(4))+(zero, zero, zero, zero)) if gc else value - def magnitude(y): - B, _ = _coil_field_values(frefs, tuple(scalar(y, i) for i in range(3)), nc, nq, False) - return jnp.sqrt(_dot(B, B)) - def norm(y, e, speed, budget): - pos = jnp.sqrt(sum(scalar(e, i)**2 for i in range(3))) - vel = jnp.abs(scalar(e, 3)) if gc else jnp.sqrt(sum(scalar(e, i)**2 for i in range(3, 6))) - return jnp.maximum(pos, vel/speed)/budget - elif vmec: - def rhs(y): - value = _vmec_rhs_values(frefs, orbit_state(y), metadata, mass, charge,0.,scalar(y,4)) - return pack(tuple(scalar(value, i) for i in range(4))+(zero, scalar(value, 4), zero, zero)) - magnitude = lambda y: _vmec_series(frefs, _vmec_chart(orbit_state(y)), metadata['bmnc'])[0] - def norm(y, e, speed, budget): - a = _vmec_xyz_values(frefs, orbit_state(y), metadata) - b = _vmec_xyz_values(frefs, orbit_state(y-e), metadata) - return jnp.maximum(jnp.sqrt(sum((u-v)**2 for u, v in zip(a, b))), jnp.abs(scalar(e, 3))/speed)/budget - else: - model = _KernelField(frefs, float(field.psi0)) - def physical(y): - value = guiding_center_rhs(model, y, y[4]/mass, mass, charge) - return pack(tuple(value[i] for i in range(4))+(zero, zero, zero, zero)) - graph = _flatten(physical, y0) - rhs = lambda y: _evaluate_kernel(graph.jaxpr, graph.consts, y)[0] - bg = _flatten(lambda y: model.modB_derivatives(jnp.sqrt(y[0]**2+y[1]**2), - jnp.arctan2(y[1], y[0]), y[2])[0], y0) - magnitude = lambda y: _evaluate_kernel(bg.jaxpr, bg.consts, y)[0] - def norm(y, e, speed, budget): - return jnp.maximum(jnp.sqrt(scalar(e, 0)**2+scalar(e, 1)**2), - jnp.maximum(jnp.abs(scalar(e, 2)), jnp.abs(scalar(e, 3))/speed))/budget - table_levels = [] - start = event_start+len(event_arrays) - for values, evaluate in tables: - table_levels.append(lambda y, evaluate=evaluate, data=refs[start:start+len(values)]:evaluate(data,y)) - start += len(values) - def level(y): - result = _evaluate_surface(eg.jaxpr,ec,y) if eg is not None else zero+jnp.inf - for function in table_levels: - result = jnp.minimum(result,function(y)) - return result if eg is not None or table_levels else zero+1. - def kick(y, h, dw): - if gc and spatial_diffusion: - vector,_=_coil_field_values(frefs,tuple(scalar(y,i) for i in range(3)),nc,nq,False) - B=jnp.sqrt(_dot(vector,vector)) - else: - B = magnitude(y) if gc else zero+1. - speed = jnp.sqrt(scalar(y, 3)**2+2*scalar(y, 4)*B/mass) if gc else jnp.sqrt(sum(scalar(y, i)**2 for i in range(3, 6))) - c = _evaluate_kernel(cg.jaxpr, cc, speed, scalarize=True) - if gc: - xi=scalar(y,3)/speed - velocity_noise=pack(tuple(scalar(dw,i) for i in (3,4))) if spatial_diffusion else dw - changed, ed, es = _evaluate_kernel(gg.jaxpr, mc[:-1], speed, xi, h, velocity_noise,mc[-1], *c, scalarize=True) - v, xi = scalar(changed, 0), scalar(changed, 1) - if spatial_diffusion: - unit=tuple(b/B for b in vector);noise=tuple(scalar(dw,i) for i in range(3)) - scale=_diffusion_root(2*guiding_center_spatial_diffusion(scalar(y,3)/speed,CollisionCoefficients(*c),charge*B/mass)) - parallel=_dot(unit,noise) - point=tuple(scalar(y,i)+scale*(noise[i]-unit[i]*parallel) for i in range(3)) - y=pack(point+tuple(scalar(y,i) for i in range(3,8))) - B=magnitude(y) - result = jnp.where(jnp.arange(8) == 3, v*xi, - jnp.where(jnp.arange(8) == 4, mass*v*v*(1-xi*xi)/(2*B), y)) - else: - # Particle map (57)-(62), scalarized in the velocity-aligned basis. - dp, dt, ddp, q, _, dk = c - drift = q+ddp+2*(dp-dt)/speed - delta = (_diffusion_root(2*dt)*scalar(dw, 0), _diffusion_root(2*dt)*scalar(dw, 1), - drift*h+_diffusion_root(2*dp)*scalar(dw, 2)+.5*ddp*(scalar(dw, 2)**2-h)) - direction = tuple(scalar(y, i)/speed for i in range(3, 6)) - a = tuple(jnp.abs(v) for v in direction) - i0 = (a[0] <= a[1]) & (a[0] <= a[2]); i1 = ~i0 & (a[1] <= a[2]) - ref = tuple(v.astype(y.dtype) for v in (i0, i1, ~i0 & ~i1)) - first = _cross(direction, ref); length = jnp.sqrt(_dot(first, first)) - first = tuple(v/length for v in first); second = _cross(direction, first) - velocity = tuple(scalar(y, 3+i)+first[i]*delta[0]+second[i]*delta[1]+direction[i]*delta[2] for i in range(3)) - result = pack(tuple(scalar(y, i) for i in range(3))+velocity+(zero, zero)) - eps = collision_tolerance*(jnp.abs(drift)*h+jnp.sqrt(jnp.maximum(2*dp*h, 0.))) - def ratio(numerator, denominator): - safe = jnp.where(denominator > 0, denominator, zero+1.) - return jnp.where(denominator > 0, numerator/safe, - jnp.where(numerator == 0, zero, zero+jnp.inf)) - ed = ratio(.5*jnp.abs(drift*dk)*h*h, eps) - es = ratio(ddp**2*jnp.abs(scalar(dw, 2))**3, 6*eps*jnp.sqrt(jnp.maximum(dp, 0.))) - valid = _kernel_is_finite(B) & (B > 0) & (speed > 0) & (c[0] >= 0) & (c[1] >= 0) - for value in c: - valid &= _kernel_is_finite(value) if isinstance(value, jax.core.Tracer) else np.isfinite(value) - return jnp.where(valid, result, jnp.nan), ed, es - return rhs, kick, norm, level - return all_arrays, factory - - -def _solve_independent(rhs, collision, initial, args=None, *, field, species, gc, - birth_speed, speed_floor, collision_tolerance, field_view=None, physics=None, - mass=M,charge=Q,spatial_diffusion=False,spatial_geometry=None,interpret=False,**options): - from .fields import BiotSavart,Vmec - for name in ('max_steps', 'max_orbit_steps', 'max_pending'): - value = options[name] - if isinstance(value, (bool, np.bool_)) or not isinstance(value, (int, np.integer)) or value < 1: - raise ValueError(name+" must be a positive integer") - nstate = initial.size - screening=getattr(species,'screening_species',None) - fast=field_view is not None and species.radial_grid is None and (screening is None or screening.radial_grid is None) - if spatial_diffusion and field_view is not None and not isinstance(field,BiotSavart):fast=False - if physics is None or fast: - arrays, factory = _prepare_split_field(field, species, gc=gc, - collision_tolerance=collision_tolerance, speed_floor=speed_floor, - event=options['event'], state_size=nstate, field_view=field_view, - mass=mass,charge=charge,spatial_diffusion=spatial_diffusion) - if physics is not None: - predictor_arrays,predictor_factory=prepare_physics(field,field_view,physics,initial,options['noise_dim'],predictor_only=True) - nfield=len(field_view[0]);nphysics=len(arrays);original_factory=factory - arrays=tuple(arrays)+tuple(predictor_arrays[nfield:]) - def factory(refs,y0): - physical=original_factory(refs[:nphysics],y0) - prediction=predictor_factory(refs[:nfield]+refs[nphysics:],y0)[4] - return (*physical,prediction) - else: - arrays,factory=prepare_physics(field,field_view,physics,initial,options['noise_dim'],spatial_geometry=spatial_geometry) - def forward(initial, parameters=None, *, record=False): - solve = _prepare_split_kernel(arrays, factory, options['ts'], adaptive=options['adaptive'], - dt0=options['dt0'], dtmax=options['dtmax'], speed=birth_speed, - tolerance=(options['rtol']+options['atol'])/2, noise_dim=options['noise_dim'], - max_steps=options['max_steps'], max_orbit_steps=options['max_orbit_steps'], - max_pending=options['max_pending'], state_size=nstate, - bank_components=4 if gc and nstate == 5 else 8, - orbit_dt0=options['orbit_dt0'], final_time=options['t1'],align_orbit=options.get('align_orbit',False), - warps=4 if isinstance(field,Vmec) and not spatial_diffusion else 1, - record=record,adjoint_checkpoint_steps=options.get('adjoint_checkpoint_steps',16), - adjoint_orbit_checkpoint_steps=options.get('adjoint_orbit_checkpoint_steps',16), - adjoint_max_accepted_steps=options.get('adjoint_max_accepted_steps'), - adjoint_max_orbit_steps=options.get('adjoint_max_orbit_steps'), - collision_indices=options.get('_collision_state_indices'),interpret=interpret) - result = solve(jnp.pad(initial, (0, 8-nstate)), jax.random.key_data(options['key'])) - if record: - result,tape=result - tape=tape._replace(states=tape.states[:,:nstate],orbit_states=tape.orbit_states[:,:nstate], - orbit_derivatives=tape.orbit_derivatives[:,:nstate], - orbit_endpoints=tape.orbit_endpoints[:,:nstate] if options.get('_collision_state_indices') is None else tape.orbit_endpoints) - ys,y,t,stats,status=result - solution = SplitSolution(options['ts'], ys[:, :nstate], t, y[:nstate], status, - SplitStats(*(stats[i].astype(jnp.int32) for i in range(6)), stats[6], - stats[7].astype(jnp.int32), stats[8].astype(jnp.int32)), jnp.empty((0,), initial.dtype)) - return (solution,tape) if record else solution - if options.get('adjoint','none') == 'accepted': - from .native_ad import solve_accepted_split, solve_gpu_accepted - direct=(isinstance(field,BiotSavart) and fast and mass==M and charge==Q) - if options.get('_reverse_execution','auto') not in ('auto','shared','independent'): - raise ValueError("reverse_execution must be 'auto', 'shared' or 'independent'") - if options.get('_reverse_execution','auto')=='independent' and not direct: - raise ValueError('independent reverse currently requires direct alpha coils and uniform background') - if direct and options.get('_reverse_execution','auto')!='shared': - return solve_gpu_accepted(forward,initial,field,species,gc=gc,speed_floor=speed_floor, - options=options,arrays=arrays,spatial_diffusion=spatial_diffusion,interpret=interpret) - ad_options={k:v for k,v in options.items() if k not in ('adjoint','_record_ad','_reverse_execution')} - return solve_accepted_split(rhs,collision,initial,args,_forward_override=forward,**ad_options) - return forward(initial,record=options.get('_record_ad',False)) - - # Spatial geometry -def move_in_cartesian(state, displacement, position, *, implicit=False): +def move_in_cartesian(state, displacement, position): """Invert the Cartesian Euler kick in the existing regular position chart. Solving the nonlinear map retains its Ito coordinate drift automatically; @@ -1703,23 +649,8 @@ def advance(carry): scale=jnp.maximum(1.,jnp.maximum(jnp.linalg.norm(target), jnp.linalg.norm(matrix)*jnp.maximum(1.,jnp.linalg.norm(point)))) return point,jnp.linalg.norm(mapping(point)-target)/scale,iteration+1 - if not implicit: - point,error,_=jax.lax.while_loop(lambda c:((c[2]==0)|(c[1]>threshold))&(c[2]<8),advance, - (state[:3],jnp.linalg.norm(displacement),jnp.asarray(0))) - return state.at[:3].set(jnp.where(error<=threshold,point,jnp.full_like(point,jnp.nan))) - def solve_root(function, guess): - del function - return jax.lax.while_loop(lambda c:((c[2]==0)|(c[1]>threshold))&(c[2]<8),advance, - (guess,jnp.linalg.norm(displacement),jnp.asarray(0)))[0] - def tangent_solve(linear, value): - a,b,c=jax.vmap(linear)(jnp.eye(3,dtype=state.dtype)) - rows=jnp.stack((jnp.cross(b,c),jnp.cross(c,a),jnp.cross(a,b))) - return (rows@value)/jnp.dot(a,rows[0]) - point=jax.lax.custom_root(lambda q:mapping(q)-target,state[:3],solve_root,tangent_solve) - matrix=jax.jacfwd(mapping)(point) - scale=jnp.maximum(1.,jnp.maximum(jnp.linalg.norm(target), - jnp.linalg.norm(matrix)*jnp.maximum(1.,jnp.linalg.norm(point)))) - error=jnp.linalg.norm(mapping(point)-target)/scale + point,error,_=jax.lax.while_loop(lambda c:((c[2]==0)|(c[1]>threshold))&(c[2]<8),advance, + (state[:3],jnp.linalg.norm(displacement),jnp.asarray(0))) return state.at[:3].set(jnp.where(error<=threshold,point,jnp.full_like(point,jnp.nan))) class BoozerGeometry(NamedTuple): @@ -2096,7 +1027,7 @@ def _particle_budget(velocity,unit_b,unit_gradient,rotation,electric,rates, def native_step_limit(field,bath,mass,charge,initial,rhs,public,position,magnitude, *,gc,coordinate=False,boozer=False, - electric_field,tmax,dtmax,tolerance=.005,reference_time=.005,orbit=True,reference_magnitude=None): + electric_field,tmax,dtmax,tolerance=.005,reference_time=.005,orbit=True): """Construct the default physical moment controller without benchmark data. Reference units use birth density-weighted temperature and the friction @@ -2113,7 +1044,7 @@ def native_step_limit(field,bath,mass,charge,initial,rhs,public,position,magnitu birth=public(initial);point=birth[:3] density=jnp.stack([bath.get_density(i,point) for i in range(bath.number_species)]) temperatures=jnp.stack([bath.get_temperature(i,point) for i in range(bath.number_species)]) - initial_magnitude=magnitude if reference_magnitude is None else reference_magnitude + initial_magnitude=magnitude speed=jnp.sqrt(birth[3]**2+2*birth[4]*initial_magnitude(point)/mass) if gc else jnp.linalg.norm(birth[3:]) temperature=jnp.sum(density*temperatures)/jnp.where(jnp.sum(density)>0,jnp.sum(density),1.) sigma=jnp.where(jnp.sum(density)>0,jnp.sqrt(ELEMENTARY_CHARGE*temperature/mass),speed/jnp.sqrt(10.)) @@ -2168,472 +1099,3 @@ def limit(t,y,args): electric if orbit else jnp.zeros(3),values,first,second,profile,jnp.maximum(reference_time,t*nu),orbit=orbit) return interval_from_budget(linear,quadratic,tolerance,cap=cap)/nu return limit - - -# GPU predictor lowering - -class KernelCalls: - """Keep native field reductions intact while scalarizing moment algebra.""" - def __init__(self):self.handlers={} - - def first_jet(self,function,kernel_function): - """Share one exact 3D value/Jacobian evaluation across JVP directions.""" - def ordinary(*point): - jacobian=jax.jacfwd(lambda x:jnp.asarray(function(*x)))(jnp.asarray(point)) - return (*function(*point),*jacobian.reshape(-1)) - packed=self.add(ordinary,kernel_function,12) - @jax.custom_jvp - def value(*point):return packed(*point)[:3] - @value.defjvp - def directional(primals,tangents): - data=packed(*primals) - return data[:3],tuple(sum(data[3+3*i+j]*tangents[j] for j in range(3)) for i in range(3)) - return value - - def add(self, function, kernel_function, count,*,kernel_jvp=None): - from jax.extend.core import Primitive - from jax._src.interpreters import ad, batching - primitive=Primitive('essos_split_field_call') - primitive.multiple_results=True - primitive.def_impl(function) - primitive.def_abstract_eval(lambda *avals:tuple(jax.core.ShapedArray((),avals[0].dtype) for _ in range(count))) - self.handlers[primitive]=kernel_function - derivatives={} - def jvp(primals,tangents): - active=tuple(i for i,t in enumerate(tangents) if not isinstance(t,ad.Zero)) - result=primitive.bind(*primals) - if not active:return result,tuple(ad.Zero.from_primal_value(x) for x in result) - if active not in derivatives: - size=len(primals) - def directional(fn): - def evaluate(*args): - def restricted(*values): - primals=list(args[:size]) - for i,v in zip(active,values):primals[i]=v - return fn(*primals) - return jax.jvp(restricted,tuple(args[i] for i in active),args[size:])[1] - return evaluate - def kernel_direction(refs,*args): - if kernel_jvp is not None: - directions=dict(zip(active,args[size:])) - return kernel_jvp(refs,args[:size],tuple(directions.get(i,jnp.zeros_like(args[i])) for i in range(size))) - graph=_flatten(directional(lambda *x:kernel_function(refs,*x)),*args) - return _evaluate_kernel(graph.jaxpr,graph.consts,*args) - derivatives[active]=self.add(directional(function),kernel_direction,count) - return result,derivatives[active](*primals,*(tangents[i] for i in active)) - ad.primitive_jvps[primitive]=jvp - def batch(values,axes): - size=next(x.shape[axis] for x,axis in zip(values,axes) if axis is not None) - rows=[primitive.bind(*(x if axis is None else jnp.take(x,i,axis=axis) for x,axis in zip(values,axes))) for i in range(size)] - return tuple(jnp.stack(column) for column in zip(*rows)),(0,)*count - batching.primitive_batchers[primitive]=batch - return lambda *args:tuple(primitive.bind(*args)) - -def _vmec_position_tangent(refs,point,direction,tables): - """Exact first derivative already returned by the native Fourier series.""" - from .rk import _vmec_series - R,dR=_vmec_series(refs,point,tables['rmnc']) - _,dZ=_vmec_series(refs,point,tables['zmns']) - radial=sum(a*b for a,b in zip(dR,direction));vertical=sum(a*b for a,b in zip(dZ,direction)) - c,s=jnp.cos(point[2]),jnp.sin(point[2]) - return radial*c-R*s*direction[2],radial*s+R*c*direction[2],vertical - -def scalar_graph(graph, constants, *arguments, calls=None, refs=(),memo=None,defer=False): - memo={} if memo is None else memo - def word_key(value): - if isinstance(value,(jax.core.Tracer,jax.Array)): - memo[('retain',id(value))]=value - return ('value',id(value)) - value=np.asarray(value) - return ('literal',value.dtype.str,value.shape,value.tobytes()) - def logical(value, shape): - if isinstance(value, tuple): - out=np.empty((len(value),),object) - for i,v in enumerate(value):out[i]=v - return out.reshape(shape) - if isinstance(value,(np.ndarray,np.generic,int,float,bool)): - return np.asarray(value).reshape(shape) - out=np.empty(shape,object) - for index in np.ndindex(shape): - item=value - for axis,i in enumerate(index):item=_select_one(item,i,axis) - out[index]=item if not item.shape else item.reshape(()) - return out - - env={v:logical(x,v.aval.shape) for v,x in zip(graph.constvars,constants)} - env.update({v:logical(x,v.aval.shape) for v,x in zip(graph.invars,arguments)}) - read=lambda v:np.asarray(v.val,dtype=v.aval.dtype) if isinstance(v,Literal) else env[v] - def elements(function,*values): - return np.frompyfunc(function,len(values),1)(*values) - def scalar(value): - while isinstance(value,np.ndarray):value=value.reshape(()).item() - return value - def device_scalar(value,dtype): - while isinstance(value,np.ndarray) and not value.shape:value=value.item() - return jnp.asarray(value,dtype=dtype) - def reduced(value,axes,combine): - result=value - for axis in sorted(axes,reverse=True): - result=np.frompyfunc(combine,2,1).reduce(result,axis=axis) - return result - - for equation in graph.eqns: - values=[read(v) for v in equation.invars];p=equation.params;name=equation.primitive.name - folded={'add':np.add,'sub':np.subtract,'mul':np.multiply,'div':np.divide, - 'neg':np.negative,'abs':np.abs,'sqrt':np.sqrt,'eq':np.equal,'ne':np.not_equal, - 'lt':np.less,'le':np.less_equal,'gt':np.greater,'ge':np.greater_equal, - 'max':np.maximum,'min':np.minimum,'and':np.bitwise_and,'or':np.bitwise_or} - static=all(np.asarray(v).dtype!=object for v in values) - if calls is not None and equation.primitive in calls.handlers: - words=tuple(scalar(v) for v in values) - key=('field',equation.primitive,tuple(word_key(v) for v in words)) - if key not in memo: - arguments=tuple(device_scalar(v,var.aval.dtype) for v,var in zip(words,equation.invars)) - if defer: - memo[key]=equation.primitive.bind(*arguments,**equation.params) - else: - memo[key]=calls.handlers[equation.primitive](refs,*arguments) - result=memo[key] - elif static and name in folded:result=[folded[name](*values)] - elif static and name=='is_finite':result=[np.isfinite(values[0])] - elif static and name=='convert_element_type':result=[values[0].astype(p['new_dtype'])] - elif static and name=='integer_pow':result=[np.power(values[0],p['y'])] - elif static and name=='select_n':result=[np.choose(values[0],values[1:])] - elif name=='platform_index': - backend='cuda' if jax.default_backend()=='gpu' else jax.default_backend() - default=next((i for i,group in enumerate(p['platforms']) if group is None),0) - result=[np.asarray(next((i for i,group in enumerate(p['platforms']) if group is not None and backend in group),default),np.int32)] - elif (name=='call_hi_primitive' and type(p['_prim']).__name__=='SearchSorted' - and np.ndim(values[0])==1 and not np.ndim(values[1])): - # JAX >= 0.12 keeps searchsorted (e.g. inside jnp.interp) as one primitive. - right,query,count='side=right' in repr(p['_prim']),scalar(values[1]),np.empty((),object) - count[()]=sum(jnp.asarray(x<=query if right else xbest) - better=better|(jnp.isnan(item)&~jnp.isnan(best)) - chosen=jnp.where(better,jnp.asarray(i,p['index_dtype']),chosen);best=jnp.where(better,item,best) - out[index]=chosen - result=[out] - elif name=='dot_general': - (lc,rc),(lb,rb)=p['dimension_numbers'];lhs,rhs=values - lf=tuple(i for i in range(lhs.ndim) if i not in lc+lb) - rf=tuple(i for i in range(rhs.ndim) if i not in rc+rb) - batch=tuple(lhs.shape[i] for i in lb) - left=tuple(lhs.shape[i] for i in lf);right=tuple(rhs.shape[i] for i in rf) - inner=int(np.prod([lhs.shape[i] for i in lc])) - lhs=np.transpose(lhs,lb+lf+lc).reshape(batch+(int(np.prod(left)),inner)) - rhs=np.transpose(rhs,rb+rc+rf).reshape(batch+(inner,int(np.prod(right)))) - result=[np.matmul(lhs,rhs).reshape(batch+left+right)] - elif name=='pad': - value=values[0];config=p['padding_config'] - shape=tuple(a+b+n+(n-1)*c for n,(a,b,c) in zip(value.shape,config)) - out=np.empty(shape,object);out.fill(scalar(values[1])) - out[tuple(slice(a,a+n*(c+1),c+1) for n,(a,b,c) in zip(value.shape,config))]=value - result=[out] - elif name.startswith('scatter'): - operand,indices,updates=values;dims=p['dimension_numbers'] - if indices.dtype==object: - raise ValueError('predictor lowering requires static scalar scatter indices') - out=np.array(operand,object,copy=True) - window_axes=[i for i in range(out.ndim) if i not in dims.inserted_window_dims] - batch_axes=[i for i in range(updates.ndim) if i not in dims.update_window_dims] - for index in np.ndindex(updates.shape): - batch=tuple(index[i] for i in batch_axes);target=[0]*out.ndim - for axis,item in zip(dims.scatter_dims_to_operand_dims,indices[batch]):target[axis]=int(item) - for axis,i in zip(window_axes,dims.update_window_dims):target[axis]+=index[i] - if any(i<0 or i>=n for i,n in zip(target,out.shape)):continue - target=tuple(target);new=updates[index] - out[target]=out[target]+new if name=='scatter-add' else new - result=[out] - elif name=='gather': - operand,indices=values;dims=p['dimension_numbers'] - if indices.dtype==object: - from itertools import product - out=np.empty(equation.outvars[0].aval.shape,object) - batch_axes=[i for i in range(out.ndim) if i not in dims.offset_dims] - window_axes=[i for i in range(operand.ndim) if i not in dims.collapsed_slice_dims] - for index in np.ndindex(out.shape): - batch=tuple(index[i] for i in batch_axes);selected=[0]*operand.ndim - for axis,item in zip(dims.start_index_map,indices[batch]):selected[axis]=jnp.clip(item,0,operand.shape[axis]-p['slice_sizes'][axis]) - for axis,i in zip(window_axes,dims.offset_dims):selected[axis]+=index[i] - choices=[range(n) if isinstance(i,jax.core.Tracer) else (int(i),) for i,n in zip(selected,operand.shape)] - value=operand[tuple(next(iter(c)) for c in choices)] - for source in product(*choices): - match=jnp.asarray(True) - for i,j in zip(selected,source):match &= i==j - value=jnp.where(match,operand[source],value) - out[index]=value - result=[out] - env.update((v,logical(x,v.aval.shape)) for v,x in zip(equation.outvars,result)) - continue - pieces=[] - for index in np.ndindex(indices.shape[:-1]): - starts=[0]*operand.ndim - for axis,item in zip(dims.start_index_map,indices[index]):starts[axis]=int(item) - block=operand[tuple(slice(a,a+n) for a,n in zip(starts,p['slice_sizes']))] - pieces.append(np.squeeze(block,axis=dims.collapsed_slice_dims)) - result=[np.asarray(pieces,dtype=object).reshape(equation.outvars[0].aval.shape)] - else: - parameters=dict(p) - if name=='convert_element_type':parameters['weak_type']=False - def bind(*xs): - if all(not isinstance(x,jax.core.Tracer) for x in xs): - if name in folded:return folded[name](*xs) - if name=='integer_pow':return np.power(xs[0],p['y']) - if name=='convert_element_type':return np.asarray(xs[0],dtype=p['new_dtype']) - if name=='select_n':return xs[1+int(xs[0])] - if name=='is_finite':return _kernel_is_finite(xs[0]) if isinstance(xs[0],jax.core.Tracer) else np.isfinite(xs[0]) - if name=='erf': - return _kernel_erf().bind(jnp.asarray(xs[0])) - if name=='integer_pow': - return _kernel_integer_power(device_scalar(xs[0],equation.invars[0].aval.dtype),p['y']) - if name=='add_any':return xs[0]+xs[1] - if name=='le_to':return jnp.less_equal(*xs)|jnp.isnan(xs[1]) - return _bind_primitive(equation.primitive,[device_scalar(x,v.aval.dtype) for x,v in zip(xs,equation.invars)],parameters) - signature=None - try: - signature=('scalar',equation.primitive,tuple(parameters.items()), - tuple((v.aval.dtype.str,v.aval.weak_type) for v in equation.invars)) - hash(signature) - except TypeError:signature=None - if equation.effects:signature=None - def cached(*xs): - if signature is None:return bind(*xs) - key=signature+tuple(word_key(v) for v in xs) - if key not in memo:memo[key]=bind(*xs) - return memo[key] - try:result=[elements(cached,*values)] - except (ValueError,TypeError,NotImplementedError) as error: - raise ValueError(f"scalar predictor operation {name}, input shapes {[v.aval.shape for v in equation.invars]}: {error}") from error - env.update((v,logical(x,v.aval.shape)) for v,x in zip(equation.outvars,result)) - return tuple(jnp.asarray(scalar(read(v)),dtype=v.aval.dtype) if not v.aval.shape else read(v) for v in graph.outvars) - -def _scalar_predictor(graph,calls): - """Simplify scalar moment algebra before expanding native field reductions.""" - variables=(*graph.jaxpr.constvars,*graph.jaxpr.invars) - def evaluate(*words): - values=[];offset=0 - for variable in variables: - size=variable.aval.size - values.append(tuple(words[offset:offset+size]));offset+=size - count=len(graph.jaxpr.constvars) - return scalar_graph(graph.jaxpr,values[:count],*values[count:],calls=calls,defer=True) - examples=tuple(jnp.zeros((),v.aval.dtype) for v in variables for _ in range(v.aval.size)) - return _flatten(evaluate,*examples) - -def prepare_physics(field, field_view, build, initial, noise_dim,*,predictor_only=False,spatial_geometry=None): - """Compile the production orbit, kick and predictor into one GPU factory. - - Only field evaluation has a private adapter. Every collision map, physical - normalization and moment derivative is the same function as shared JAX. - """ - from .rk import _coil_field_values, _dot, _cross, _vmec_quantities, _vmec_series, _KernelField - from .dynamics import _gc_quantities - from .fields import BiotSavart, Vmec - from .boozer import BoozerField - calls=KernelCalls();coil=isinstance(field,BiotSavart);vmec=isinstance(field,Vmec) - boozer=isinstance(field,BoozerField) - arrays,metadata=field_view if field_view is not None else ((),None) - nsource=len(arrays) - extra=[] - - def flatten_values(values): - return tuple(v for item in values for v in (tuple(item) if isinstance(item,(tuple,list)) or getattr(item,'ndim',0) else (item,))) - def ordinary(fn): - graph=_flatten(fn,*(jnp.zeros((),jnp.float64) for _ in range(3))) - start=len(arrays)+len(extra);extra.extend(graph.consts) - def evaluate(refs,*args): - constants=[tuple(ref[index] for index in np.ndindex(v.shape)) if v.ndim else ref[()] - for v,ref in zip(graph.consts,refs[start:start+len(graph.consts)])] - return scalar_graph(graph.jaxpr,constants,*args) - return evaluate - def geometric(*point):return flatten_values(_gc_quantities(field,jnp.asarray(point))) - def fast_geometry(refs,*point): - if vmec:return flatten_values(_vmec_quantities(refs,point,metadata)) - B,J=_coil_field_values(refs,point,*metadata,True) - magnitude=jnp.sqrt(_dot(B,B)) - gradient=tuple(sum(J[i][j]*B[i]/magnitude for i in range(3)) for j in range(3)) - curl=(J[2][1]-J[1][2],J[0][2]-J[2][0],J[1][0]-J[0][1]) - correction=_cross(B,gradient) - curl=tuple(curl[i]/magnitude+correction[i]/magnitude**2 for i in range(3)) - curvature=tuple(-v/magnitude for v in _cross(B,curl)) - return (*B,*B,magnitude,*gradient,*curl,*curvature,jnp.asarray(1.,magnitude.dtype)) - geometry=None if boozer else calls.add(geometric,fast_geometry if coil or vmec else ordinary(geometric),17) - - def absb(*point):return (field.modB(*point) if boozer else field.AbsB(jnp.asarray(point)),) - def fast_absb(refs,*point): - if coil: - B,_=_coil_field_values(refs,point,*metadata,False) - return (jnp.sqrt(_dot(B,B)),) - if vmec:return (_vmec_series(refs,point,metadata['bmnc'])[0],) - model=_KernelField(refs[:nsource],float(field.psi0)) - graph=_flatten(lambda r,theta,z:model.modB_derivatives(r,theta,z)[0],jnp.sqrt(point[0]),point[1],point[2]) - return _evaluate_kernel(graph.jaxpr,graph.consts,jnp.sqrt(point[0]),point[1],point[2]) - magnitude=calls.add(absb,fast_absb if coil or vmec or boozer else ordinary(absb),1) - if not boozer: - vector=lambda *point:tuple(field.B_contravariant(jnp.asarray(point))) - fast_vector=(lambda refs,*point:_coil_field_values(refs,point,*metadata,False)[0]) if coil else ( - lambda refs,*point:_vmec_quantities(refs,point,metadata)[1]) if vmec else ordinary(vector) - magnetic_vector=calls.add(vector,fast_vector,3) - - class View: - def gc_quantities(self,point): - v=geometry(*point) - return jnp.asarray(v[:3]),jnp.asarray(v[3:6]),v[6],jnp.asarray(v[7:10]),jnp.asarray(v[10:13]),jnp.asarray(v[13:16]),v[16] - def AbsB(self,point):return magnitude(*point)[0] - def B_contravariant(self,point):return jnp.asarray(magnetic_vector(*point)) - def B_covariant(self,point):return self.gc_quantities(point)[0] - view=View() - if vmec: - xyz=lambda s,theta,z:tuple(field.to_xyz(jnp.array([s,theta,z]))) - def fast_xyz(refs,s,theta,z): - R=_vmec_series(refs,(s,theta,z),metadata['rmnc'])[0] - Z=_vmec_series(refs,(s,theta,z),metadata['zmns'])[0] - return R*jnp.cos(z),R*jnp.sin(z),Z - position=calls.add(xyz,fast_xyz,3,kernel_jvp=lambda refs,p,d:_vmec_position_tangent(refs,p,d,metadata)) - view.to_xyz=lambda point:jnp.asarray(position(*point)) - if boozer: - def modb(r,theta,z):return tuple(field.modB_derivatives(r,theta,z)) - def profiles(s):return flatten_values(field.profiles(s)) - def boozer_fast(fn): - def evaluate(refs,*args): - model=_KernelField(refs[:nsource],float(field.psi0)) - graph=_flatten(lambda *x:fn(model,*x),*args) - return _evaluate_kernel(graph.jaxpr,graph.consts,*args) - return evaluate - mod=calls.add(modb,boozer_fast(lambda model,*x:tuple(model.modB_derivatives(*x))),4) - prof=calls.add(profiles,boozer_fast(lambda model,s:flatten_values(model.profiles(s))),6) - view.modB_derivatives=lambda *point:mod(*point) - view.modB=lambda *point:magnitude(*point)[0] - view.profiles=lambda s:(prof(s)[:3],prof(s)[3:]) - view.psi0=field.psi0 - if spatial_geometry is not None: - xyz=lambda *q:tuple(spatial_geometry(jnp.asarray(q))) - if hasattr(spatial_geometry,'_kernel_view'): - data,evaluate=spatial_geometry._kernel_view();start=nsource+len(extra);extra.extend(data) - fast_xyz=lambda refs,*q:evaluate(refs[start:start+len(data)],*q) - else:fast_xyz=ordinary(xyz) - position=calls.add(xyz,fast_xyz,3) - view.to_xyz=lambda point:jnp.asarray(position(*point)) - nfield=len(arrays)+len(extra) - arrays=tuple(arrays)+tuple(extra) - rhs,kick,norm,level,limit=build(view) - nstate=initial.size - def kicked(y,h,dw): - result=kick(0.,y,h,dw,None) - return (*result.y,result.drift_error,result.diffusion_error) - graphs=([] if predictor_only else [ - _flatten(lambda y:tuple(rhs(0.,y,None)),initial), - _flatten(kicked,initial,jnp.asarray(1e-9),jnp.zeros(noise_dim)), - _flatten(lambda y,e:norm(y,y,e),initial,jnp.zeros_like(initial)), - _flatten(lambda y:level(0.,y,None),initial)]) - if coil and nstate==6: - def vector_jet(refs,*point): - B,J=_coil_field_values(refs,point,*metadata,True) - return (*B,*(x for row in J for x in row)) - magnetic_vector=calls.first_jet(vector,vector_jet) - graphs.append(_flatten(lambda t,y:limit(t,y,None),jnp.asarray(0.),initial)) - prediction=_scalar_predictor(graphs[-1],calls) - all_arrays=tuple(arrays)+tuple(v for graph in graphs for v in graph.consts) - def factory(refs,y0): - offset=nfield;constants=[] - for graph in graphs: - values=[] - for v,ref in zip(graph.consts,refs[offset:offset+len(graph.consts)]): - values.append(tuple(ref[index] for index in np.ndindex(v.shape)) if v.ndim else ref[()]) - constants.append(values);offset+=len(graph.consts) - def evaluate(i,*args): - if i==len(graphs)-1: - words=tuple(word for value in (*constants[i],*args) - for word in (value if isinstance(value,tuple) else (value,))) - return scalar_graph(prediction.jaxpr,prediction.consts,*words,calls=calls,refs=refs[:nfield]) - return scalar_graph(graphs[i].jaxpr,constants[i],*args,calls=calls,refs=refs[:nfield]) - state=lambda y:tuple(scalar(y,i) for i in range(nstate)) - if predictor_only: - return None,None,None,None,lambda t,y:evaluate(0,t,state(y))[0] - def padded(values):return pack(tuple(values)+(scalar(y0,7),)*(8-len(values))) - def rhs(y):return padded(evaluate(0,state(y))) - def kick(y,h,dw): - result=evaluate(1,state(y),h,tuple(scalar(dw,i) for i in range(noise_dim))) - return padded(result[:nstate]),result[-2],result[-1] - def norm(y,e,speed,budget):return evaluate(2,state(y),state(e))[0] - return rhs,kick,norm,lambda y:evaluate(3,state(y))[0],lambda t,y:evaluate(4,t,state(y))[0] - return all_arrays,factory diff --git a/examples/benchmarks/benchmark_native.py b/examples/benchmarks/benchmark_native.py new file mode 100644 index 00000000..e1aa06b9 --- /dev/null +++ b/examples/benchmarks/benchmark_native.py @@ -0,0 +1,517 @@ +"""Timing and validation benchmarks for the native ESSOS orbit and collision solvers. + +Suites (``--suite``) + orbit Collisionless native orbits: ``essos.rk.prepare_solver`` (coils, VMEC) or + ``essos.rk.trace_boozer_orbits`` (Boozer), shared JAX or the fused CUDA kernel. + collisions Collision maps and Brownian driver against independent NumPy/SciPy + finite-volume Fokker-Planck references. ``--study thermal`` relaxes a + monoenergetic beam in a fixed Maxwellian proton bath; ``--study passage`` + measures the first passage from 5 T to 2 T. ``--law`` selects the fixed + Coulomb-logarithm control (exact Maxwell equilibrium) or the production + ``impact`` law. + +Every timed case reports compile time, first execution, warm median, particles per second, +compiled buffer size, peak memory (device allocator peak on a GPU, process RSS on a CPU) and +the lost/failed counts. Peak memory is cumulative within a process, so compare one case per +process. Setup is excluded from the timings. ``--out DIR`` writes ``result.json`` and, with +``--save-states``, ``states.npz``. + +Execution: ``jax`` (or ``shared``) is ordinary JAX, ``cuda`` (or ``independent``) is the +persistent GPU kernel and ``both`` times each in turn. The default is ``cuda`` on a GPU and +``jax`` otherwise. + +CPU smoke test:: + + python -m examples.benchmarks.benchmark_native --suite orbit --field coils -N 4 \\ + --execution jax --stepping fixed --steps 7 --horizon 2e-8 --repeats 1 +""" +import argparse +import json +import resource +import sys +from pathlib import Path +from time import perf_counter + +import jax +import jax.numpy as jnp +import numpy as np +from scipy.linalg import eigh_tridiagonal +from scipy.special import gammainc +from scipy.stats import kstest, maxwell, norm, t as student_t + +from essos.background_species import BackgroundSpecies, chandrasekhar, d_chandrasekhar +from essos.boozer import BoozerField, psi0_from_vmec +from essos.coils import Coils +from essos.constants import (ALPHA_PARTICLE_CHARGE as Q, ALPHA_PARTICLE_MASS as M, ELEMENTARY_CHARGE, + EPSILON_0, HBAR, ONE_EV, PROTON_MASS) +from essos.dynamics import Particles, _to_axis_regular +from essos.fields import BiotSavart, Vmec +from essos.rk import RKStatus, prepare_solver, trace_boozer_orbits +from essos.split import (CollisionCoefficients, collision_coefficients, guiding_center_collision_step, + particle_collision_step, solve_split) + +ROOT = Path(__file__).resolve().parents[2] +COILS = ROOT / 'examples/input_files/ESSOS_biot_savart_LandremanPaulQA.json' +WOUT = ROOT / 'examples/input_files/wout_LandremanPaul2021_QA_reactorScale_lowres.nc' +BATH_DENSITY, BATH_TEMPERATURE_EV = 1e20, 1000. +EXECUTIONS = {'jax': 'jax', 'shared': 'jax', 'cuda': 'cuda', 'independent': 'cuda'} +GPU_FORWARD = ({'compiler_options': {'xla_gpu_enable_command_buffer': 'FUSION,WHILE,CONDITIONAL', + 'xla_gpu_graph_min_graph_size': 1}} + if jax.default_backend() == 'gpu' else {}) + + +# --------------------------------------------------------------------------- measurement + +def measure(function, *args, repeats=3, **jit_options): + """Compile ``function`` ahead of time, then time blocked executions. + + Returns the last output and a dict with the compile time, the first execution, the + warm median (all in seconds), the compiler's buffer estimate (arguments + outputs + + temporaries - aliases, excluding executable constants) and the peak memory in bytes. + """ + start = perf_counter() + executable = jax.jit(function, **jit_options).lower(*args).compile() + compile_s = perf_counter() - start + start = perf_counter() + out = jax.block_until_ready(executable(*args)) + first_s = perf_counter() - start + samples = [] + for _ in range(repeats): + start = perf_counter() + out = jax.block_until_ready(executable(*args)) + samples.append(perf_counter() - start) + memory = executable.memory_analysis() + buffers = None if memory is None else int(memory.argument_size_in_bytes + memory.output_size_in_bytes + + memory.temp_size_in_bytes - memory.alias_size_in_bytes) + rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * (1 if sys.platform == 'darwin' else 1024) + peak = (jax.devices()[0].memory_stats() or {}).get('peak_bytes_in_use', rss) + return out, dict(compile_s=compile_s, first_s=first_s, warm_s=float(np.median(samples)), + buffers_bytes=buffers, peak_bytes=int(peak)) + + +def report_row(name, particles, timing, status, **extra): + """One result row: timing, throughput and lost/failed counts (RKStatus codes).""" + status = np.asarray(status) + row = dict(name=name, particles=particles, particles_per_s=particles / timing['warm_s'], **timing, + lost=int(np.sum(status == RKStatus.EVENT)), failed=int(np.sum(status > RKStatus.EVENT)), **extra) + peak = row['peak_bytes'] / 2**20 + print(f"{name}: compile {row['compile_s']:.2f} s, first {row['first_s']:.3f} s, warm " + f"{1e3 * row['warm_s']:.2f} ms, {row['particles_per_s']:.4g} particles/s, peak {peak:.0f} MiB, " + f"lost {row['lost']}, failed {row['failed']}", flush=True) + return row + + +# --------------------------------------------------------------------------- fields and births + +def boozer_field(options): + """Boozer field from the wout file (needs ``essos[booz]``).""" + from booz_xform_jax import Booz_xform + from netCDF4 import Dataset + transform = Booz_xform(verbose=0, mboz=options.boozer_modes, nboz=options.boozer_modes) + transform.read_wout(str(options.wout), flux=False) + transform.run() + with Dataset(options.wout) as data: + psi = psi0_from_vmec(data['phi'][-1]) + field = BoozerField.from_booz(transform.s_b, transform.bmnc_b, transform.xm_b, transform.xn_b, transform.iota, + transform.Boozer_G, transform.Boozer_I, psi, transform.nfp, mode_tolerance=1e-4) + return field + + +class Case: + """Field, birth points and pitches for the orbit suite. + + Coil births lie on the outboard midplane; VMEC and Boozer births are at s=0.3 with random + angles. + """ + + def __init__(self, options): + self.options = options + n = options.particles + rng = np.random.default_rng(options.seed) + self.speed = float(np.sqrt(2 * options.energy_ev * ONE_EV / M)) + if options.field == 'coils': + self.field = BiotSavart(Coils.from_json(str(COILS))) + self.points = jnp.column_stack((jnp.linspace(1.23, 1.27, n), jnp.zeros((n, 2)))) + self.pitch = jnp.linspace(.35, .65, n) + return + if options.field == 'vmec': + self.field = Vmec(str(options.wout), ntheta=8, nphi=8) + else: + self.field = boozer_field(options) + self.points = jnp.asarray(np.column_stack((np.full(n, .3), rng.uniform(0, 2 * np.pi, n), + rng.uniform(0, 2 * np.pi / self.field.nfp, n)))) + self.pitch = jnp.asarray(rng.uniform(-.7, .7, n)) + + def particles(self): + """Coil-field births for the collisionless solvers.""" + options = self.options + return Particles(self.points, mass=M, charge=Q, energy=options.energy_ev * ONE_EV, + initial_vparallel_over_v=self.pitch, + field=self.field if options.model == 'lorentz' else None, phase_angle_full_orbit=jnp.pi) + + def orbit_state(self): + """Initial states and magnetic moments (Boozer only) for the collisionless solvers.""" + options = self.options + mu = jnp.zeros(options.particles) + if options.field == 'coils': + birth = self.particles() + if options.model == 'gc': + return jnp.column_stack((self.points, birth.initial_vparallel)), mu + return jnp.column_stack((birth.initial_xyz_fullorbit, birth.initial_vxvyvz)), mu + state = jnp.column_stack((self.points, self.speed * self.pitch)) + if options.field == 'vmec': + return jax.vmap(_to_axis_regular)(state), mu + s, theta, zeta, vpar = state.T + modb = jax.vmap(self.field.modB)(s, theta, zeta) + return (jnp.column_stack((jnp.sqrt(s) * jnp.cos(theta), jnp.sqrt(s) * jnp.sin(theta), zeta, vpar)), + self.speed**2 * (1 - self.pitch**2) / (2 * modb)) + + +def schedule(options): + """Save times and the orbit initial step; the fixed step divides the horizon evenly.""" + times = np.linspace(0., options.horizon, options.saves) + adaptive = options.stepping == 'adaptive' + return times, adaptive, options.horizon / 4 if adaptive else options.horizon / options.steps + + +# --------------------------------------------------------------------------- orbit suite + +def orbit_suite(options, case, execution): + """Collisionless native orbit timing for one execution.""" + times, adaptive, dt = schedule(options) + y0, mu = case.orbit_state() + if options.field == 'boozer': + def run(y, m): + states, _, _, _, counts, status = trace_boozer_orbits( + case.field, y, m, M, Q, case.speed, times, dt, options.rtol, options.atol, options.method, + adaptive, 'i', .4, .3, options.max_steps, execution=execution) + return states, status, counts + else: + solve = prepare_solver(case.field, mass=M, charge=Q, energy=options.energy_ev * ONE_EV, times=times, + dt0=dt, model=options.model, method=options.method, adaptive=adaptive, + rtol=options.rtol, atol=options.atol, max_steps=options.max_steps, + execution=execution) + + def run(y, m): + out = jax.vmap(solve)(y) + return out.ys, out.status, jnp.stack(out.stats, axis=-1) + (states, status, counts), timing = measure(run, y0, mu, repeats=options.repeats, **GPU_FORWARD) + row = report_row(f'orbit/{execution}', options.particles, timing, status, + accepted=int(np.sum(counts[:, 1])), rejected=int(np.sum(counts[:, 2]))) + return [row], {row['name']: np.asarray(states)} + + +# --------------------------------------------------------------------------- collision references + +class Reference: + """Independent reversible finite-volume backward Fokker-Planck generator. + + Speed ``r = v / sqrt(T/m)`` and time ``t * nu_s(sqrt(T/m))`` are dimensionless. The + Maxwell weight is exact in each cell, and the symmetrized tridiagonal generator is + diagonalized once, so moments and survival probabilities need no time stepping or SDE + solver. Legendre sectors ``ell`` carry pitch-angle scattering. With ``absorbing`` the + lower speed is absorbing, otherwise both ends reflect. ``law(r)`` returns the + dimensionless (parallel, perpendicular) diffusion. + """ + + def __init__(self, law, *, lower=0., upper=8., nodes=768, start=np.sqrt(10.), pitch=.6, absorbing=False): + edges = np.linspace(lower, upper, nodes + 1) + dr = edges[1] - edges[0] + self.grid, self.start, self.pitch = (edges[1:] + edges[:-1]) / 2, start, pitch + mass = np.where(edges[:-1] > 4., -np.diff(maxwell.sf(edges)), np.diff(maxwell.cdf(edges))) + conductance = np.zeros(nodes + 1) + conductance[1:-1] = law(edges[1:-1])[0] * maxwell.pdf(edges[1:-1]) / dr + if absorbing: + conductance[0] = 2 * law(edges[:1])[0][0] * maxwell.pdf(edges[0]) / dr + self.absorbing, self.root = absorbing, np.sqrt(mass) + self.diagonal = -(conductance[:-1] + conductance[1:]) / mass + self.off = conductance[1:-1] / np.sqrt(mass[:-1] * mass[1:]) + self.pitch_rate = 2 * law(self.grid)[1] / self.grid**2 + i = np.searchsorted(self.grid, start) - 1 + self.initial = np.zeros(nodes) + self.initial[i:i + 2] = 1 - (start - self.grid[i]) / dr, (start - self.grid[i]) / dr + self.spectra = {} + + def spectrum(self, ell): + if ell not in self.spectra: + values, vectors = eigh_tridiagonal(self.diagonal - ell * (ell + 1) * self.pitch_rate / 2, self.off) + if ell == 0 and not self.absorbing: + if abs(values[-1]) > 2e-8: + raise RuntimeError('reflecting generator has no stationary mode') + values[-1] = 0. + self.spectra[ell] = values, vectors, (self.initial / self.root) @ vectors + return self.spectra[ell] + + def amplitude(self, observable, ell=0): + values, vectors, initial = self.spectrum(ell) + return values, initial * (vectors.T @ (self.root * observable)) + + def expect(self, times, observable, ell=0): + values, amplitude = self.amplitude(observable, ell) + return np.exp(np.asarray(times, float)[..., None] * values) @ amplitude + + def moments(self, times): + """Speed and pitch means and variances; the monoenergetic start is exact at t=0.""" + t, r = np.asarray(times, float), self.grid + m1, m2 = self.expect(t, r), self.expect(t, r * r) + p2 = (3 * self.pitch**2 - 1) / 2 + xi = self.pitch * self.expect(t, 1., 1) + xi2 = (1 + 2 * p2 * self.expect(t, 1., 2)) / 3 + result = dict(speed_mean=m1, speed_variance=m2 - m1**2, pitch_mean=xi, pitch_variance=xi2 - xi**2) + initial = dict(speed_mean=self.start, speed_variance=0., pitch_mean=self.pitch, pitch_variance=0.) + return {name: np.where(t == 0, initial[name], value) for name, value in result.items()} + + def passage(self): + """First-passage CDF, mean and variance of the speed to the absorbing boundary.""" + values, amplitude = self.amplitude(1.) + mean, second = -np.sum(amplitude / values), 2 * np.sum(amplitude / values**2) + return (lambda t: np.clip(1 - np.exp(np.asarray(t, float)[..., None] * values) @ amplitude, 0., 1.), + mean, second - mean**2) + + +def reference_law(law): + """Independent NumPy coefficients (Chandrasekhar function from the incomplete gamma). + + ``constant`` fixes the Coulomb logarithm; ``impact`` is the smooth impact-parameter law of + the production ``coulomb_log_model="impact"``. Both are normalized to D_parallel(1) = 1. + """ + sigma = np.sqrt(BATH_TEMPERATURE_EV * ELEMENTARY_CHARGE / PROTON_MASS) + debye = np.sqrt(EPSILON_0 * BATH_TEMPERATURE_EV / (BATH_DENSITY * ELEMENTARY_CHARGE)) + classical = ELEMENTARY_CHARGE**2 / (4 * np.pi * EPSILON_0 * PROTON_MASS / 2) + + def logarithm(r): + relative = sigma**2 * (r * r + 3) + return (np.log(debye) + np.log(relative) - .5 * np.log(classical**2 + (HBAR / PROTON_MASS)**2 * relative) + if law == 'impact' else np.ones_like(r)) + + def raw(r): + x = r / np.sqrt(2.) + g = gammainc(1.5, x * x) / (2 * x * x) + return logarithm(r) * g / r, .5 * logarithm(r) * (gammainc(.5, x * x) - g) / r + + scale = raw(np.ones(1))[0] + return lambda r: tuple(c / scale for c in raw(np.asarray(r, float))) + + +def native_law(law): + """Dimensionless coefficients of the production collision maps, for ``solve_split``.""" + if law == 'constant': + scale = chandrasekhar(jnp.asarray(1 / np.sqrt(2.))) + + def values(v): + x = v / jnp.sqrt(2.) + g = chandrasekhar(x) + dp = g / (v * scale) + return jnp.array([dp, (jax.scipy.special.erf(x) - g) / (2 * v * scale), + (d_chandrasekhar(x) / (jnp.sqrt(2.) * v) - g / v**2) / scale, -v * dp]) + + def coefficients(speed, full_orbit=False): + (dp, dt, ddp, q), slope = jax.jvp(values, (speed,), (jnp.ones_like(speed),)) + dk = (slope[3] + slope[2] + 2 * (slope[0] - slope[1]) / speed - 2 * (dp - dt) / speed**2 + if full_orbit else jnp.zeros_like(speed)) + return CollisionCoefficients(dp, dt, ddp, q, slope[3], dk) + return coefficients + species = BackgroundSpecies(1, jnp.array([1.]), jnp.array([1.]), jnp.array([BATH_DENSITY]), + jnp.array([BATH_TEMPERATURE_EV]), coulomb_log_model='impact') + sigma = float(np.sqrt(BATH_TEMPERATURE_EV * ELEMENTARY_CHARGE / PROTON_MASS)) + + def si(speed, full_orbit=False): + return collision_coefficients(species, PROTON_MASS, ELEMENTARY_CHARGE, speed, jnp.zeros(3), + full_orbit=full_orbit) + nu = float(-si(sigma).friction / sigma) + + def coefficients(r, full_orbit=False): + c = si(sigma * r, full_orbit) + return CollisionCoefficients(c.parallel / (sigma**2 * nu), c.perpendicular / (sigma**2 * nu), + c.parallel_derivative / (sigma * nu), c.friction / (sigma * nu), + c.friction_derivative / nu, c.particle_drift_derivative / nu) + return coefficients + + +def collision_solver(law, model, adaptive, times, horizon, threshold=None, event_interval=1.220703125e-5): + """Batched ``solve_split`` of the collision maps alone (zero orbit), as a function of + (initial states, keys, collision tolerance, step cap). + + GC states are (speed, pitch) with a reflecting speed floor; Lorentz states are + velocities. With ``threshold`` the run stops when the speed first crosses it, and the + step is limited near the boundary to ``event_interval`` (detected crossings are refined + by the driver; this is not an exact Brownian crossing sampler). + """ + gc = model == 'gc' + speed_of = (lambda y: y[0]) if gc else jnp.linalg.norm + + def batch(y0, keys, tolerance, cap): + def kick(t, y, h, dw, args): + c = law(speed_of(y), full_orbit=not gc) + if gc: + return guiding_center_collision_step(y[0], y[1], h, dw, c, speed_floor=.05 * np.sqrt(2.), + tolerance=tolerance) + return particle_collision_step(y, h, dw, c, tolerance=tolerance) + + def limit(t, y, args): + speed = speed_of(y) + c = law(speed) + inward = jnp.maximum(-(c.friction + c.parallel_derivative + 2 * c.parallel / speed), 0.) + distance = jnp.maximum(speed - threshold, 0.) + diffusive = jnp.where(c.parallel > 0, .2 * distance**2 / (2 * jnp.where(c.parallel > 0, c.parallel, 1.)), + jnp.inf) + advective = jnp.where(inward > 0, .2 * distance / jnp.where(inward > 0, inward, 1.), jnp.inf) + return jnp.maximum(event_interval, jnp.minimum(diffusive, advective)) + + def one(y, key): + return solve_split(lambda t, y, args: jnp.zeros_like(y), kick, y, t1=horizon, dt0=cap, dtmax=cap, + ts=jnp.asarray(times), key=key, noise_dim=2 if gc else 3, adaptive=adaptive, + method='dopri5', max_steps=100_000, + event=None if threshold is None else (lambda t, y, args: speed_of(y) - threshold), + step_limit=None if threshold is None else limit, + event_dtmax=None if threshold is None else event_interval) + return jax.vmap(one)(y0, keys) + return batch + + +def estimate(values, expected, variance=False): + """Sample mean or variance against a reference, with standard error and 95% half-width.""" + x = np.asarray(values, float) + n = len(x) + influence, value = (n / (n - 1) * (x - x.mean())**2, x.var(ddof=1)) if variance else (x, x.mean()) + se = influence.std(ddof=1) / np.sqrt(n) + return dict(value=float(value), reference=float(expected), difference=float(value - expected), se=float(se), + half95=float(student_t.ppf(.975, n - 1) * se), z=float((value - expected) / se) if se > 1e-14 else None) + + +def collisions_suite(options): + """Thermalisation or first passage of the production collision maps against the references.""" + law, reference_coefficients = native_law(options.law), reference_law(options.law) + n, horizon, passage = options.particles, options.horizon, options.study == 'passage' + times = np.linspace(0., horizon, 2 if passage else 41) + keys = jax.vmap(lambda i: jax.random.fold_in(jax.random.key(options.seed), i))(jnp.arange(n)) + rows, states = [], {} + for model in ('gc', 'lorentz'): + gc = model == 'gc' + lower = .05 * np.sqrt(2.) if gc else 0. + y0 = jnp.broadcast_to(jnp.array([np.sqrt(10.), .6] if gc else [.8 * np.sqrt(10.), 0., .6 * np.sqrt(10.)]), + (n, 2 if gc else 3)) + solver = collision_solver(law, model, options.stepping == 'adaptive', times, horizon, + threshold=2. if passage else None) + out, timing = measure(solver, y0, keys, jnp.asarray(options.collision_tolerance), jnp.asarray(options.cap), + repeats=options.repeats) + if passage: + row = passage_row(options, reference_coefficients, out) + else: + row = thermal_row(reference_coefficients, lower, times, out, gc) + rows.append(report_row(f'collisions/{options.study}/{model}', n, timing, out.status, **row)) + print(f" {model}: {row['summary']}", flush=True) + states[f'{options.study}_{model}'] = np.asarray(out.ys if not passage else out.t) + return rows, states + + +def thermal_row(law, lower, times, out, gc): + """Moment comparison at every saved time against the reference and its mesh refinement.""" + expected = Reference(law, lower=lower).moments(times) + refined = Reference(law, lower=lower, nodes=1536).moments(times) + refinement = max(float(np.max(abs(expected[k] - refined[k])) / max(1., np.max(abs(refined[k])))) + for k in expected) + ys = np.asarray(out.ys) + r = ys[..., 0] if gc else np.linalg.norm(ys, axis=-1) + xi = ys[..., 1] if gc else ys[..., 2] / r + stats = [dict(speed_mean=estimate(r[:, i], expected['speed_mean'][i]), + speed_variance=estimate(r[:, i], expected['speed_variance'][i], True), + pitch_mean=estimate(xi[:, i], expected['pitch_mean'][i]), + pitch_variance=estimate(xi[:, i], expected['pitch_variance'][i], True)) for i in range(len(times))] + worst = max(abs(v['z']) for s in stats[1:] for v in s.values() if v['z'] is not None) + critical = float(norm.ppf(1 - .01 / (2 * 4 * (len(times) - 1)))) + return dict(final=stats[-1], reference_refinement=refinement, max_abs_z=worst, critical_z=critical, + consistent=bool(worst <= critical), + summary=f'max |z| {worst:.2f} (familywise {critical:.2f}), reference refinement {refinement:.1e}') + + +def passage_row(options, law, out): + """First-passage mean, variance and CDF distance against the absorbing-boundary reference.""" + reference = Reference(law, lower=2., absorbing=True) + cdf, mean, variance = reference.passage() + refined = Reference(law, lower=2., absorbing=True, nodes=1536).passage() + hit = np.asarray(out.status) == RKStatus.EVENT + samples = np.asarray(out.t)[hit] + row = dict(hits=int(hit.sum()), censored=int((~hit).sum()), reference_mean=float(mean), + reference_variance=float(variance), reference_refinement=float(abs(refined[1] - mean) / mean)) + if hit.sum() < 2: + return row | dict(summary='fewer than two particles reached the boundary') + row |= dict(mean=estimate(samples, mean), variance=estimate(samples, variance, True), + ks_distance=float(kstest(samples, cdf).statistic)) + return row | dict(summary=f"mean z {row['mean']['z']:.2f}, variance z {row['variance']['z']:.2f}, " + f"KS {row['ks_distance']:.3f}, censored {row['censored']}") + + +# --------------------------------------------------------------------------- driver + +def parse(argv): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument('--suite', choices=('orbit', 'collisions'), required=True) + parser.add_argument('--field', choices=('coils', 'vmec', 'boozer'), default='coils') + parser.add_argument('--model', choices=('gc', 'lorentz'), default='gc', help='Lorentz needs coil fields') + parser.add_argument('-N', '--particles', type=int, default=64) + parser.add_argument('--stepping', choices=('fixed', 'adaptive'), default='adaptive') + parser.add_argument('--execution', choices=(*EXECUTIONS, 'both')) + parser.add_argument('--method', choices=('dopri5', 'dopri8'), default='dopri8', help='orbit suite only') + parser.add_argument('--repeats', type=int, default=3) + parser.add_argument('--horizon', type=float, help='seconds; collision time 1/nu_s for the collisions suite') + parser.add_argument('--steps', type=int, help='fixed steps over the horizon; divisible by saves-1') + parser.add_argument('--saves', type=int, default=8) + parser.add_argument('--energy-ev', type=float, default=4000.) + parser.add_argument('--rtol', type=float, default=1e-9) + parser.add_argument('--atol', type=float, default=1e-9) + parser.add_argument('--seed', type=int, default=42) + parser.add_argument('--max-steps', type=int, default=4096) + parser.add_argument('--wout', type=Path, default=WOUT) + parser.add_argument('--boozer-modes', type=int, default=16, help='mboz = nboz of the Boozer transform') + parser.add_argument('--study', choices=('thermal', 'passage'), default='thermal') + parser.add_argument('--law', choices=('constant', 'impact'), default='constant') + parser.add_argument('--collision-tolerance', type=float, default=1e-3) + parser.add_argument('--cap', type=float, default=.0025, help='collision step cap, collisions suite') + parser.add_argument('--out', type=Path) + parser.add_argument('--save-states', action='store_true') + options = parser.parse_args(argv) + lorentz = options.model == 'lorentz' + if lorentz and options.field != 'coils' and options.suite != 'collisions': + parser.error('Lorentz requires coil fields') + if options.horizon is None: + options.horizon = (40. if options.study == 'passage' else 20.) if options.suite == 'collisions' else ( + 2e-6 if lorentz else 2e-5) + if options.steps is None: + options.steps = 1792 if lorentz else 224 + if min(options.particles, options.repeats, options.steps) < 1 or options.saves < 2 or options.horizon <= 0: + parser.error('particles, repeats and steps must be positive, saves >= 2 and the horizon > 0') + if options.stepping == 'fixed' and options.steps % (options.saves - 1): + parser.error('fixed steps must be divisible by saves - 1') + names = ['cuda' if jax.default_backend() == 'gpu' else 'jax'] if options.execution is None else ( + ['jax', 'cuda'] if options.execution == 'both' else [options.execution]) + options.executions = ['jax'] if options.field == 'boozer' else [EXECUTIONS[name] for name in names] + return options + + +def main(argv=None): + options = parse(sys.argv[1:] if argv is None else argv) + print(f'{options.suite}: backend {jax.default_backend()}, device {jax.devices()[0]}, N={options.particles}', + flush=True) + rows, states = [], {} + if options.suite == 'collisions': + rows, states = collisions_suite(options) + else: + case = Case(options) + for execution in options.executions: + new_rows, new_states = orbit_suite(options, case, execution) + rows, states = rows + new_rows, states | new_states + if options.out is not None: + options.out.mkdir(parents=True, exist_ok=True) + settings = {k: str(v) if isinstance(v, Path) else v for k, v in vars(options).items()} + (options.out / 'result.json').write_text(json.dumps( + dict(settings=settings, backend=jax.default_backend(), device=jax.devices()[0].device_kind, + jax=jax.__version__, rows=rows), indent=2) + '\n') + if options.save_states: + np.savez_compressed(options.out / 'states.npz', **states) + print(f'wrote {options.out / "result.json"}', flush=True) + return rows + + +if __name__ == '__main__': + main() diff --git a/examples/benchmarks/benchmark_native_ad.py b/examples/benchmarks/benchmark_native_ad.py deleted file mode 100644 index 355ee5d7..00000000 --- a/examples/benchmarks/benchmark_native_ad.py +++ /dev/null @@ -1,227 +0,0 @@ -"""Reproduce accepted-path gradient checks and blocked timing/memory measurements. - -Run each field/model/stepping case in a fresh process for allocator peaks. -No particle chunking, field interpolation, fitted controller or GPU selection -is imposed. JAX uses the machine's selected backend. -""" -import argparse -import json -from pathlib import Path -from time import perf_counter -import jax -import jax.numpy as jnp -import numpy as np -from essos.constants import ALPHA_PARTICLE_MASS as M, ALPHA_PARTICLE_CHARGE as Q, ONE_EV -from essos.coils import Coils -from essos.fields import BiotSavart, Vmec -from essos.boozer import BoozerField, psi0_from_vmec -from essos.dynamics import Particles -from essos.background_species import BackgroundSpecies -from essos.split import trace_split, BoozerGeometry - -ROOT=Path(__file__).resolve().parents[2] - - -def build(options): - if options.kernel_audit: - return kernel_audit(options) - geometry=None; n=options.particles - if options.field=='coils': - base=Coils.from_json(str(ROOT/'examples/input_files/ESSOS_biot_savart_LandremanPaulQA.json')) - coil_dofs=jnp.asarray(base.dofs) - points=jnp.column_stack((jnp.linspace(1.23,1.27,n),jnp.zeros((n,2)))) - field=None - elif options.field=='vmec': - field=Vmec(str(options.wout),ntheta=8,nphi=8) - coil_dofs=jnp.empty(0) - else: - from booz_xform_jax import Booz_xform - from netCDF4 import Dataset - transform=Booz_xform(verbose=0,mboz=options.mboz,nboz=options.nboz) - transform.read_wout(str(options.wout),flux=False);transform.run() - with Dataset(options.wout) as data: psi=psi0_from_vmec(data['phi'][-1]) - field=BoozerField.from_booz(transform.s_b,transform.bmnc_b,transform.xm_b,transform.xn_b, - transform.iota,transform.Boozer_G,transform.Boozer_I,psi,transform.nfp, - mode_tolerance=options.mode_cut) - geometry=BoozerGeometry.from_booz_xform(transform) - coil_dofs=jnp.empty(0) - if options.field!='coils': - rng=np.random.default_rng(options.seed) - points=jnp.asarray(np.column_stack((np.full(n,.3),rng.uniform(0,2*np.pi,n), - rng.uniform(0,2*np.pi/field.nfp,n)))) - bath=BackgroundSpecies(1,jnp.array([1.]),jnp.array([1.]),jnp.array([1e20]),jnp.array([1000.])) - pitch=jnp.linspace(.35,.65,n) - parameters=jnp.concatenate((coil_dofs,jnp.zeros(3))) - gc=options.model=='gc';adaptive=options.stepping=='adaptive' - dt=options.horizon/4 if adaptive else options.horizon/options.steps - times=jnp.linspace(0.,options.horizon,options.saves) - def run(p, *, adjoint='none', record=False, tapes=None): - source=BiotSavart(base.with_dofs(p[:-3])) if options.field=='coils' else field - shifted=points.at[:,0].add(p[-3]) - birth=Particles(shifted,mass=M,charge=Q,energy=options.energy_ev*ONE_EV*jnp.exp(p[-1]), - initial_vparallel_over_v=pitch+p[-2],field=source if not gc else None,phase_angle_full_orbit=jnp.pi) - birth.random_keys=jax.random.split(jax.random.key(options.seed),n) - return trace_split(source,birth,bath,model='GuidingCenter' if gc else 'FullOrbit', - tmax=options.horizon,timestep=dt,collision_dtmax=options.horizon/2 if adaptive else dt, - times=times,orbit_dt0=dt,adaptive=adaptive,orbit_adaptive=adaptive, - rtol=options.rtol,atol=options.atol,collision_tolerance=.001, - moment_tolerance=options.moment_tolerance,spatial_geometry=geometry, - spatial_diffusion=gc,execution='shared' if tapes is not None else options.execution, - reverse_execution=options.reverse_execution,adjoint=adjoint, - max_steps=options.max_steps,max_orbit_steps=options.max_orbit_steps, - adjoint_max_accepted_steps=options.capacity,adjoint_max_orbit_steps=options.capacity, - adjoint_checkpoint_steps=options.segment,adjoint_orbit_checkpoint_steps=options.segment, - _record_ad=record,_replay_tapes=tapes) - def metric(saved,final): - # The same smooth scalar in every chart, in that chart's public units. - # Position/pitch/energy derivatives are all exercised. This is an AD - # diagnostic, not a proposed confinement objective. - speed=np.sqrt(2*options.energy_ev*ONE_EV/M) - return jnp.mean(jnp.sum(saved[...,:3]**2,axis=-1))+jnp.mean(final[...,3]**2)/speed**2 - return parameters,run,metric - - -def kernel_audit(options): - """Small field/kernel regression fixtures, separate from physical statistics.""" - from types import SimpleNamespace - from essos.coils import CreateEquallySpacedCurves - gc=options.model=='gc';coordinate=options.field!='coils';geometry=None - if options.field=='coils': - base=Coils(CreateEquallySpacedCurves(1,1,1.,.4,7,1,False),[1e6]) - dofs=jnp.asarray(base.dofs) - elif options.field=='vmec': - field=Vmec(str(options.wout),ntheta=8,nphi=8) - else: - s=np.linspace(.005,.995,5) - field=BoozerField.from_booz(s,np.stack((s*0+5.,-.2*np.sqrt(s))),[0,1],[0,0], - s*0+.4,s*0+40.,s*0,40.,1,angle_grid=None) - radial=np.array([.1,.3,.5]);r=.2*np.sqrt(radial) - geometry=BoozerGeometry.from_booz_xform(SimpleNamespace(s_b=radial,xm_b=np.array([0,1]),xn_b=np.zeros(2), - rmnc_b=np.stack((radial*0+2.,r)),zmns_b=np.stack((radial*0,r)), - numns_b=np.stack((radial*0,.05*np.sqrt(radial))))) - bath=BackgroundSpecies(1,jnp.array([1.]),jnp.array([1.]),jnp.array([1e20]),jnp.array([1000.])) - options.particles=1;options.horizon=2e-10 if coordinate else 2e-9;options.saves=3 - options.energy_ev=4000.;options.steps=2 - p=jnp.array([.4,1.,.3 if coordinate else 0.]) - def run(p, *, adjoint='none', record=False, tapes=None): - source=field if coordinate else BiotSavart(base.with_dofs(dofs.at[0].add(p[2]))) - points=jnp.array([[p[2],.2,.1]]) if coordinate else jnp.array([[1.,.03,.01]]) - birth=Particles(points,mass=M,charge=Q,energy=4000*ONE_EV*p[1],initial_vparallel_over_v=p[:1], - field=source if not gc else None) - return trace_split(source,birth,bath,model='GuidingCenter' if gc else 'FullOrbit', - tmax=options.horizon,timestep=options.horizon/2,n_save=3, - execution='shared' if tapes is not None else options.execution, - adjoint=adjoint,reverse_execution=options.reverse_execution, - adaptive=options.stepping=='adaptive',orbit_adaptive=options.stepping=='adaptive', - moment_tolerance=.01,max_steps=16,max_orbit_steps=16, - adjoint_max_accepted_steps=16,adjoint_max_orbit_steps=32, - adjoint_orbit_checkpoint_steps=2,spatial_geometry=geometry,_record_ad=record,_replay_tapes=tapes) - speed=np.sqrt(8000*ONE_EV/M) - def metric(saved,final): - spatial=jnp.sum(saved[...,:3]**2) - return (spatial+jnp.sum(final[...,3])/speed+jnp.sum(final[...,4])/(4000*ONE_EV) if coordinate else - spatial+jnp.sum(final[...,3]**2)/1e12) - return p,run,metric - - -def measure(function, parameters, repeats): - start=perf_counter();exe=jax.jit(function).lower(parameters).compile() - compile_s=perf_counter()-start - start=perf_counter();result=jax.block_until_ready(exe(parameters)) - first_s=perf_counter()-start - stats=jax.devices()[0].memory_stats() - jax.block_until_ready(exe(parameters)) - samples=[] - for _ in range(repeats): - start=perf_counter();jax.block_until_ready(exe(parameters));samples.append(1000*(perf_counter()-start)) - memory=exe.memory_analysis() - buffers=sum(getattr(memory,k,0) or 0 for k in - ('argument_size_in_bytes','output_size_in_bytes','temp_size_in_bytes'))-(getattr(memory,'alias_size_in_bytes',0) or 0) - return result,dict(compile_s=compile_s,first_s=first_s,warm_ms=float(np.median(samples)), - samples_ms=samples,compiled_buffers_bytes=buffers,allocator_after_first=stats) - - -def main(): - parser=argparse.ArgumentParser(description=__doc__) - parser.add_argument('--field',choices=('coils','vmec','boozer'),default='coils') - parser.add_argument('--model',choices=('gc','lorentz'),default='gc') - parser.add_argument('--stepping',choices=('fixed','adaptive'),default='adaptive') - parser.add_argument('--execution',choices=('shared','independent')) - parser.add_argument('--reverse-execution',choices=('auto','shared','independent'),default='auto') - parser.add_argument('--particles',type=int,default=64);parser.add_argument('--repeats',type=int,default=3) - parser.add_argument('--horizon',type=float,default=2e-6);parser.add_argument('--steps',type=int,default=28) - parser.add_argument('--saves',type=int,default=8);parser.add_argument('--seed',type=int,default=42) - parser.add_argument('--energy-ev',type=float,default=4000.) - parser.add_argument('--rtol',type=float,default=1e-9);parser.add_argument('--atol',type=float,default=1e-9) - parser.add_argument('--moment-tolerance',type=float,default=.01) - parser.add_argument('--segment',type=int,default=16);parser.add_argument('--capacity',type=int,default=4096) - parser.add_argument('--max-steps',type=int,default=4096);parser.add_argument('--max-orbit-steps',type=int,default=256) - parser.add_argument('--wout',type=Path,default=ROOT/'examples/input_files/wout_LandremanPaul2021_QA_reactorScale_lowres.nc') - parser.add_argument('--mboz',type=int,default=16);parser.add_argument('--nboz',type=int,default=16) - parser.add_argument('--mode-cut',type=float,default=1e-4) - parser.add_argument('--check-gradient',action=argparse.BooleanOptionalAction,default=True) - parser.add_argument('--kernel-audit',action='store_true',help='One-particle derivative fixtures; manufactured Boozer spectrum/geometry') - parser.add_argument('--forward-only',action='store_true',help='Fresh-process forward allocator peak without building AD') - parser.add_argument('--cold-compile',action='store_true',help='Disable the persistent compilation cache for this process') - parser.add_argument('--output',type=Path,default=Path('output/native_ad/comparison.json')) - o=parser.parse_args() - if o.cold_compile:jax.config.update('jax_enable_compilation_cache',False) - if o.execution is None:o.execution='independent' if jax.default_backend()=='gpu' else 'shared' - if o.model=='lorentz' and o.field!='coils':parser.error('Lorentz requires Cartesian coil fields') - if min(o.particles,o.repeats,o.steps,o.saves-1,o.segment,o.capacity,o.max_steps,o.max_orbit_steps)<1: - parser.error('particle/save/work limits must be positive') - if not np.isfinite(o.horizon) or o.horizon<=0:parser.error('positive finite horizon required') - print('Preparing',o.field,o.model,o.stepping,jax.devices()[0],flush=True) - p,run,metric=build(o) - if o.forward_only: - out,forward=measure(run,p,o.repeats) - np.testing.assert_array_equal(out.status,0) - report=dict(field=o.field,model=o.model,stepping=o.stepping,particles=o.particles, - horizon_s=o.horizon,forward=forward,particle_chunking=False,kernel_audit=o.kernel_audit, - peak_scope='Fresh process: setup and first forward execution; no AD executable built', - jax=jax.__version__,device=jax.devices()[0].device_kind,cold_compile=o.cold_compile) - o.output.parent.mkdir(parents=True,exist_ok=True) - o.output.write_text(json.dumps(report,indent=2)+'\n') - print('Forward',forward['warm_ms'],'ms; saved',o.output,flush=True) - return - def objective(p): - out=run(p,adjoint='accepted') - return metric(out.ys,out.y),out - # Compile and execute AD first: the allocator peak includes this case's - # setup/AD compilation/execution, before other executables are built. - ((value,out),gradient),ad=measure(jax.value_and_grad(objective,has_aux=True),p,o.repeats) - print('Complete AD',ad['warm_ms'],'ms',flush=True) - ordinary,forward=measure(run,p,o.repeats) - ad['allocator_peak_scope']='Fresh case setup/AD compilation/first execution, before forward executable' - forward['allocator_peak_scope']='Cumulative after AD; use --forward-only for an isolated forward peak' - np.testing.assert_array_equal(out.status,0) - np.testing.assert_allclose(out.ys,ordinary.ys,rtol=3e-12,atol=1e-15) - np.testing.assert_array_equal(out.stats[:6],ordinary.stats[:6]) - if not np.isfinite(gradient).all():raise AssertionError('nonfinite gradient') - report=dict(field=o.field,model=o.model,stepping=o.stepping,particles=o.particles,horizon_s=o.horizon, - execution=o.execution,reverse_execution=o.reverse_execution,spatial_kicks=o.model=='gc', - moment_tolerance=o.moment_tolerance,rtol=o.rtol,atol=o.atol,seed=o.seed, - jax=jax.__version__,device=jax.devices()[0].device_kind,particle_chunking=False, - dofs=('pitch, energy, first coil DOF' if o.field=='coils' else 'pitch, energy, birth radius') if o.kernel_audit else ( - 'coil Fourier/current parameters and three birth controls' if o.field=='coils' else 'three birth controls; fixed equilibrium'), - kernel_audit=o.kernel_audit,cold_compile=o.cold_compile, - forward=forward,value_and_gradient=ad,total_over_forward=ad['warm_ms']/forward['warm_ms'], - value=float(value),gradient_norm=float(jnp.linalg.norm(gradient)), - collision_updates=np.asarray(out.stats.accepted).tolist(), - accepted_orbit_steps=np.asarray(out.stats.accepted_orbit_steps).tolist()) - if o.check_gradient: - recorded,tapes=jax.jit(lambda p:run(p,record=True))(p) - def frozen(p):return metric(*run(p,tapes=tapes)) - direction=jnp.array([.2,.3,.1]) if o.kernel_audit else jnp.asarray(np.random.default_rng(24).normal(size=p.size)) - if not o.kernel_audit:direction/=jnp.linalg.norm(direction) - _,expected=jax.jit(lambda p:jax.jvp(frozen,(p,),(direction,)))(p) - actual=jnp.vdot(gradient,direction) - np.testing.assert_allclose(actual,expected,rtol=5e-7,atol=1e-9) - report.update(directional_reverse=float(actual),directional_frozen_jvp=float(expected)) - o.output.parent.mkdir(parents=True,exist_ok=True) - o.output.write_text(json.dumps(report,indent=2)+'\n') - print('Forward',forward['warm_ms'],'ms; complete/forward',report['total_over_forward'],flush=True) - print('Saved',o.output,flush=True) - - -if __name__=='__main__':main() diff --git a/examples/benchmarks/benchmark_native_boozer_alpha.py b/examples/benchmarks/benchmark_native_boozer_alpha.py deleted file mode 100644 index 7d1671ed..00000000 --- a/examples/benchmarks/benchmark_native_boozer_alpha.py +++ /dev/null @@ -1,204 +0,0 @@ -"""8192-alpha Boozer workload: native Dopri5/8 and the existing fixed RK4 tracer. - -Setup/field transformation is excluded from trace timings. JAX chooses the -machine backend; --execution selects the native solver layout, never a device. -""" -import argparse -import json -import hashlib -from pathlib import Path -from time import perf_counter -import jax -import jax.numpy as jnp -import numpy as np -from netCDF4 import Dataset -from essos.boozer import BoozerField, _start, _advance, psi0_from_vmec -from essos.rk import _boozer_batch as _native_orbits -from essos.constants import ALPHA_PARTICLE_MASS as MASS, ALPHA_PARTICLE_CHARGE as CHARGE, ONE_EV -from essos.rk import RKStatus - -ROOT=Path(__file__).resolve().parents[2] -PARAMETERS=dict(particles=8192,horizon=.02,saves=101,energy_ev=3.52e6,birth_s=.3,seed=42, - timestep=1.25e-7,integrator="essos_dopri8",rtol=1e-10,atol=1e-10,controller="i", - max_steps=1_000_000,mboz=32,nboz=32,mode_cut=1e-4,repeats=2,execution="auto", - wout=ROOT/"examples/input_files/wout_LandremanPaul2021_QA_reactorScale_lowres.nc", - output=Path("output/native_boozer_alpha")) - -def digest(path): - return hashlib.sha256(Path(path).read_bytes()).hexdigest() - - -def setup(options): - if options.field_cache is None: - from booz_xform_jax import Booz_xform - transform = Booz_xform(verbose=0, mboz=options.mboz, nboz=options.nboz) - transform.read_wout(str(options.wout), flux=False) - transform.run() - with Dataset(options.wout) as data: - psi0 = psi0_from_vmec(data["phi"][-1]) - values = {name:np.asarray(getattr(transform, key)) for name,key in ( - ("s_grid","s_b"),("bmnc","bmnc_b"),("xm","xm_b"),("xn","xn_b"), - ("iota","iota"),("G","Boozer_G"),("I","Boozer_I"))} - sine = getattr(transform, "bmns_b", None) - if sine is not None: - values["bmns"] = np.asarray(sine) - values.update(psi0=psi0, nfp=int(transform.nfp), mboz=options.mboz, - nboz=options.nboz, wout_sha256=digest(options.wout)) - else: - with np.load(options.field_cache) as cache: - names = ("s_grid","bmnc","xm","xn","iota","G","I","psi0","nfp","mboz","nboz","wout_sha256") - values = {name:cache[name] for name in names} - if "bmns" in cache: - values["bmns"] = cache["bmns"] - if (str(values["wout_sha256"]) != digest(options.wout) or - int(values["mboz"]) != options.mboz or int(values["nboz"]) != options.nboz): - raise ValueError("Cached transform does not match --wout/--mboz/--nboz") - field = BoozerField.from_booz(values["s_grid"], values["bmnc"], values["xm"], values["xn"], - values["iota"], values["G"], values["I"], float(values["psi0"]), int(values["nfp"]), - mode_tolerance=options.mode_cut, bmns=values.get("bmns")) - rng = np.random.default_rng(options.seed) - s = np.full(options.particles, options.birth_s) - theta = rng.uniform(0., 2*np.pi, options.particles) - zeta = rng.uniform(0., 2*np.pi/field.nfp, options.particles) - pitch = rng.uniform(-1., 1., options.particles) - speed = float(np.sqrt(2*options.energy_ev*ONE_EV/MASS)) - magnetic = jax.vmap(field.modB)(jnp.asarray(s), jnp.asarray(theta), jnp.asarray(zeta)) - if not np.all(np.isfinite(magnetic) & (np.asarray(magnetic) > 0)): - raise ValueError("Birth field must be positive and finite") - initial = jnp.asarray(np.column_stack((np.sqrt(s)*np.cos(theta), np.sqrt(s)*np.sin(theta), zeta, speed*pitch))) - mu = speed**2*(1-jnp.asarray(pitch)**2)/(2*magnetic) - births = dict(s=s, theta=theta, zeta=zeta, pitch=pitch, initial=np.asarray(initial), mu=np.asarray(mu)) - return field, initial, mu, speed, births, values - - -def plots(folder, times, states, loss_times, errors, counts): - import matplotlib - matplotlib.use("Agg") - import matplotlib.pyplot as plt - fig, axes = plt.subplots(2, 2, figsize=(11, 8), constrained_layout=True) - lost = (loss_times[:,None] >= 0) & (loss_times[:,None] <= times) - axes[0,0].plot(times*1e3, 100*lost.mean(axis=0)) - axes[0,0].set(xlabel="Time (ms)", ylabel="Lost (%)", title="QA equilibrium boundary: s=1") - for q in (50, 95, 100): - axes[0,1].plot(times*1e3, np.percentile(errors,q,axis=0), label=f"{q}th percentile") - axes[0,1].set(xlabel="Time (ms)", ylabel="Relative energy drift at saved states") - axes[0,1].legend() - axes[1,0].hist(counts[:,1], bins=40, alpha=.7, label="Accepted") - axes[1,0].hist(counts[:,2], bins=40, alpha=.7, label="Rejected") - axes[1,0].set(xlabel="Steps per particle", ylabel="Particles") - axes[1,0].legend() - angles = np.linspace(0., 2*np.pi, 201) - axes[1,1].plot(np.cos(angles), np.sin(angles), "k--", lw=.8) - selected = np.random.default_rng(17).choice(len(states), min(10,len(states)), replace=False) - for i in selected: - radius = np.sqrt(np.maximum(states[i,:,0],0.)) - axes[1,1].plot(radius*np.cos(states[i,:,1]), radius*np.sin(states[i,:,1]), lw=.8) - axes[1,1].set(xlabel="sqrt(s) cos(theta)", ylabel="sqrt(s) sin(theta)", aspect="equal", - title="Up to ten trajectories: flux chart, not metres") - fig.savefig(folder/"benchmark.png", dpi=160) - plt.close(fig) - - -def standard_rk4_orbits(field, initial, mu, speed, times, timestep, seed): - """Call the existing production RK4 driver; no new RK or field formula.""" - intervals = len(times)-1 - n_sub = max(1, int(np.ceil(float(times[-1])/(intervals*timestep)-1e-9))) - dt = float(times[-1])/(intervals*n_sub) - keys = jax.random.split(jax.random.PRNGKey(seed), len(initial)) - - def run(y0, mu0): - carry, first = _start(field, y0, mu0) - carry, values = _advance(field, dt, n_sub, MASS, CHARGE, None, 1.5, - carry, keys, 0, intervals, "rk4", 1e-10) - states = jnp.concatenate((first[:, None], values), axis=1) - status = jnp.where(carry[6] >= 0, RKStatus.NONFINITE, - jnp.where(carry[4] >= 0, RKStatus.EVENT, RKStatus.SUCCESS)) - counts = jnp.stack((carry[10], carry[10], jnp.zeros_like(carry[10])), axis=-1) - return states, carry[4], carry[6], carry[7], counts, status - - return run, dt, n_sub*intervals - - - -def compiler_options(): - if jax.default_backend()=="gpu": - return dict(xla_gpu_enable_command_buffer="FUSION,WHILE,CONDITIONAL",xla_gpu_graph_min_graph_size=1) - return {} - - -def compiled_bytes(executable): - memory=executable.memory_analysis() - return int(memory.argument_size_in_bytes+memory.output_size_in_bytes+memory.temp_size_in_bytes-memory.alias_size_in_bytes) - - -def main(argv=None): - parser=argparse.ArgumentParser(description=__doc__) - for name,value in PARAMETERS.items(): - parser.add_argument('--'+name.replace('_','-'),type=type(value),default=value) - parser.add_argument('--field-cache',type=Path) - options=parser.parse_args(argv) - if options.field_cache is not None and not options.field_cache.is_file(): - parser.error('--field-cache must name an existing boozer_inputs.npz; omit it to build the field') - if options.integrator not in ('essos_rk45','essos_dopri8','rk4') or options.execution not in ('auto','jax','cuda'): - parser.error('Require integrator essos_rk45/essos_dopri8/rk4 and execution auto/jax/cuda') - if (min(options.particles,options.horizon,options.energy_ev,options.timestep,options.atol,options.repeats)<=0 - or options.saves<2 or options.rtol<0 or not 0=2, rtol>=0, 0=0)),failed=int(np.count_nonzero(failed>=0)), - max_relative_energy_error=float(energy.max()) if np.isfinite(energy.max()) else None,fixed_dt=dt,fixed_grid_slots=slots, - collisions=False,reverse_ad=False,particle_chunking=False, - peak_memory_scope='one compiled trace in this process; setup and compilation included', - gpu_memory=jax.devices()[0].memory_stats()) - (options.output/'benchmark.json').write_text(json.dumps(report,indent=2,allow_nan=False)+'\n') - np.savez_compressed(options.output/'traces.npz',times=np.asarray(times),states=states,loss_times=loss,failed_times=failed, - energy_error=energy,counts=counts,status=status) - plots(options.output,np.asarray(times),states,loss, - np.abs(states[:,:,4]**2/speed**2-1),counts) - print(f'Warm median {np.median(samples):.3f} s; lost {report["lost"]}; failed {report["failed"]}; saved {options.output.resolve()}',flush=True) - - -if __name__=='__main__': main() diff --git a/examples/benchmarks/benchmark_native_orbits.py b/examples/benchmarks/benchmark_native_orbits.py deleted file mode 100644 index 2bf7b621..00000000 --- a/examples/benchmarks/benchmark_native_orbits.py +++ /dev/null @@ -1,234 +0,0 @@ -"""Forward native/Diffrax comparison on bundled coil or VMEC/Boozer fields. - -Each pair uses identical births, fields, save times, error blocks and stepping -mode. Fixed runs use the same dt. All executions block on device completion; -setup and reference integration are excluded. No collisions, AD or chunking. -""" -import argparse -import csv -import hashlib -import json -from pathlib import Path -from time import perf_counter -import diffrax -import equinox as eqx -import jax -import jax.numpy as jnp -import numpy as np -from essos.constants import ALPHA_PARTICLE_MASS as M, ALPHA_PARTICLE_CHARGE as Q, ONE_EV -from essos.coils import Coils -from essos.fields import BiotSavart, Vmec -from essos.dynamics import Particles, GuidingCenter, Lorentz, _axis_regular, _to_axis_regular, _from_axis_regular -from essos.electric_field import Electric_field_zero -from essos.boozer import guiding_center_rhs -from essos.rk import prepare_solver -from essos.rk import _boozer_error_norm -from essos.rk import OrbitErrorNorm -from essos.rk import solve_rk -from .benchmark_native_boozer_alpha import PARAMETERS, setup, compiler_options, compiled_bytes - -ROOT=Path(__file__).resolve().parents[2] - - -def case(options): - rng=np.random.default_rng(options.seed) - speed=float(np.sqrt(2*options.energy_ev*ONE_EV/M)) - if options.field=='coils': - field=BiotSavart(Coils.from_json(str(ROOT/'examples/input_files/ESSOS_biot_savart_LandremanPaulQA.json'))) - points=jnp.column_stack((jnp.linspace(1.23,1.27,options.particles),jnp.zeros((options.particles,2)))) - p=Particles(points,mass=M,charge=Q,energy=options.energy_ev*ONE_EV, - initial_vparallel_over_v=jnp.linspace(.35,.65,options.particles), - field=field if options.model=='lorentz' else None,phase_angle_full_orbit=jnp.pi) - initial=(jnp.column_stack((points,p.initial_vparallel)) if options.model=='gc' else - jnp.column_stack((p.initial_xyz_fullorbit,p.initial_vxvyvz))) - rhs=(lambda t,y,mu:GuidingCenter(t,y,(field,p,Electric_field_zero()))) if options.model=='gc' else ( - lambda t,y,mu:Lorentz(t,y,(field,p))) - norm=OrbitErrorNorm(options.rtol,options.atol,speed,velocity_components=1 if options.model=='gc' else 3) - event=None - moment=jnp.zeros(options.particles) - elif options.field=='vmec': - field=Vmec(str(options.wout),ntheta=8,nphi=8) - points=jnp.asarray(np.column_stack((np.full(options.particles,.3),rng.uniform(0,2*np.pi,options.particles), - rng.uniform(0,2*np.pi/field.nfp,options.particles)))) - p=Particles(points,mass=M,charge=Q,energy=options.energy_ev*ONE_EV, - initial_vparallel_over_v=jnp.asarray(rng.uniform(-.7,.7,options.particles))) - initial=jax.vmap(_to_axis_regular)(jnp.column_stack((points,p.initial_vparallel))) - rhs=_axis_regular(lambda t,y,mu:GuidingCenter(t,y,(field,p,Electric_field_zero()))) - norm=OrbitErrorNorm(options.rtol,options.atol,speed, - position_map=lambda y:field.to_xyz(_from_axis_regular(y)[:3])) - event=lambda t,y,args,**kwargs:1.-y[0]**2-y[1]**2 - moment=jnp.zeros(options.particles) - else: - settings=argparse.Namespace(**(PARAMETERS|vars(options))) - settings.birth_s=.3 - field,initial,moment,speed,_,_=setup(settings) - rhs=lambda t,y,mu:guiding_center_rhs(field,y,mu,M,Q) - norm=lambda y0,y1,error:_boozer_error_norm(error,speed,options.rtol,options.atol) - event=lambda t,y,args,**kwargs:1.-y[0]**2-y[1]**2 - return field,initial,moment,speed,rhs,norm,event - - -def functions(options,data,method,tolerance=None): - field,initial,moment,speed,rhs,norm,event=data - adaptive=options.stepping=='adaptive' - times=np.linspace(0.,options.horizon,options.saves) - dt=options.horizon/4 if adaptive else options.horizon/options.steps - rtol,atol=(options.rtol,options.atol) if tolerance is None else (tolerance,tolerance) - if tolerance is not None: - # Scale the same normalized physical/chart error budget for the reference. - original=norm - norm=lambda y0,y1,e:original(y0,y1,e)*(options.rtol+options.atol)/(rtol+atol) - if options.field=='boozer': - if options.execution=='cuda' and tolerance is None: - from essos.rk import _prepare_boozer_gpu as prepare_orbit_solver - solve=prepare_orbit_solver(field,M,Q,speed,times,dt,rtol,atol,options.max_steps, - method=method,adaptive=adaptive) - else: - solve=lambda y,mu:solve_rk(rhs,y,mu,t1=options.horizon,dt0=dt,ts=times,method=method, - adaptive=adaptive if tolerance is None else True,error_norm=norm,event=event, - rtol=rtol,atol=atol,max_steps=options.max_steps) - else: - prepared=prepare_solver(field,mass=M,charge=Q,energy=options.energy_ev*ONE_EV,times=times, - dt0=dt,model=options.model,method=method,adaptive=adaptive,rtol=rtol,atol=atol, - max_steps=options.max_steps,execution=options.execution) - solve=lambda y,mu:prepared(y) - def native_one(y,mu): - out=solve(y,mu) - return out.ys,out.y,out.t,out.status,jnp.stack(out.stats) - - # Feed the same scalar embedded block error to Diffrax's ordinary I controller. - # Its default component RMS error would impose a different acceptance test. - base=diffrax.Dopri5 if method=='dopri5' else diffrax.Dopri8 - class PhysicalSolver(base): - physical_norm: object=eqx.field(static=True,default=None) - def step(self,terms,t0,t1,y0,args,solver_state,made_jump): - y1,error,dense,state,result=super().step(terms,t0,t1,y0,args,solver_state,made_jump) - return y1,jnp.full_like(error,self.physical_norm(y0,y1,error)),dense,state,result - solver=PhysicalSolver(physical_norm=norm) if adaptive or tolerance is not None else base() - control=(diffrax.PIDController(rtol=0.,atol=1.,pcoeff=0.,icoeff=1.,dcoeff=0.,factormin=.2,factormax=5.) - if adaptive or tolerance is not None else diffrax.ConstantStepSize()) - def diffrax_one(y,mu): - out=diffrax.diffeqsolve(diffrax.ODETerm(rhs),solver,t0=0.,t1=options.horizon,dt0=dt,y0=y,args=mu, - stepsize_controller=control,event=None if event is None else diffrax.Event(event, - root_finder=__import__('optimistix').Bisection(rtol=1e-12,atol=1e-12)), - saveat=diffrax.SaveAt(subs=[diffrax.SubSaveAt(ts=jnp.asarray(times)),diffrax.SubSaveAt(t1=True)]), - max_steps=options.max_steps,throw=False) - status=jnp.where(out.result==diffrax.RESULTS.successful,0, - jnp.where(out.result==diffrax.RESULTS.event_occurred,1,2)) - counts=jnp.stack([out.stats[k] for k in ('num_steps','num_accepted_steps','num_rejected_steps')]) - return out.ys[0],out.ys[1][-1],out.ts[1][-1],status,counts - return jax.jit(jax.vmap(native_one),compiler_options=compiler_options()),jax.jit(jax.vmap(diffrax_one),compiler_options=compiler_options()) - - -def _suite(options): - """Regenerate the README workloads and their aggregate files under output/.""" - rows = [] - for field,model,horizon,steps in (('boozer','gc',options.horizon,options.steps), - ('vmec','gc',options.horizon,options.steps), - ('coils','gc',options.horizon,224), - ('coils','lorentz',2e-6,1792)): - for stepping in ('adaptive','fixed'): - output = options.output/f'{field}_{model}_{stepping}' - settings = vars(options) | dict(field=field,model=model,horizon=horizon, - steps=steps,stepping=stepping,output=output) - arguments = [] - for name,value in settings.items(): - if name=='suite' or value is None or (name=='field_cache' and field!='boozer'): - continue - arguments += ['--'+name.replace('_','-'),*(str(v) for v in value)] if isinstance(value,list) else [ - '--'+name.replace('_','-'),str(value)] - main(arguments) - report = json.loads((output/'comparison.json').read_text()) - for name,row in report['rows'].items(): - rows.append(dict(field=field,model=model,stepping=stepping,solver=name, - **{k:v for k,v in row.items() if k!='warm_ms'})) - with (options.output/'forward.csv').open('w',newline='') as stream: - writer = csv.DictWriter(stream,fieldnames=list(rows[0])) - writer.writeheader() - writer.writerows(rows) - table = ['| Field / model | Stepping | Solver | Warm ms | Buffers MiB |', - '|---|---|---|---:|---:|'] - for row in rows: - table.append(f'| {row["field"]} / {row["model"]} | {row["stepping"]} | {row["solver"]} | ' - f'{row["warm_median_ms"]:.3f} | {row["compiled_buffers_bytes"]/2**20:.3f} |') - (options.output/'forward.md').write_text('\n'.join(table)+'\n') - - -def main(argv=None): - parser=argparse.ArgumentParser(description=__doc__) - parser.add_argument('--field',choices=['coils','vmec','boozer'],default='coils') - parser.add_argument('--model',choices=['gc','lorentz'],default='gc') - parser.add_argument('--stepping',choices=['fixed','adaptive'],default='adaptive') - parser.add_argument('--execution',choices=['jax','cuda'],default='cuda') - parser.add_argument('--suite',action='store_true',help='Regenerate all eight README workloads and aggregate CSV/Markdown') - parser.add_argument('--cases',nargs='+',default=['native_dopri8','diffrax_dopri8']) - for name,value in dict(particles=64,horizon=2e-5,saves=8,steps=160,repeats=3,seed=42,energy_ev=4000., - rtol=1e-9,atol=1e-9,max_steps=4096,mboz=32,nboz=32,mode_cut=1e-4).items(): - parser.add_argument('--'+name.replace('_','-'),type=type(value),default=value) - parser.add_argument('--wout',type=Path,default=PARAMETERS['wout']) - parser.add_argument('--field-cache',type=Path) - parser.add_argument('--output',type=Path,default=Path('output/native_forward_comparison')) - options=parser.parse_args(argv) - if options.model=='lorentz' and options.field!='coils': parser.error('Lorentz requires Cartesian coils') - if options.field_cache is not None and not options.field_cache.is_file(): parser.error('Missing --field-cache') - if min(options.particles,options.horizon,options.steps,options.repeats,options.atol)<=0 or options.saves<2 or options.rtol<0: - parser.error('Require positive counts, horizon, steps, repeats and atol; saves>=2, rtol>=0') - if options.suite: - return _suite(options) - if (options.output/'comparison.json').exists(): parser.error('Choose a fresh output directory') - options.output.mkdir(parents=True,exist_ok=True) - print(f'{jax.devices()}; {options.field}/{options.model}; {options.stepping}; {options.execution}',flush=True) - data=case(options) - initial,moment=data[1:3] - jax.block_until_ready(data[1:4]) - report=dict(settings={k:str(v) if isinstance(v,Path) else v for k,v in vars(options).items()}, - device=jax.devices()[0].device_kind,jax=jax.__version__,collisions=False,reverse_ad=False, - source_sha256={name:hashlib.sha256((ROOT/'essos'/name).read_bytes()).hexdigest() - for name in ('rk.py','fields.py','boozer.py','dynamics.py')}, - birth_sha256=hashlib.sha256(np.asarray(initial).tobytes()).hexdigest(), - input_sha256=hashlib.sha256((ROOT/'examples/input_files/ESSOS_biot_savart_LandremanPaulQA.json' - if options.field=='coils' else options.wout).read_bytes()).hexdigest(), - compiled_buffers_scope='argument + output + temporary - alias; excludes executable constants and resident setup data', - rows={}) - outputs={}; executables={} - for name in options.cases: - backend,method=name.split('_',1) - if backend not in ('native','diffrax') or method not in ('dopri5','dopri8'): parser.error('Invalid --cases') - native,reference=functions(options,data,method) - print('Compiling '+name,flush=True) - start=perf_counter(); exe=(native if backend=='native' else reference).lower(initial,moment).compile() - compile_s=perf_counter()-start - output=jax.block_until_ready(exe(initial,moment)) - jax.block_until_ready(exe(initial,moment)) - outputs[name]=output;executables[name]=exe - report['rows'][name]=dict(compile_s=compile_s,compiled_buffers_bytes=compiled_bytes(exe),warm_ms=[]) - for iteration in range(options.repeats): - for name in list(executables) if iteration%2==0 else reversed(list(executables)): - start=perf_counter(); jax.block_until_ready(executables[name](initial,moment)) - report['rows'][name]['warm_ms'].append(1000*(perf_counter()-start)) - _,reference=functions(options,data,'dopri8',tolerance=1e-12) - truth=jax.block_until_ready(reference(initial,moment))[0] - for name,row in report['rows'].items(): - states,_,_,status,counts=map(np.asarray,outputs[name]); target=np.asarray(truth) - valid=np.isfinite(states).all(-1)&np.isfinite(target).all(-1) - if options.field=='vmec': - mapped=jax.jit(jax.vmap(jax.vmap(lambda y:data[0].to_xyz(_from_axis_regular(y)[:3])))) - position=np.asarray(mapped(jnp.asarray(states)))-np.asarray(mapped(jnp.asarray(target))) - else: position=states[:,:,:3]-target[:,:,:3] - row.update(warm_median_ms=float(np.median(row['warm_ms'])),accepted=int(counts[:,1].sum()), - rejected=int(counts[:,2].sum()),failed=int(np.count_nonzero(status>1)),lost=int(np.count_nonzero(status==1)), - max_position_or_chart_error=float(np.max(np.linalg.norm(position[valid],axis=-1))), - max_velocity_error_m_s=float(np.max(np.linalg.norm( - (states[:,:,3:6 if options.model=='lorentz' else 4]-target[:,:,3:6 if options.model=='lorentz' else 4])[valid],axis=-1))), - max_velocity_error_over_birth_speed=float(np.max(np.linalg.norm( - (states[:,:,3:6 if options.model=='lorentz' else 4]-target[:,:,3:6 if options.model=='lorentz' else 4])[valid],axis=-1))/data[3])) - print(f'{name}: {row["warm_median_ms"]:.3f} ms, {row["accepted"]} accepted, {row["rejected"]} rejected',flush=True) - report['gpu_memory']=jax.devices()[0].memory_stats() - report['peak_scope']='cumulative across setup, compilation, all cases and reference; not a per-solver peak' - (options.output/'comparison.json').write_text(json.dumps(report,indent=2,allow_nan=False)+'\n') - with (options.output/'comparison.csv').open('w',newline='') as f: - rows=[dict(case=name,**{k:v for k,v in row.items() if k!='warm_ms'}) for name,row in report['rows'].items()] - writer=csv.DictWriter(f,fieldnames=rows[0]);writer.writeheader();writer.writerows(rows) - - -if __name__=='__main__': main() diff --git a/examples/benchmarks/benchmark_split_orbits.py b/examples/benchmarks/benchmark_split_orbits.py deleted file mode 100644 index 8cb5d218..00000000 --- a/examples/benchmarks/benchmark_split_orbits.py +++ /dev/null @@ -1,221 +0,0 @@ -"""Matched forward split timing and memory on bundled coil/VMEC/Boozer fields. - -Each case runs in a fresh process when --case all is selected. Fixed cases -use identical orbit/collision intervals; adaptive native cases adapt both. -Diffrax uses the existing fixed SDE tracer. Boozer's existing collisional -baseline is its fixed Dopri8/Euler implementation, labelled separately. -Field construction is excluded from blocked warm timings and included in -allocator peaks. No custom reverse rule or particle chunking is used. -""" -import argparse -import hashlib -import json -from pathlib import Path -import subprocess -import sys -from time import perf_counter -import diffrax -import jax -import jax.numpy as jnp -import numpy as np -from essos.background_species import BackgroundSpecies -from essos.constants import ALPHA_PARTICLE_MASS as M, ALPHA_PARTICLE_CHARGE as Q, ONE_EV, SPEED_OF_LIGHT -from essos.dynamics import Particles, _to_axis_regular, _from_axis_regular, _axis_regular, _SDE_MODELS, _EVENT_ROOT_FINDER -from essos.electric_field import Electric_field_zero -from essos.split import trace_split -from .benchmark_native_orbits import case, functions -from .benchmark_native_boozer_alpha import PARAMETERS, compiler_options, compiled_bytes - -MODULE = 'examples.benchmarks.benchmark_split_orbits' - - -def build(options): - data = case(options); field, initial, moment, speed, *_ = data - if options.field=='coils': - # Materialize cached geometry during field setup, before the outer JIT. - field.coils.gamma, field.coils.gamma_dash, field.coils.currents - if options.field == 'vmec': - physical = jax.vmap(_from_axis_regular)(initial); points = physical[:, :3]; pitches = physical[:, 3]/speed - elif options.field == 'boozer': - points = jnp.column_stack((initial[:,0]**2+initial[:,1]**2,jnp.arctan2(initial[:,1],initial[:,0]),initial[:,2])) - pitches = initial[:,3]/speed - else: - points = initial[:,:3]; pitches = jnp.linspace(.35,.65,options.particles) - birth = Particles(points,mass=M,charge=Q,energy=options.energy_ev*ONE_EV,initial_vparallel_over_v=pitches, - initial_xyz_fullorbit=initial[:,:3] if options.model=='lorentz' else None, - initial_vxvyvz=initial[:,3:] if options.model=='lorentz' else None) - birth.random_keys = jax.random.split(jax.random.PRNGKey(options.seed),options.particles) - background = BackgroundSpecies(1,jnp.array([1.]),jnp.array([1.]),jnp.array([1e20]),jnp.array([1000.])) - times = jnp.linspace(0.,options.horizon,options.saves) - adaptive = options.stepping == 'adaptive'; dt = options.horizon/4 if adaptive else options.horizon/options.steps - fingerprint = hashlib.sha256(np.asarray(jnp.column_stack((points,pitches))).tobytes()).hexdigest() - if options.case.startswith('native_') and options.case != 'native_collisionless': - def run(): - out = trace_split(field,birth,background,model='GuidingCenter' if options.model=='gc' else 'FullOrbit', - tmax=options.horizon,timestep=dt,collision_dtmax=options.horizon/2 if adaptive else dt, - orbit_dt0=dt,adaptive=adaptive,orbit_adaptive=adaptive,times=times, - rtol=options.rtol,atol=options.atol,collision_tolerance=options.collision_tolerance, - collision_controller=options.collision_controller,moment_tolerance=options.moment_tolerance, - spatial_diffusion=options.spatial_diffusion, - max_steps=options.max_steps,max_orbit_steps=options.max_orbit_steps, - execution='independent' if options.case=='native_independent' else 'shared') - return out.ys,out.y,out.t,out.status,jnp.stack(out.stats,axis=-1) - label = 'Native Dopri8 / Milstein; '+(options.collision_controller if adaptive else 'fixed') - elif options.case == 'native_collisionless': - native,_ = functions(options,data,'dopri8') - def run(): - ys,y,t,status,counts = native.__wrapped__(initial,moment) - counts = jnp.pad(counts,((0,0),(0,4))) - return ys,y,t,status,counts - label = 'Native Dopri8; no collisions' - elif options.field == 'boozer': - from essos.boozer import _start, _advance - def run(): - carry,first = _start(field,initial,moment) - carry,states = _advance(field,dt,options.steps//(options.saves-1),M,Q,background,0., - carry,birth.random_keys,0,options.saves-1,'dopri8',1e-9) - states = jnp.concatenate((first[:,None],states),axis=1) - status = jnp.where(carry[6]>=0,4,jnp.where(carry[4]>=0,1,0)) - counts = jnp.stack((carry[10],carry[10],jnp.zeros_like(carry[10])),axis=-1) - return states,states[:,-1],carry[2],status,jnp.pad(counts,((0,0),(0,4))) - label = 'Existing Boozer fixed Dopri8 / Euler collisions (not Diffrax)' - else: - gc, vmec = options.model=='gc', options.field=='vmec' - drift,noise,stepper,dimension = _SDE_MODELS['GuidingCenterCollisionsMuFixed' if gc else 'FullOrbitCollisions'] - scale = jnp.array([1.,1.,1.,SPEED_OF_LIGHT,SPEED_OF_LIGHT**2*M] if gc else [1.]*6) - if gc: - B = jax.vmap(field.AbsB)(points) - y0 = jnp.column_stack((points,birth.initial_vparallel,M*birth.initial_vperpendicular**2/(2*B)))/scale - if vmec: y0 = jax.vmap(_to_axis_regular)(y0) - else: y0 = initial/scale - args = (field,birth,Electric_field_zero(),background,1.) if gc else (field,birth,background) - rhs = lambda t,y,a:drift(t,y,args) - diffusion = lambda t,y,a:noise(t,y,args) - if vmec: rhs,diffusion = _axis_regular(rhs),_axis_regular(diffusion) - def one(y,key): - bm = diffrax.VirtualBrownianTree(0.,options.horizon,tol=dt*.5,shape=(dimension,),key=key, - levy_area=diffrax.SpaceTimeTimeLevyArea) - terms = diffrax.MultiTerm(diffrax.ODETerm(rhs),diffrax.ControlTerm(diffusion,bm)) - event = diffrax.Event(lambda t,y,args,**k:1-y[0]**2-y[1]**2,root_finder=_EVENT_ROOT_FINDER) if vmec else None - out = diffrax.diffeqsolve(terms,stepper(),t0=0.,t1=options.horizon,dt0=dt,y0=y, - stepsize_controller=diffrax.ConstantStepSize(),event=event, - saveat=diffrax.SaveAt(subs=[diffrax.SubSaveAt(ts=times),diffrax.SubSaveAt(t1=True)]), - max_steps=options.max_steps,throw=False) - saved,terminal = out.ys[0],out.ys[1][-1] - saved = jnp.where(jnp.isfinite(saved).all(axis=1)[:,None],saved,terminal) - if vmec: saved,terminal = jax.vmap(_from_axis_regular)(saved),_from_axis_regular(terminal) - status = jnp.where(out.result==diffrax.RESULTS.successful,0,jnp.where(out.result==diffrax.RESULTS.event_occurred,1,4)) - counts = jnp.array([out.stats[k] for k in ('num_steps','num_accepted_steps','num_rejected_steps')]+[0]*4) - return saved*scale,terminal*scale,out.ts[1][-1],status,counts - run = lambda:jax.vmap(one)(y0,birth.random_keys) - label = f'Existing Diffrax {stepper.__name__}, fixed SDE steps' - return run,label,fingerprint - - -def main(): - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument('--field',choices=('coils','vmec','boozer'),default='coils') - parser.add_argument('--model',choices=('gc','lorentz'),default='gc') - parser.add_argument('--stepping',choices=('fixed','adaptive'),default='fixed') - parser.add_argument('--case',choices=('all','native_shared','native_independent','reference_fixed','native_collisionless'),default='all') - parser.add_argument('--particles',type=int,default=64); parser.add_argument('--saves',type=int,default=8) - parser.add_argument('--horizon',type=float); parser.add_argument('--steps',type=int) - parser.add_argument('--repeats',type=int,default=3); parser.add_argument('--seed',type=int,default=42) - parser.add_argument('--energy-ev',type=float,default=4000.) - parser.add_argument('--rtol',type=float,default=1e-9); parser.add_argument('--atol',type=float,default=1e-9) - parser.add_argument('--collision-tolerance',type=float,default=.001) - parser.add_argument('--collision-controller',choices=('moments','conservative'),default='moments') - parser.add_argument('--moment-tolerance',type=float,default=.005) - parser.add_argument('--spatial-diffusion',action=argparse.BooleanOptionalAction,default=None, - help='GC position diffusion; enabled by default for native GC; legacy reference omits it') - parser.add_argument('--max-steps',type=int,default=4096); parser.add_argument('--max-orbit-steps',type=int,default=256) - parser.add_argument('--field-cache',type=Path); parser.add_argument('--output',type=Path,default=Path('output/split_forward')) - parser.add_argument('--resume',action='store_true') - parser.add_argument('--cold-compile',action='store_true',help='Disable the persistent cache for measured solver compilation') - options = parser.parse_args() - if min(options.particles,options.repeats,options.max_steps,options.max_orbit_steps)<1 or options.saves<2: - parser.error('require positive work limits and at least two saves') - if not all(np.isfinite(x) and x>0 for x in (options.energy_ev,options.atol,options.collision_tolerance,options.moment_tolerance)) or options.rtol<0: - parser.error('require finite positive energy, atol and collision tolerance; rtol>=0') - if options.field_cache is not None and not options.field_cache.is_file(): parser.error('missing --field-cache') - if options.model=='lorentz' and options.field!='coils': parser.error('Lorentz requires Cartesian coil fields') - if options.case=='reference_fixed' and options.stepping!='fixed': parser.error('the reference is fixed-step only') - if options.horizon is None: options.horizon=2e-6 if options.model=='lorentz' else 2e-5 - if options.steps is None: options.steps=1792 if options.model=='lorentz' else 224 - if options.steps<1 or not np.isfinite(options.horizon) or options.horizon<=0: parser.error('positive steps and horizon required') - if options.steps%(options.saves-1): parser.error('fixed steps must be divisible by saves-1 for aligned clocks') - options.output.mkdir(parents=True,exist_ok=True) - if options.case=='all': - rows={}; cases=['native_independent' if jax.default_backend()=='gpu' else 'native_shared','native_collisionless'] - if options.stepping=='fixed': cases.append('reference_fixed') - for name in cases: - saved=options.output/name/'comparison.json' - if saved.is_file() and options.resume: - previous=json.loads(saved.read_text()) - expected={k:getattr(options,k) for k in ('field','model','stepping','particles','saves','steps','seed','rtol','atol','collision_tolerance','collision_controller','moment_tolerance')} - expected['horizon_s']=options.horizon - expected['spatial_diffusion']=bool(options.model=='gc' and name in ('native_shared','native_independent') and options.spatial_diffusion is not False) - if not all(previous.get(k)==v for k,v in expected.items()) or previous.get('energy_ev',4000.)!=options.energy_ev: - parser.error('saved case settings differ; choose a fresh output directory') - rows[name]=previous - continue - command=[sys.executable,'-m',MODULE,*sys.argv[1:],'--case',name,'--output',str(options.output/name)] - subprocess.run(command,check=True) - rows[name]=json.loads((options.output/name/'comparison.json').read_text()) - (options.output/'comparison.json').write_text(json.dumps(dict(rows=rows),indent=2)+'\n') - return - # Reuse the orbit PR's field/birth preparation and collisionless baseline. - for name,value in PARAMETERS.items(): - if not hasattr(options,name): setattr(options,name,value) - options.execution='cuda' if jax.default_backend()=='gpu' else 'jax' - if options.case=='native_independent' and options.execution!='cuda': parser.error('independent execution requires GPU') - run,label,fingerprint=build(options) - if options.cold_compile: - from jax.experimental.compilation_cache import compilation_cache - jax.config.update('jax_enable_compilation_cache',False) - # Setup may already have initialized JAX's cached cache-use decision. - compilation_cache.reset_cache() - if options.cold_compile:print(f'{options.field}/{options.model}/{options.case}: lowering',flush=True) - start=perf_counter(); lowered=jax.jit(run,compiler_options=compiler_options()).lower() - lower_s=perf_counter()-start - if options.cold_compile:print(f'Lowering {lower_s:.3f} s; compiling',flush=True) - start=perf_counter();executable=lowered.compile();backend_compile_s=perf_counter()-start - compile_s=lower_s+backend_compile_s - if options.cold_compile:print(f'Compilation {backend_compile_s:.3f} s; first execution',flush=True) - start=perf_counter();out=jax.block_until_ready(executable());first_execution_s=perf_counter()-start - jax.block_until_ready(executable()) - samples=[] - for _ in range(options.repeats): - start=perf_counter(); out=jax.block_until_ready(executable()); samples.append(1000*(perf_counter()-start)) - states,terminal,end,status,counts=map(np.asarray,out) - memory=jax.devices()[0].memory_stats() or {} - report=dict(field=options.field,model=options.model,stepping=options.stepping,case=options.case, - reference=label,backend=jax.default_backend(),device=jax.devices()[0].device_kind,jax=jax.__version__, - particles=options.particles,energy_ev=options.energy_ev,horizon_s=options.horizon,saves=options.saves,steps=options.steps, - seed=options.seed,birth_sha256=fingerprint,rtol=options.rtol,atol=options.atol, - collision_tolerance=options.collision_tolerance,collision_controller=options.collision_controller, - moment_tolerance=options.moment_tolerance, - spatial_diffusion=bool(options.model=='gc' and options.case in ('native_shared','native_independent') and options.spatial_diffusion is not False), - lower_s=lower_s,backend_compile_s=backend_compile_s,compile_s=compile_s,lower_and_compile_s=compile_s, - first_execution_s=first_execution_s,startup_s=compile_s+first_execution_s, - persistent_compilation_cache=bool(jax.config.jax_enable_compilation_cache),samples_ms=samples, - warm_ms=float(np.median(samples)),compiled_buffers_bytes=compiled_bytes(executable), - peak_allocator_bytes=memory.get('peak_bytes_in_use'),live_allocator_bytes=memory.get('bytes_in_use'), - reserved_pool_bytes=memory.get('pool_bytes'),memory_scope='Fresh process; setup, compilation and warm executions; JAX allocator only', - lost=int(np.sum(status==1)),failed=int(np.sum(status>1)), - counts_total=counts[:,:5].sum(axis=0).tolist(), - count_columns=(['collision_trials','collision_accepted','collision_rejected','orbit_trials','orbit_rejected'] - if options.case in ('native_shared','native_independent') else ['solver_steps','solver_accepted','solver_rejected','unused','unused']), - peak_pending=int(counts[:,5].max()),max_accepted_collision_error=float(counts[:,6].max()), - terminal_sha256=hashlib.sha256(terminal.tobytes()).hexdigest()) - if options.case in ('native_shared','native_independent'): - report.update(collision_updates_total=int(counts[:,1].sum()),accepted_orbit_steps_total=int(counts[:,7].sum()), - max_orbit_steps_per_collision=int(counts[:,8].max())) - if report['failed']: raise RuntimeError(f"solver failed: {report['failed']} particles") - np.savez_compressed(options.output/'states.npz',states=states,terminal=terminal,status=status,counts=counts) - (options.output/'comparison.json').write_text(json.dumps(report,indent=2)+'\n') - print(f"{options.field}/{options.model}/{options.stepping}/{options.case}: {report['warm_ms']:.3f} ms; " - f"buffers {report['compiled_buffers_bytes']/2**20:.3f} MiB; peak {report['peak_allocator_bytes']}",flush=True) - - -if __name__=='__main__': main() diff --git a/examples/coil_optimization/optimize_coils_native_ad.py b/examples/coil_optimization/optimize_coils_native_ad.py deleted file mode 100644 index c5ae5c86..00000000 --- a/examples/coil_optimization/optimize_coils_native_ad.py +++ /dev/null @@ -1,52 +0,0 @@ -"""Minimize the mean final minor radius of 8 alpha guiding centers with native reverse AD. - -The gradient with respect to the coil Fourier dofs comes from the accepted-step adjoint -(`native_adjoint="accepted"`); L-BFGS-B steps stay within MAX_CHANGE of the starting dofs. -Run time: about 2 min on a laptop CPU, mostly compilation. -""" -from pathlib import Path -import jax -import jax.numpy as jnp -import numpy as np -from scipy.optimize import minimize -from essos.coils import Coils -from essos.constants import ALPHA_PARTICLE_MASS, ALPHA_PARTICLE_CHARGE, ONE_EV -from essos.dynamics import Particles, Tracing -from essos.fields import BiotSavart -from essos.particle_losses import mean_final_radius - -COILS = Path(__file__).resolve().parents[1]/"input_files"/"ESSOS_biot_savart_LandremanPaulQA.json" -N_PARTICLES, ENERGY_EV, TMAX = 8, 4000., 1e-5 -MAX_CHANGE, MAXITER, LENGTH_WEIGHT = 0.01, 5, 1. - -base = Coils.from_json(str(COILS)) -theta0 = jnp.asarray(base.dofs) -axis = BiotSavart(base) -r_axis, z_axis = float(axis.r_axis), float(axis.z_axis) -length_limit = 1.05*float(jnp.max(base.length)) -xyz = jnp.column_stack((jnp.linspace(1.23, 1.27, N_PARTICLES), jnp.zeros((N_PARTICLES, 2)))) -pitch = jnp.linspace(0.35, 0.65, N_PARTICLES) - - -def objective(delta): - field = BiotSavart(base.with_dofs(theta0 + MAX_CHANGE*delta)) - birth = Particles(xyz, initial_vparallel_over_v=pitch, mass=ALPHA_PARTICLE_MASS, - charge=ALPHA_PARTICLE_CHARGE, energy=ENERGY_EV*ONE_EV) - out = Tracing(field=field, particles=birth, model="GuidingCenterAdaptative", integrator="essos_dopri8", - native_adjoint="accepted", rtol=1e-9, atol=1e-9, maxtime=TMAX, timestep=TMAX/2, - times_to_trace=32, max_steps=4096, progress=False) - penalty = jnp.mean(jnp.maximum(field.coils.length - length_limit, 0.)) - return mean_final_radius(out.trajectories, r_axis, z_axis) + LENGTH_WEIGHT*penalty - - -value_and_grad = jax.jit(jax.value_and_grad(objective)) - - -def fun(x): - value, gradient = value_and_grad(jnp.asarray(x)) - return float(value), np.asarray(gradient, dtype=float) - - -result = minimize(fun, np.zeros(theta0.size), jac=True, method="L-BFGS-B", - bounds=[(-1., 1.)]*theta0.size, options=dict(maxiter=MAXITER)) -print(f"mean final radius: {fun(np.zeros(theta0.size))[0]:.4f} -> {result.fun:.4f} in {result.nit} iterations") diff --git a/tests/test_native_ad.py b/tests/test_native_ad.py deleted file mode 100644 index d64ed117..00000000 --- a/tests/test_native_ad.py +++ /dev/null @@ -1,276 +0,0 @@ -"""Accepted-map reverse AD of orbit and split orbit/collision solves. - -Reverse gradients are checked against forward mode through the frozen accepted -schedule (the derivative the rule is meant to deliver), against finite differences -of that map, through the adjoint identity = , against checkpoint -layout, and against exact conservation laws. The persistent GPU reverse kernel is -run on CPU through the Pallas interpreter and compared with shared-JAX reverse. -No long tracing or Monte Carlo runs. -""" -from types import SimpleNamespace -import jax -import jax.numpy as jnp -import numpy as np -import pytest -from essos.background_species import BackgroundSpecies -from essos.constants import ALPHA_PARTICLE_MASS as M, ALPHA_PARTICLE_CHARGE as Q, ONE_EV -from essos.dynamics import Particles -from essos.native_ad import replay_accepted, replay_accepted_split, _OrbitRecorder -from essos.rk import solve_rk -from essos.split import CollisionStep, solve_split, trace_split, move_in_cartesian -from tests.test_native_orbit import UniformField, coil_field -from tests.test_native_split import bath - - -def kernels(): - def rhs(t, y, p): - return p[0]*jnp.sin(y+12*t) - def kick(t, y, h, dw, p): - return CollisionStep(y-p[2]*h*y+p[1]*(.7+.1*y)*dw, (h/(.04*jnp.exp(p[3])))**2, jnp.array(0.)) - return rhs, kick - - -def settings(method='dopri8', outer=3, inner=3): - return dict(t1=.4, dt0=.3, ts=jnp.array([0., 0., .13, .4]), key=jax.random.key(71), noise_dim=1, - method=method, rtol=1e-8, atol=1e-8, max_steps=64, max_orbit_steps=32, adjoint_checkpoint_steps=outer, - adjoint_orbit_checkpoint_steps=inner, adjoint_max_accepted_steps=32, adjoint_max_orbit_steps=128) - - -def cost(ys, y): - return jnp.sum(ys**2)+.3*jnp.sum(y**2) - - -@pytest.mark.parametrize('method', ['dopri5', 'dopri8']) -def test_split_reverse_equals_frozen_forward_mode_fd_and_adjoint_identity(method): - rhs, kick = kernels() - p, y0, o = jnp.array([.8, .1, .2, 0.]), jnp.array([.7]), settings(method) - out, tape = jax.jit(lambda p: solve_split(rhs, kick, y0, p, _record_ad=True, **o))(p) - assert out.status == 0 and out.stats.rejected > 0 and out.stats.orbit_rejected > 0 - frozen = lambda p: replay_accepted_split(rhs, kick, y0, p, tape, o['ts'], method=method) - def reverse(p, **changes): - out = solve_split(rhs, kick, y0, p, adjoint='accepted', **(o | changes)) - return out.ys, out.y - d = jnp.array([.2, -.3, .4, .5]) - # Primal values are those of the production solve; reverse equals forward mode on the frozen schedule. - np.testing.assert_allclose(jax.jit(reverse)(p)[1], out.y, rtol=3e-13) - tangent = jax.jit(lambda p, d: jax.jvp(frozen, (p,), (d,))[1])(p, d) - w = (jnp.cos(jnp.arange(4.))[:, None], jnp.array([1.7])) - bar = jax.jit(lambda p, w: jax.vjp(reverse, p)[1](w)[0])(p, w) - np.testing.assert_allclose(bar@d, sum(jnp.vdot(a, b) for a, b in zip(w, tangent)), rtol=3e-10, atol=1e-11) - assert bar[3] == 0 # the acceptance-only parameter is frozen - gradient = jax.jit(jax.grad(lambda p: cost(*reverse(p))))(p) - # Finite differences of the frozen map converge at second order to the reverse derivative. - errors = [] - for h in (1e-2, 1e-3, 1e-4): - central = (cost(*frozen(p+h*d))-cost(*frozen(p-h*d)))/(2*h) - errors.append(abs(central-gradient@d)/abs(gradient@d)) - assert errors[1] < errors[0]/30 and errors[2] < 3e-8 - # Checkpoint layout (including no checkpointing at all) never changes the gradient. - for outer, inner in ((1, 1), (8, 2), (64, 64)): - other = jax.jit(jax.grad(lambda p: cost(*reverse(p, adjoint_checkpoint_steps=outer, - adjoint_orbit_checkpoint_steps=inner))))(p) - np.testing.assert_allclose(other, gradient, rtol=1e-11, atol=1e-13) - - -@pytest.mark.parametrize('method', ['dopri5', 'dopri8']) -def test_orbit_reverse_with_dense_output_matches_forward_mode_for_every_checkpoint_size(method): - rhs, y, p = (lambda t, y, p: p*jnp.sin(y+t)), jnp.array([.7]), jnp.array(.3) - o = dict(t1=.4, dt0=.12, ts=jnp.array([0., .031, .17, .4]), method=method, max_steps=32) - out, tape = jax.jit(lambda p: solve_rk(rhs, y, p, _recorder=_OrbitRecorder(32, 3), **o))(p) - def frozen(p): - saved, terminal = replay_accepted(rhs, y, p, tape, o['ts'], method=method) - return cost(saved, terminal) - expected, tangent = jax.jvp(frozen, (p,), (jnp.ones_like(p),)) - for segment in (1, 3, 8): - value, gradient = jax.jit(jax.value_and_grad(lambda p: cost(*(lambda r: (r.ys, r.y))(solve_rk( - rhs, y, p, adjoint='accepted', adjoint_checkpoint_steps=segment, **o)))))(p) - np.testing.assert_allclose(value, expected, rtol=3e-13) - np.testing.assert_allclose(gradient, tangent, rtol=2e-10) - - -@pytest.mark.parametrize('method', ['dopri5', 'dopri8']) -def test_gradient_of_a_conserved_energy_is_its_partial_derivative_at_the_start(method): - # H = y1^2/2 + p y0^2/2 is conserved by y0' = y1, y1' = -p y0, so H(final) = H(start) for all - # (y0, y1, p): the reverse gradient must equal the partial derivatives of H at t=0. - def final_energy(x): - out = solve_rk(lambda t, y, p: jnp.array([y[1], -p*y[0]]), x[:2], x[2], t1=1., dt0=.1, - ts=jnp.array([1.]), method=method, rtol=1e-10, atol=1e-10, max_steps=1024, adjoint='accepted') - return out.y[1]**2/2+x[2]*out.y[0]**2/2 - gradient = jax.jit(jax.grad(final_energy))(jnp.array([.7, -.3, 1.3])) - np.testing.assert_allclose(gradient, [1.3*.7, -.3, .7**2/2], rtol=5e-8) - - -def test_chart_kick_has_exact_inverse_derivative_including_gauge(): - def position(y): - u, w = y[0]*jnp.cos(y[5])-y[1]*jnp.sin(y[5]), y[1]*jnp.cos(y[5])+y[0]*jnp.sin(y[5]) - return jnp.array([(3+u)*jnp.cos(y[2]), (3+u)*jnp.sin(y[2]), w]) - y, delta = jnp.array([.2, .1, .3, 2., .4, .2]), jnp.array([.03, -.02, .04]) - def fn(y, delta): - return position(move_in_cartesian(y, delta, position, implicit=True)) - by, bd = jax.jit(jax.jacrev(fn, argnums=(0, 1)))(y, delta) - np.testing.assert_allclose(bd, jnp.eye(3), atol=5e-14) - np.testing.assert_allclose(by, jax.jacfwd(position)(y), atol=8e-14) - np.testing.assert_array_equal(move_in_cartesian(y, delta, position)[3:], y[3:]) - - -def test_split_events_hold_saves_and_skip_collision_derivatives(): - rhs = lambda t, y, p: jnp.ones_like(y)*p[0] - kick = lambda t, y, h, dw, p: CollisionStep(y+p[1]*dw, jnp.array(0.), jnp.array(0.)) - o = dict(t1=1., dt0=.5, ts=jnp.array([0., .7, 1.]), key=jax.random.key(1), noise_dim=1, - event=lambda t, y, p: .2-y[0], adaptive=False, orbit_adaptive=False, max_steps=8, max_orbit_steps=8) - def run(p): - out = solve_split(rhs, kick, jnp.zeros(1), p, adjoint='accepted', **o) - return cost(out.ys, out.y) - value, g = jax.jit(jax.value_and_grad(run))(jnp.array([1., .3])) - assert np.isfinite(value) and g[1] == 0 - # The crossing time is frozen: the endpoint sensitivity is that of y = p0 * t_cross. - np.testing.assert_allclose(g[0], 2*.2*.2*2.3, atol=2e-13) - - -def test_failed_or_capacity_limited_reverse_raises(): - rhs, kick = kernels() - o = settings() | dict(adaptive=False, orbit_adaptive=False, dt0=.1, orbit_dt0=.01, adjoint_max_orbit_steps=1) - f = jax.jit(jax.grad(lambda p: cost(*(lambda r: (r.ys, r.y))( - solve_split(rhs, kick, jnp.array([.7]), p, adjoint='accepted', **o))))) - with pytest.raises(Exception, match='capacity exceeded'): - jax.block_until_ready(f(jnp.array([.8, .1, .2, 0.]))) - - -def test_ad_configuration_rejects_invalid_modes_and_capacities(): - for value in (0, -1, True, 1.5): - with pytest.raises(ValueError): - _OrbitRecorder(8, value) - for options in (dict(adjoint='bad'), dict(reverse_execution='bad'), - dict(adjoint='accepted', adjoint_orbit_checkpoint_steps=0), - dict(adjoint='accepted', reverse_execution='independent'), - dict(adjoint='accepted', _replay_tapes=object())): - with pytest.raises(ValueError): - trace_split(None, None, None, tmax=.1, timestep=.1, **options) - - -def test_closure_parameters_initial_state_and_custom_jvp_are_differentiated(): - @jax.custom_jvp - def nonlinear(y): - return jnp.sin(y) - @nonlinear.defjvp - def tangent(primals, tangents): - (y,), (dy,) = primals, tangents - return nonlinear(y), 3*jnp.cos(y)*dy - o = dict(t1=.2, dt0=.1, ts=jnp.array([0., .13, .2]), adaptive=False, max_steps=4) - def run(p, **kw): - return solve_rk(lambda t, y, a: p[0]*nonlinear(y), p[1:], **o, **kw) - p, d = jnp.array([.3, .7]), jnp.array([.2, .4]) - _, tape = jax.jit(lambda p: run(p, _recorder=_OrbitRecorder(4, 2)))(p) - frozen = lambda p: cost(*replay_accepted(lambda t, y, a: p[0]*nonlinear(y), p[1:], None, tape, o['ts'], method='dopri5')) - gradient = jax.jit(jax.grad(lambda p: cost(*(lambda r: (r.ys, r.y))(run(p, adjoint='accepted')))))(p) - np.testing.assert_allclose(gradient@d, jax.jvp(frozen, (p,), (d,))[1], rtol=3e-11) - - -def test_profile_and_collision_coefficient_derivatives_remain_active(): - from essos.split import collision_coefficients, guiding_center_collision_step - o = dict(t1=2e-7, dt0=1e-7, ts=jnp.array([0., 2e-7]), key=jax.random.key(33), noise_dim=2, adaptive=False, - orbit_adaptive=False, orbit_dt0=5e-8, method='dopri5', max_steps=8, max_orbit_steps=8, adjoint_max_orbit_steps=16) - def problem(p): - background = BackgroundSpecies(1, jnp.array([1.]), jnp.array([1.]), jnp.exp(p[0])*jnp.array([[1e20, 2e20]]), - jnp.array([[1000., 1200.]]), radial_grid=jnp.array([0., 1.]), coulomb_log_model='impact') - def kick(t, y, h, dw, a): - coefficients = collision_coefficients(background, M, Q, y[1], jnp.array([y[0], 0., 0.])) - step = guiding_center_collision_step(y[1], y[2], h, dw, coefficients, speed_floor=1.) - return step._replace(y=jnp.concatenate((y[:1], step.y))) - return (lambda t, y, a: jnp.array([.02*y[1]*y[2]/4e5, 0., 0.])), kick, jnp.array([.3+p[1], 4e5*jnp.exp(p[2]), .4]) - p, d = jnp.array([.1, .02, .03]), jnp.array([.1, .2, .1]) - metric = lambda y: y[0]+y[1]/4e5+y[2] - def reverse(p): - rhs, kick, y = problem(p) - return metric(solve_split(rhs, kick, y, adjoint='accepted', **o).y) - rhs, kick, y = problem(p) - _, tape = jax.jit(lambda: solve_split(rhs, kick, y, _record_ad=True, **o))() - def frozen(p): - rhs, kick, y = problem(p) - return metric(replay_accepted_split(rhs, kick, y, None, tape, o['ts'], method='dopri5')[1]) - gradient = jax.jit(jax.grad(reverse))(p) - np.testing.assert_allclose(gradient@d, jax.jvp(frozen, (p,), (d,))[1], rtol=2e-10, atol=1e-12) - assert abs(gradient[0]) > 1e-8 # the background density is an active parameter - - -def coil_trace(model, coils, p, **kwargs): - from essos.fields import BiotSavart - field = BiotSavart(coils.with_dofs(coils.dofs.at[0].add(p[0]))) - birth = Particles(jnp.array([[1., .03, .01], [1., .02, 0.]]), mass=M, charge=Q, - energy=4000*ONE_EV*jnp.exp(p[1]), initial_vparallel_over_v=jnp.array([.5, .3])+p[2], - field=field, phase_angle_full_orbit=0.) - return trace_split(field, birth, bath(), model=model, tmax=2e-8, timestep=1e-8, n_save=3, max_steps=64, - max_orbit_steps=64, spatial_diffusion=model == 'GuidingCenter', adaptive=False, orbit_adaptive=False, - adjoint_orbit_checkpoint_steps=4, - adjoint_max_accepted_steps=64, adjoint_max_orbit_steps=256, **kwargs) - - -coil_metric = lambda saved, final: jnp.sum(saved[..., :3]**2)+jnp.sum(final[..., 3]**2)/1e10 -coil_point, coil_direction = jnp.array([.01, .02, .03]), jnp.array([.4, -.5, .3]) - - -@pytest.mark.parametrize('model', ['GuidingCenter', 'FullOrbit']) -def test_coil_split_gradient_matches_frozen_forward_mode(model): - coils, p, d = coil_field().coils, coil_point, coil_direction - def objective(p): - out = coil_trace(model, coils, p, adjoint='accepted') - return coil_metric(out.ys, out.y) - value, gradient = jax.jit(jax.value_and_grad(objective))(p) - _, tape = jax.jit(lambda p: coil_trace(model, coils, p, _record_ad=True))(p) - frozen = lambda p: coil_metric(*coil_trace(model, coils, p, _replay_tapes=tape)) - np.testing.assert_allclose(value, frozen(p), rtol=1e-12) - np.testing.assert_allclose(gradient@d, jax.jvp(frozen, (p,), (d,))[1], rtol=5e-9) - - -@pytest.mark.parametrize('model', ['GuidingCenter', 'FullOrbit']) -def test_persistent_reverse_kernel_gradient_equals_shared_reverse(model): - # The persistent GPU forward/reverse kernels run on CPU through the Pallas interpreter. - coils = coil_field().coils - def gradient(**kwargs): - def objective(p): - out = coil_trace(model, coils, p, adjoint='accepted', **kwargs) - return coil_metric(out.ys, out.y) - return jax.jit(jax.value_and_grad(objective))(coil_point) - shared, independent = gradient(), gradient(execution='independent', interpret=True) - np.testing.assert_allclose(independent[0], shared[0], rtol=1e-11) - np.testing.assert_allclose(independent[1], shared[1], rtol=1e-9) - - -def test_classifier_stop_is_the_surface_classifier_level_and_stops_both_executions_alike(): - from essos.split import ClassifierStop - from essos.surfaces import SurfaceClassifier, SurfaceRZFourier - torus = SurfaceRZFourier(jnp.array([1., .1]), jnp.array([0., .1]), 1, 1, 0, ntheta=24, nphi=24, range_torus='full torus') - classifier = SurfaceClassifier(torus, h=.05, padding=.1) - stop = ClassifierStop(classifier, .01) - points = jnp.array([[1.02, .0, .01], [0., 1.05, -.02], [-1., .3, .0], [.5, .5, .03]]) - np.testing.assert_allclose(jax.vmap(stop.evaluate_xyz)(points), jax.vmap(classifier.evaluate_xyz)(points), atol=1e-12) - np.testing.assert_allclose(jax.vmap(jax.grad(stop.evaluate_xyz))(points), - jax.vmap(jax.grad(classifier.evaluate_xyz))(points), atol=1e-9) - field = coil_field() - birth = Particles(jnp.array([[1., .03, .01]]), mass=M, charge=Q, energy=4000*ONE_EV, - initial_vparallel_over_v=jnp.array([.5]), field=field) - kw = dict(tmax=2e-7, timestep=1e-8, n_save=3, max_steps=64, max_orbit_steps=64, adaptive=False, - orbit_adaptive=False, event=ClassifierStop(classifier, -.05)) - shared = trace_split(field, birth, bath(), **kw) - independent = trace_split(field, birth, bath(), execution='independent', interpret=True, **kw) - assert shared.status[0] == 1 and np.all(independent.status == shared.status) - np.testing.assert_allclose(independent.t, shared.t, rtol=1e-9) - - -def test_native_tracing_api_and_array_objectives_receive_physical_gradients(): - from essos.dynamics import Tracing - from essos.particle_losses import mean_final_radius, classifier_loss_fraction - def objective(v): - p = Particles(jnp.zeros((1, 3)), initial_vparallel_over_v=[.2], mass=1., charge=1., energy=.5, - initial_xyz_fullorbit=jnp.zeros((1, 3)), initial_vxvyvz=jnp.array([[v, 0., .2]])) - out = Tracing(field=UniformField(), particles=p, model='FullOrbitAdaptative', integrator='essos_dopri8', - native_adjoint='accepted', maxtime=.2, timestep=.1, times_to_trace=3, max_steps=16, progress=False) - return jnp.sum(out.trajectories[0, -1, :3]**2) - # Uniform B along z: x(t) = (v/B) sin(Bt) with B = 1, so d(x^2)/dv = 2 x sin(t) at t = .2 (x and y parts). - np.testing.assert_allclose(jax.jit(jax.grad(objective))(jnp.array(.3)), 8*.3*np.sin(.1)**2, rtol=2e-10) - states = jnp.array([[[0., 0., 0., .2, .3, .4], [.2, 0., 0., .2, .3, .4]], - [[0., 0., 0., .2, .3, .4], [.3, 0., 0., .2, .3, .4]]]) - boundary = SimpleNamespace(evaluate_xyz=lambda x: .5-x[0]) - np.testing.assert_allclose(mean_final_radius(states, 0., 0.), .25, atol=2e-12) - np.testing.assert_allclose(classifier_loss_fraction(states, boundary, softness=4.), - jnp.mean(jax.nn.sigmoid(jnp.array([-1.2, -.8]))), rtol=1e-14) diff --git a/tests/test_native_orbit.py b/tests/test_native_orbit.py index 6b6ca6c8..03bb1059 100644 --- a/tests/test_native_orbit.py +++ b/tests/test_native_orbit.py @@ -1,6 +1,5 @@ """Small forward contracts; GPU algebra is exercised on CPU via Pallas interpretation.""" from pathlib import Path -from types import SimpleNamespace import diffrax import jax import jax.numpy as jnp @@ -34,6 +33,28 @@ def test_tableau_fsal_and_standard_dense_match_diffrax(method): np.testing.assert_array_equal(a,b) +@pytest.mark.parametrize('method', ['dopri5','dopri8']) +def test_trajectories_match_diffrax_with_the_same_tolerances(method): + """Fixed steps and the PI controller reproduce diffrax's Dopri5/Dopri8 solves step for step. + + Fixed steps agree to rounding with every supported diffrax. The adaptive solves take the same number of + steps and agree to within the tolerance (about 1e-11 with diffrax 0.7.2, 2e-10 at rtol 1e-9 with 0.7.0). + """ + def rhs(t,y,args): + return jnp.array([y[1],-y[0]*(1+.3*jnp.sin(3*t))-.05*y[1]]) + y0,ts,solver=jnp.array([1.,.2]),jnp.linspace(0,6,7),(diffrax.Dopri5() if method=='dopri5' else diffrax.Dopri8()) + def reference(dt0,controller): + return diffrax.diffeqsolve(diffrax.ODETerm(rhs),solver,0.,6.,dt0,y0,saveat=diffrax.SaveAt(ts=ts), + stepsize_controller=controller,max_steps=10000) + fixed=rk.solve_rk(rhs,y0,ts=ts,t1=6.,dt0=.05,adaptive=False,method=method) + np.testing.assert_allclose(fixed.ys,reference(.05,diffrax.ConstantStepSize()).ys,rtol=0,atol=1e-14) + for rtol in (1e-6,1e-9): + native=rk.solve_rk(rhs,y0,ts=ts,t1=6.,dt0=.1,rtol=rtol,atol=rtol,method=method,controller='pi') + other=reference(.1,diffrax.PIDController(rtol=rtol,atol=rtol,pcoeff=.4,icoeff=.3)) + assert native.stats.accepted==other.stats['num_accepted_steps'] + np.testing.assert_allclose(native.ys,other.ys,rtol=0,atol=2*rtol) + + @pytest.mark.parametrize('method', ['dopri5','dopri8']) def test_adaptive_control_rejects_and_is_independent_of_saves(method): for controller in ('i','pi'): @@ -214,15 +235,8 @@ def test_boozer_public_native_outputs_and_failures(monkeypatch): assert legacy.terminal_states.shape==(1,5) and np.isnan(legacy.terminal_states).all() for change in (dict(integrator='x'),dict(method='rk4'),dict(species=object()),dict(adaptive='yes'),dict(rtol=-1.),dict(atol=0.)): with pytest.raises(ValueError): trace_boozer(field,[.3],[.4],[0.],[.2],**(kwargs|change)) - from essos import rk as module - original=module._prepare_boozer_gpu - monkeypatch.setattr(module,'_prepare_boozer_gpu',lambda *a,**k:original(*a,**k,interpret=True)) - out=trace_boozer(field,[.3],[.4],[0.],[.2],**(kwargs|dict(native_execution='cuda',progress=lambda *a:None))) - assert not out.failed.any() - with pytest.raises(ValueError,match='I control'): - trace_boozer(field,[.3],[.4],[0.],[.2],**(kwargs|dict(native_execution='cuda',native_controller='pi'))) - with pytest.raises(ValueError,match='cosine'): - original(SimpleNamespace(table=1.,sine_coef=None),1.,1.,1.,[0.,.1],.1,1e-9,1e-9,8,interpret=True) + with pytest.raises(ValueError,match='native_execution'): + trace_boozer(field,[.3],[.4],[0.],[.2],**(kwargs|dict(native_execution='cuda'))) def test_vmec_private_values_match_current_main(monkeypatch): @@ -319,44 +333,21 @@ def changing(refs,point,invariant): def test_rhs_preparation_and_gpu_helpers_preserve_values(monkeypatch): - from essos.rk import _select_one, _kernel_is_finite, _bind_primitive + from essos.rk import _select_one, _kernel_is_finite for dtype in (jnp.float32,jnp.float64): values=jnp.array([1.,np.inf,np.nan,-np.inf,-0.],dtype=dtype) np.testing.assert_array_equal(_kernel_is_finite(values),jnp.isfinite(values)) assert np.signbit(_select_one(values,4,0).reshape(())) - class Primitive: - def get_bind_params(self,p): return ([],p) - def bind(self,*args,**kwargs): return args[0]+kwargs['offset'] - assert _bind_primitive(Primitive(),[2.],dict(offset=3.))==5. - # JAX 0.6 calls this boundary "pjit"; current releases call it "jit". - nested=jax.make_jaxpr(lambda x:x*jnp.array([2.,3.]))(jnp.ones(2)) - for name in ('jit','pjit'): - graph=SimpleNamespace(constvars=(),invars=('x',),outvars=('y',),eqns=( - SimpleNamespace(primitive=SimpleNamespace(name=name),params={'jaxpr':nested}, - invars=('x',),outvars=('y',)),)) - inline=lambda x:rk._evaluate_inline(graph,(),x)[0].sum() - np.testing.assert_allclose(jax.jit(inline)(jnp.ones(2)),5.,atol=0) - np.testing.assert_array_equal(jax.grad(inline)(jnp.ones(2)),[2.,3.]) def value(p): rhs=lambda t,y,args:jnp.sin(p)*y+t prepared=rk._prepare_rhs(rhs,jnp.asarray(0.),jnp.ones(2),None) return prepared(.2,jnp.array([2.,3.]),None).sum() np.testing.assert_allclose(jax.grad(value)(.3),5*np.cos(.3),atol=2e-14) with pytest.raises(TypeError): _kernel_is_finite(jnp.array([1],dtype=jnp.int32)) - import essos.rk as module - from essos import boozer as boozer_module - model=SimpleNamespace(modB_derivatives=lambda *a:(1.,0.,0.,0.)) - for f in (lambda field,y,*a:jnp.concatenate((y[:2],y[:2])), - lambda field,y,*a:jnp.concatenate((jax.lax.dynamic_slice(y,(y[0].astype(jnp.int32),),(2,)),jnp.ones(2)))): - monkeypatch.setattr(boozer_module,'guiding_center_rhs',f) - with pytest.raises(ValueError,match='slice'): - module._kernel_rhs(model,jnp.ones(4),.1,1.,1.)(jnp.ones(4)) def test_fixed_field_adapter_solvers_on_cpu(): from essos.rk import prepare_solver - from essos.rk import _prepare_boozer_gpu as prepare_orbit_solver - from essos.rk import _boozer_batch as _native_orbits field=coil_field() args=dict(mass=1.,charge=1.,energy=.5,times=np.array([0.,.001]),dt0=.001,max_steps=8) with pytest.raises(ValueError): prepare_solver(UniformField(),execution='cuda',**args) @@ -369,11 +360,15 @@ def test_fixed_field_adapter_solvers_on_cpu(): gpu=jax.jit(prepare_solver(field,execution='cuda',interpret=True,**args))(y) assert gpu.status==0 np.testing.assert_allclose(gpu.y,reference.y,atol=3e-13) - boozer=tokamak(); y=jnp.array([[.5,0.,0.,.2]]); moments=jnp.array([.08]) - times=np.array([0.,.001]) - for method in ('dopri5','dopri8'): - solve=prepare_orbit_solver(boozer,1.,1.,1.,times,.001,1e-9,1e-9,8,method=method,interpret=True) - args=(boozer,y,moments,1.,1.,1.,jnp.asarray(times),.001,1e-9,1e-9,method,True,'i',.4,.3,8) - expected=_native_orbits(*args) - actual=_native_orbits(*args,orbit_solver=solve) - for a,b in zip(actual,expected): np.testing.assert_allclose(a,b,atol=3e-13) + + +def test_unsupported_pallas_lowering_falls_back_to_portable_path(monkeypatch): + from essos.rk import prepare_solver + monkeypatch.setattr(rk,'require_cuda',lambda interpret:None) + monkeypatch.setattr(rk,'_kernel_lowers',lambda:False) + args=dict(mass=1.,charge=1.,energy=.5,times=np.array([0.,.001]),dt0=.001,max_steps=8) + y=jnp.array([1.,.03,.01,.3]) + with pytest.warns(UserWarning,match='portable'): + fallback=jax.jit(prepare_solver(coil_field(),execution='cuda',**args))(y) + reference=jax.jit(prepare_solver(coil_field(),**args))(y) + np.testing.assert_array_equal(fallback.y,reference.y) diff --git a/tests/test_native_split.py b/tests/test_native_split.py index 37432240..b0c240f8 100644 --- a/tests/test_native_split.py +++ b/tests/test_native_split.py @@ -1,10 +1,9 @@ -"""Split orbit/collision solver: physics, numerics and execution-equivalence contracts. +"""Split orbit/collision solver: physics and numerics contracts. Collision maps are checked against the Coulomb Fokker-Planck generator (Lorentz pitch-angle moments, Maxwellian stationarity, Einstein relation), the Brownian bridge against its conditional law, and the Lie splitting against its exact -first-order weak/strong errors. The persistent GPU kernel is checked on CPU through -the Pallas interpreter against shared JAX execution. CPU-only CI. +first-order weak/strong errors. CPU-only CI. """ from pathlib import Path from types import SimpleNamespace @@ -18,7 +17,7 @@ from essos.constants import PROTON_MASS as M, ELEMENTARY_CHARGE as Q, EPSILON_0, HBAR, ONE_EV from essos.dynamics import Particles from essos.electric_field import Electric_field_zero -from tests.test_native_orbit import UniformField, coil_field, tokamak +from tests.test_native_orbit import UniformField, tokamak VMEC_FILE = Path(__file__).resolve().parents[1]/'examples/input_files/wout_LandremanPaul2021_QA_reactorScale_lowres.nc' @@ -154,7 +153,8 @@ def test_guiding_center_map_has_lorentz_and_speed_generator_moments(): for h in (.02, .01): mean = expectation(kick(h, xi), h, 2) # E[dv] = A h + O(h^2): the 2 D_par/v drift enters as a perpendicular kick, v -> sqrt(v1^2 + 4 D_par h). - np.testing.assert_allclose(mean[0]-v-drift*h, 0., atol=2e-2*abs(drift)*h) + speed_error = mean[0]-v-drift*h + np.testing.assert_allclose(speed_error, 0., atol=2e-2*abs(drift)*h) np.testing.assert_allclose(mean[1]-xi, -nu*xi*h, rtol=1e-11) # Lorentz: E[dxi] = -nu xi h # E[dxi^2] -> nu(1-3xi^2) h is the l=2 Legendre eigenvalue, 3 nu; errors are O(h^2). residual.append([mean[3]-xi**2-nu*(1-3*xi**2)*h, mean[2]-v**2-(2*v*drift+2*c.parallel)*h]) @@ -390,11 +390,9 @@ def test_cartesian_gc_and_full_orbit_trace_with_the_actual_maps(): assert s.trace_split(field, birth, bath(), tmax=1e-10, timestep=5e-11, n_save=3, orbit=False, collision_controller='conservative').status[0] == 0 for change in (dict(integrator='bad'), dict(model='bad'), dict(collision_controller='paper'), dict(collision_tolerance=0.), - dict(n_save=1), dict(execution='bad'), dict(spatial_diffusion=True, model='FullOrbit'), + dict(n_save=1), dict(spatial_diffusion=True, model='FullOrbit'), dict(speed_floor=-1.), dict(times=jnp.zeros((1, 1)))): with pytest.raises(ValueError): s.trace_split(field, birth, bath(), tmax=1e-10, timestep=1e-10, **change) - for change in (dict(step_limit=lambda t, y, a: 1., execution='independent'),): - with pytest.raises(ValueError): s.trace_split(field, birth, bath(), tmax=1e-10, timestep=1e-10, **change) with pytest.raises(ValueError): s.trace_split(field, birth, None, tmax=1e-10, timestep=1e-10) with pytest.raises(ValueError): s.trace_split(field, birth, SimpleNamespace(number_species=0), tmax=1e-10, timestep=1e-10) for parameter, value in (('mass', -1.), ('charge', 0.), ('initial_vparallel_over_v', jnp.array([2.]))): @@ -454,7 +452,7 @@ def test_vmec_and_boozer_spatial_kicks_equal_the_cartesian_formula(): if field is not vmec: with pytest.raises(ValueError, match='spatial_geometry'): s.trace_split(field, birth, bath(), tmax=1e-11, timestep=1e-11) - for change in (dict(model='FullOrbit'), dict(execution='independent')): + for change in (dict(model='FullOrbit'),): with pytest.raises(ValueError): s.trace_split(field, birth, bath(), tmax=1e-11, timestep=1e-11, **change) assert out.stats.accepted[0] > 0 output = s.trace_split(field, birth, bath(), tmax=1e-11, timestep=1e-11, n_save=2, spatial_geometry=embedding, @@ -600,101 +598,3 @@ def gc_quantities(self, p): def dAbsB_by_dX(self, p): return self.gc_quantities(p)[3] def curl_b(self, p): return self.gc_quantities(p)[4] def kappa(self, p): return self.gc_quantities(p)[5] - - -# Independent GPU kernel, interpreted on CPU, against shared JAX execution - -def run_pair(field, background, model='GuidingCenter', **kwargs): - options = dict(model=model, tmax=2e-8, timestep=1e-8, n_save=3, max_steps=64, max_orbit_steps=64) | kwargs - if hasattr(field, 'coils'): - birth = Particles(jnp.array([[1., .03, .01], [1., .02, .0]]), mass=M, charge=Q, energy=4000*ONE_EV, - initial_vparallel_over_v=jnp.array([.5, .3]), field=field, phase_angle_full_orbit=0.) - else: - birth = Particles(jnp.array([[.3, .2, .1]]), mass=M, charge=Q, energy=4000*ONE_EV, initial_vparallel_over_v=jnp.array([.4])) - shared = s.trace_split(field, birth, background, **options) - independent = s.trace_split(field, birth, background, execution='independent', interpret=True, **options) - return shared, independent - - -def assert_equivalent(shared, independent): - assert np.all(shared.status == independent.status) and np.all(shared.status == 0) - np.testing.assert_allclose(independent.ys, shared.ys, rtol=1e-11, atol=1e-14) - np.testing.assert_allclose(independent.y, shared.y, rtol=1e-11, atol=1e-14) - for name in ('steps', 'accepted', 'rejected', 'orbit_steps', 'accepted_orbit_steps'): - np.testing.assert_array_equal(getattr(independent.stats, name), getattr(shared.stats, name)) - - -@pytest.mark.parametrize('model,adaptive', [('GuidingCenter', False), ('GuidingCenter', True), ('FullOrbit', True)]) -def test_independent_kernel_matches_shared_execution_in_coil_fields(model, adaptive): - options = dict(adaptive=adaptive, orbit_adaptive=adaptive, spatial_diffusion=model == 'GuidingCenter') - assert_equivalent(*run_pair(coil_field(), bath(), model, **options)) - - -def test_independent_kernel_matches_shared_execution_in_vmec_and_boozer_fields(): - from essos.fields import Vmec - vmec = Vmec(str(VMEC_FILE), ntheta=3, nphi=3) - assert_equivalent(*run_pair(vmec, bath(), adaptive=False, orbit_adaptive=False, spatial_diffusion=False)) - # Adaptive Boozer with the moment predictor and nonlinear Cartesian spatial kicks. - assert_equivalent(*run_pair(tokamak(), bath(), adaptive=True, orbit_adaptive=True, spatial_diffusion=True, - spatial_geometry=boozer_geometry())) - - -def test_independent_kernel_handles_radial_profiles_events_and_failures(): - varying = BackgroundSpecies(1, jnp.array([1.]), jnp.array([1.]), jnp.array([[.5e20, 1.5e20]]), jnp.array([[800., 1200.]]), - radial_grid=jnp.array([.0, 2.]), coulomb_log_model='impact') - field = coil_field() - assert_equivalent(*run_pair(field, varying, adaptive=True, orbit_adaptive=True, spatial_diffusion=False)) - # A stopping surface ends the kernel run exactly like the shared solver. - shared, independent = run_pair(field, bath(), adaptive=False, orbit_adaptive=False, tmax=1e-6, timestep=1e-7, n_save=3, - event=lambda t, y, a: .03-y[2], max_steps=200, max_orbit_steps=200) - np.testing.assert_array_equal(shared.status, independent.status) - np.testing.assert_allclose(independent.t, shared.t, rtol=1e-9) - # A tiny step budget reports MAX_STEPS rather than a wrong answer. - shared, independent = run_pair(field, bath(), adaptive=False, orbit_adaptive=False, max_steps=1) - assert np.all(independent.status == shared.status) and np.all(independent.status == s.SplitStatus.MAX_STEPS) - - -def test_scalar_lowering_reproduces_plain_jax_results(): - """The kernel's jaxpr interpreters (scalar-only, no small arrays) match direct evaluation.""" - x = jnp.array([.2, -.3, .4, .5, 0., 0., 0., 0.]) - functions = (lambda x: x[1:2], lambda x: jax.lax.dynamic_slice(x, (x[0].astype(int),), (1,)), - lambda x: jnp.sin(x)+jnp.cos(x), jax.lax.erf, lambda x: jnp.array([jnp.sum(x), jnp.max(x), jnp.min(x)]), - lambda x: jnp.stack([x]), lambda x: x[jnp.array([2])], lambda x: (jnp.any(x > 0), jnp.all(x < 1)), - tuple, lambda x: tuple(jnp.split(x, len(x))), lambda x: x**-2+x**3+x**6) - for function in functions: - graph = s._flatten(function, x) - actual = jax.jit(lambda x: s._evaluate_kernel(graph.jaxpr, graph.consts, x))(x) - for a, b in zip(actual, jax.tree.leaves(function(x))): - np.testing.assert_allclose(a, b, atol=2e-15, rtol=1e-14) - for function in (lambda y: 1-jnp.sum(y[:3]**2), lambda y: jnp.dot(y[:3], y[:3])-.1, lambda y: jnp.max(y[:3])-jnp.min(y[:3]), - lambda y: y[jnp.array([1])][0], lambda y: sum(y[:3])): - graph = s._flatten(function, x) - np.testing.assert_allclose(s._evaluate_surface(graph.jaxpr, graph.consts, x), function(x), atol=2e-15) - - def scanned(point, reverse): - carry, ys = jax.lax.scan(lambda a, b: (a+point[0]*b, a), point[1], point[2:5], reverse=reverse) - return jnp.r_[carry, ys] - interpolation = lambda point: jnp.interp(point[0], jnp.array([-1., 0., 1.]), point[1:4]) - loop = lambda p: jax.lax.while_loop(lambda c: c[0] < 3+p[0], lambda c: (c[0]+1, c[1]+p[1:3]), (jnp.asarray(0), jnp.ones(2)))[1] - for fn in (lambda p: scanned(p, False), lambda p: scanned(p, True), interpolation, jax.jacfwd(interpolation), loop): - flat = lambda p: tuple(jnp.ravel(fn(p))) - graph = s._flatten(flat, x) - actual = jax.jit(lambda p: jnp.asarray(s.scalar_graph(graph.jaxpr, graph.consts, p)))(x) - np.testing.assert_allclose(actual, jnp.asarray(flat(x)), rtol=3e-13, atol=3e-14) - - -def test_independent_execution_validates_its_capabilities(): - field = coil_field() - birth = Particles(jnp.array([[1., .03, .01]]), mass=M, charge=Q, energy=4000*ONE_EV, initial_vparallel_over_v=jnp.array([.5])) - base = dict(tmax=1e-8, timestep=1e-8, n_save=2) - with pytest.raises(ValueError, match='GPU'): # no silent fallback on CPU - s.trace_split(field, birth, bath(), execution='independent', **base) - pinned = dict(execution='independent', interpret=True, **base) - for change in (dict(integrator='essos_rk45'), dict(controller='pi'), dict(orbit_adaptive=False), dict(reuse_orbit_step=False), - dict(electric_field=object()), dict(record_steps=True), dict(times=jnp.array([])), - dict(event=lambda t, y, a: True), dict(event=lambda t, y, a: jnp.ones(2)), dict(event=lambda t, y, a: 1.-t), - dict(max_steps=0), dict(max_orbit_steps=False), dict(max_pending=0)): - with pytest.raises(ValueError): s.trace_split(field, birth, bath(), **(pinned | change)) - old = birth.random_keys; birth.random_keys = jax.random.key(0, impl='rbg')[None] - with pytest.raises(ValueError): s.trace_split(field, birth, bath(), **pinned) - birth.random_keys = old