Differential privacy for aggregates - add dp_laplace and dp_gaussian built-in functions with budget accounting by andersonm-ibm · Pull Request #2539 · apache/systemds

andersonm-ibm · 2026-07-09T21:41:55Z

Add two native DML built-ins for differentially private aggregate release "colMeans", "colSums" or "identity":

result = dp_laplace(X, "colMeans", sensitivity, epsilon)
result = dp_gaussian(X, "colSums", sensitivity, epsilon, delta)

Wire them through the full compilation pipeline:
Builtins → BuiltinFunctionExpression → ParameterizedBuiltinOp HOP → ParameterizedBuiltin LOP → DPBuiltinCPInstruction.
Introduce DPBudgetAccountant, a session-scoped privacy budget tracker stored on ExecutionContext. Laplace releases use exact pure-ε composition; Gaussian releases use Rényi DP composition (Mironov 2017) with RDP → (ε,δ) conversion for tighter bounds. Raises DMLRuntimeException if cumulative spend exceeds the budget.
Unit tests covering constructor validation, Laplace/Gaussian composition, budget exhaustion for both mechanisms, mixed composition, release counting, and RDP mathematical invariants (sensitivity cancellation, ε-monotonicity).
End-to-end DML integration tests in DPBuiltinDMLTest verify noisy output differs from clean means by a statistically plausible amount.
Differential Privacy Benchmark:
Four federated workers simulated on localhost, a logistic regression FedAvg loop in DML where the coordinator applies dp_gaussian to the aggregated gradient, a sweep over ε ∈ {0.5, 1, 4, 8} plus a non-private baseline, and a matplotlib accuracy-vs-ε plot saved as a PNG.

CC @ywcb00

…ransformation matrix T internally, returning T %*% X with noise fused into a single matrix multiply.

…ng to David's suggestion

Lets a DML script declare its session-wide differential-privacy budget once at the top, instead of always falling back to the hardcoded default. Resolved entirely at compile time: epsilon/delta must be literals, validated in BuiltinFunctionExpression and stored on DMLProgram during HOP construction, then read by ExecutionContext.getDPBudgetAccountant().

Four federated workers simulated on localhost, a logistic regression FedAvg loop in DML where the coordinator applies dp_gaussian to the aggregated gradient, a sweep over ε ∈ {0.5, 1, 4, 8} plus a non-private baseline, and a matplotlib accuracy-vs-ε plot saved as a PNG. Add clip_norm (default 4.0) as a script parameter. Inside the private == 1 branch, each row's gradient contribution is clipped to L2-norm less than clip_norm.

codecov · 2026-07-27T07:27:26Z

Codecov Report

❌ Patch coverage is 5.76923% with 245 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.24%. Comparing base (e557836) to head (19f6fd0).
⚠️ Report is 27 commits behind head on main.

Files with missing lines	Patch %	Lines
...untime/instructions/cp/DPBuiltinCPInstruction.java	0.00%	94 Missing ⚠️
...apache/sysds/parser/BuiltinFunctionExpression.java	0.00%	52 Missing ⚠️
...e/sysds/runtime/privacy/dp/DPBudgetAccountant.java	0.00%	40 Missing ⚠️
...in/java/org/apache/sysds/parser/DMLTranslator.java	0.00%	29 Missing ⚠️
.../org/apache/sysds/hops/ParameterizedBuiltinOp.java	9.09%	7 Missing and 3 partials ⚠️
...va/org/apache/sysds/lops/ParameterizedBuiltin.java	0.00%	8 Missing ⚠️
.../main/java/org/apache/sysds/parser/DMLProgram.java	0.00%	6 Missing ⚠️
...ntime/controlprogram/context/ExecutionContext.java	16.66%	5 Missing ⚠️
...ysds/runtime/instructions/CPInstructionParser.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##               main    #2539       +/-   ##
=============================================
- Coverage     71.61%   17.24%   -54.37%     
+ Complexity    49867    11450    -38417     
=============================================
  Files          1602     1625       +23     
  Lines        193054   194562     +1508     
  Branches      37792    38005      +213     
=============================================
- Hits         138247    33553   -104694     
- Misses        44031   155193   +111162     
+ Partials      10776     5816     -4960

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ywcb00

Hi @andersonm-ibm,
I did a pass through the code and left some comments inline related to code style and integration in SystemDS.
The failing Java Test workflow was not caused by the changes of this PR.
All the best,
David

ywcb00 · 2026-07-27T10:18:16Z

 	MMChain,
 	Union,
 	EINSUM,
+	DPBuiltin,


Do we need a separate instruction type or could we also use the already existing ParameterizedBuiltin type?

ywcb00 · 2026-07-27T10:19:37Z

+	DP_LAPLACE("dp_laplace", false),
+	DP_GAUSSIAN("dp_gaussian", false),
+	DP_SET_BUDGET("dp_set_budget", false),


Keep the alphabetical order.

ywcb00 · 2026-07-27T10:24:37Z

+			String dpLaplaceQuery = getDPQueryLiteral(getSecondExpr());
+			long[] dpLaplaceDims = getDPOutputDims(dpLaplaceQuery,
+				getFirstExpr().getOutput().getDim1(), getFirstExpr().getOutput().getDim2());


Since these two functions are always called together, can we merge them to have a single function call to get the output dimensions?

ywcb00 · 2026-07-27T10:29:43Z

 		case DECOMPRESS:
 			currBuiltinOp = new UnaryOp(target.getName(), target.getDataType(), ValueType.FP64, OpOp1.DECOMPRESS, expr);
 			break;
+		case DP_LAPLACE: {


Move these cases to the method processParameterizedBuiltinFunctionExpression above to simplify the parameter parsing.

ywcb00 · 2026-07-27T10:31:50Z

+ * analysis that derives sensitivity from {@code T}'s column norms and a declared per-record bound on {@code X}; every
+ * other line in this class would stay unchanged.
+ */
+public class DPBuiltinCPInstruction extends ComputationCPInstruction {


Can we extend from ParameterizedBuiltinCPInstruction, since the DP builtins belong to the parameterized builtin functions?

ywcb00 · 2026-07-27T11:43:45Z

+	private static final double EPS = 1e-9;
+
+	// =======================================================================
+	// 1. DPBudgetAccountant unit tests


What does the '1.' refer to?

ywcb00 · 2026-07-27T11:44:11Z

+	}
+
+	// =======================================================================
+	// 1b. DMLProgram / ExecutionContext.getDPBudgetAccountant() (dp_set_budget)


ywcb00 · 2026-07-27T11:46:41Z

+ *
+ * Full integration tests extend AutomatedTestBase and drive the DML runner.
+ * Each test:
+ *   (a) Writes a DML script to a temp file.


Create the DML scripts directly as files under test/scripts, similar to other unit tests.

ywcb00 · 2026-07-27T11:47:56Z

+ * Full integration tests extend AutomatedTestBase and drive the DML runner.
+ * Each test:
+ *   (a) Writes a DML script to a temp file.
+ *   (b) Provides input matrices via TestUtils.


Use the method getRandomMatrix() from AutomatedTestBase for generating random test matrices.

ywcb00 · 2026-07-27T11:49:08Z

 				<artifactId>maven-surefire-plugin</artifactId>
 				<version>${maven-surefire-plugin.version}</version>
 				<configuration>
+					<reportFormat>plain</reportFormat>


What is the reason for this change? What is it needed for?

Maya Anderson added 19 commits July 7, 2026 23:33

Intro notebooks

ef3c0c0

Add CLAUDE.md

021bb7d

Initial DPBuiltinCPInstruction and RDPAccountant

4e9bdd1

Add dp_gaussian and dp_laplace operations

19c2504

CLAUDE_CODE_PLAN.md

7dc493c

Fix test package name.

bd7a956

Fix Laplace accountant

5d31fda

Add unit tests

4b58bc4

Rename RDPAccountant to DPBudgetAccountant

2936970

Delete RDPAccountant

c6b7c82

Fix parseInstruction() comment

ef4680f

Integration tests

1d0fac3

Update test comments

ae2afcb

Fix integration test to compare clean to noisy data

394c021

Make integration test more readable

67fe1b2

Comment fix

fbe18f0

Fix L1 L2 sensitivity parameter documentation

f22c283

junit output format

461da60

Remove Claude files

b253d14

github-project-automation Bot added this to SystemDS PR Queue Jul 9, 2026

github-project-automation Bot moved this to In Progress in SystemDS PR Queue Jul 9, 2026

Remove files irrelevant to the PR

379ddcd

ywcb00 self-assigned this Jul 10, 2026

Maya Anderson added 6 commits July 15, 2026 00:13

dp_laplace and dp_gaussian now take the original matrix and build a t…

4159947

…ransformation matrix T internally, returning T %*% X with noise fused into a single matrix multiply.

Remove 4th and 5th parameters and instead get them in a loop, accordi…

d8f0f45

…ng to David's suggestion

Change DPBudgetAccountant defaults to be constants

025f9bb

Cleanup comments

f29cd3e

andersonm-ibm force-pushed the diff_privacy_pub branch from 12a37d5 to d8912b1 Compare July 14, 2026 22:37

andersonm-ibm marked this pull request as ready for review July 14, 2026 22:38

andersonm-ibm changed the title ~~WIP: Differential privacy for aggregates - add dp_laplace and dp_gaussian built-in functions with budget accounting~~ Differential privacy for aggregates - add dp_laplace and dp_gaussian built-in functions with budget accounting Jul 14, 2026

Maya Anderson added 4 commits July 21, 2026 23:39

Checkstyle fix

c517af2

Add missing licenses

ae8a136

More checkstyle fixes

4db9f40

Add missing break. Fixes test

19f6fd0

ywcb00 requested changes Jul 27, 2026

View reviewed changes

github-project-automation Bot moved this from In Progress to In Review in SystemDS PR Queue Jul 27, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Differential privacy for aggregates - add dp_laplace and dp_gaussian built-in functions with budget accounting - #2539

Differential privacy for aggregates - add dp_laplace and dp_gaussian built-in functions with budget accounting#2539
andersonm-ibm wants to merge 30 commits into
apache:mainfrom
andersonm-ibm:diff_privacy_pub

andersonm-ibm commented Jul 9, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 27, 2026

Uh oh!

ywcb00 left a comment

Uh oh!

ywcb00 Jul 27, 2026

Uh oh!

ywcb00 Jul 27, 2026

Uh oh!

ywcb00 Jul 27, 2026

Uh oh!

ywcb00 Jul 27, 2026

Uh oh!

ywcb00 Jul 27, 2026

Uh oh!

ywcb00 Jul 27, 2026

Uh oh!

ywcb00 Jul 27, 2026

Uh oh!

ywcb00 Jul 27, 2026

Uh oh!

ywcb00 Jul 27, 2026

Uh oh!

ywcb00 Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

andersonm-ibm commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jul 27, 2026

Codecov Report

Uh oh!

ywcb00 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

andersonm-ibm commented Jul 9, 2026 •

edited

Loading